Search Results: "zed"

15 February 2012

Niels Thykier: Python optimization and profiling

For the past 5 days, I have worked on replacing a part of Britney (our beloved testing migration tool). The part I am trying replacing is her installability tester , which is probably one of the most used parts of Britney. I know premature optimization is the root of all evil, but I felt compelled to be a bit evil this time. The reason is that a na ve algorithm could easily spend decades on computing the result of our most complete tests (each consists of roughly 2 * 35 000 packages). Within a couple of days, I felt I had a reasonable candidate to use on the large tests. On the large tests, it ran with a noticeable speed regression and it produced one different result. The result difference was about 20-40 source packages and their binaries. You may have noticed I used difference and not regression . In the given test, our current implementation at some point gives up with an AIEEE error. So I decided to turn my attention to the speed regression first. As any good programmer, I decided to use a profiler (cProfile) to determine my bottle-necks. This turned out to be a good idea, as the bottleneck was not quite where I thought it was. I wish I could say I just patched it out the issue, but unfortunately not. I played around with various changes such as using implied for loops rather than explicit ones etc. Some times I would manage to cut the runtime in half only to be unable to reproduce it later. Yesterday I finally realized what caused this. I was working in 3 different chroots, two 64-bit sid chroots[1] and a 32-bit stable chroot. Turns out that my replacement is only slow in the stable chroot. As soon as I moved to a sid chroot, the runtime was more or less cut in half. I have not bothered to check if it is the 32 vs 64 bit, the stable vs sid part or maybe the python profile in stable is just slow[2]. I simply moved to my sid chroots for now and continued working. With that taken care of, I figured I would remind myself of what I was up against. On the largest test I have available, I profiled my implementation to around 4:30 5:00 minutes. Regardless of my changes, I always got times in that range. I did manage to patch a bug in my code that reduced by only diff to 10 source packages (plus their binaries) at the price of 30 seconds. So I was back to 5 minutes according to the profiler, but I noticed that my test runner disagreed. According to my test runner my implementation had a runtime of approx. 3 minutes and 40 seconds. For comparison my test runner puts the original implementation at 3 minutes and 20 seconds for that test. In short, I have spent most of my day trying to optimize my code to compensate for the +20% overhead the profiler introduced. Admittedly, my replacement still needs some speed improvement on some other tests where it still has a factor 2 runtime regression. I also need to fix the last diff, which now I suspect is in my replacement. [1] In case you were wondering, yes those chroots are on two different machines. [2] Of course, it could be a combination of that as well Anyhow, I am too lazy to research it at the moment.

12 February 2012

Christian Perrier: 2012 update 9 for Debian Installer localization

Status for D-I level 1 (core D-I files): Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.): Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader) Full 100% completeness (hall of fame) for 18 languages: Bulgarian, Czech, German, Esperanto, Spanish, Persian, French, Indonesian, Italian, Japanese, Kazakh, Dutch, Polish, Portuguese, Russian, Slovak, Turkish Simplified Chinese

Jose Luis Rivas Contreras: Express Sessions with connect-redis + Socket.IO in CoffeeScript

If you are working with Socket.IO and Express Framework while developing your applications you must know they work together so Socket.IO will listen on the same port that your HTTP/S server and will serve the requests by that way, but Socket.IO doesn t works on the same paradigm where you pass a middleware to the route and you could get authentication on this manner. Instead, you must configure Socket.IO with a global authorization method or a handshake authorization manner. Since in my applications I serve cookies already where I save session info and this is already stored in Redis with help of the module connect-redis then I should be able to get this info onto Socket.IO authorization method and with this allow only to authorized users to get info from my websockets. I found how to do it with the default session-store in-memory from Express, and adapted that so it could work with Redis. Here s the snippet: <script src="https://gist.github.com/1805743.js"> </script><noscript>You don t have JavaScript enabled, check it out at gist.github.com </noscript> Hope anyone can found it useful!

9 February 2012

Vincent Bernat: Recipes for extending Net-SNMP

SNMP stands for Simple Network Management Protocol. It allows a manager to query information from an agent. A popular use of SNMP is to retrieve network interface counters to plot a bandwidth graph. While the S in SNMP stands for simple, SNMP can be quite difficult to deal with. However, it is still the de facto standard for retrieving metrics in an heterogeneous environment. Net-SNMP is a suite of SNMP applications including an agent. Out of the box, this agent exports a lot of information but if something is missing, there are several ways to extend it. Which ones? tl;dr: in my opinion, extending Net-SNMP should be done with either of those methods:

SNMP in a nutshell The variables accessible with SNMP through the agent are organized in a tree. Each variable is typed and associated to an object identifier (or OID). For example, .1.3.6.1.2.1.31.1.1.1.1.2 is the OID of the name of the second network card. Usually, SNMP is operated over UDP, port 161. For the purpose of this article, we consider a manager can issue three kinds of requests:
GET
retrieve a variable.
GETNEXT
retrieve the next variable, in lexical order of their OID. This operation enables an SNMP manager to walk through all available variables, for example to discover the list of interfaces of an agent. This operation is what makes an SNMP agent difficult to implement but this is also a critical feature.
SET
request the modification of a variable.
Since such an OID may be difficult to handle by humans, available variables are described by a MIB module which is a file that should be parsed to get a mapping between OID and variable names and types. For example, the above OID can also be referred as IF-MIB::ifName.2. Those MIB modules are described using a subset of ASN.1 defined in RFC 2578. A MIB module allows to group several variables into a conceptual table. For example, IF-MIB::ifDescr, IF-MIB::ifType, IF-MIB::ifSpeed are columns of IF-MIB::ifTable. Each row is associated to an index. In the case of IF-MIB::ifTable, it is the interface index. Net-SNMP includes snmptable allowing one to display such a table in a natural manner1: 
$ snmptable localhost IF-MIB::ifTable
SNMP table: IF-MIB::ifTable
 ifIndex ifDescr           ifType ifMtu   ifSpeed
       1      lo softwareLoopback 16436  10000000
       2    eth0   ethernetCsmacd  1500 100000000
       3    eth1   ethernetCsmacd  1500  10000000
       4     br0   ethernetCsmacd  1500         0
Variables inside a table are called columnar objects while those outside are scalar objects. SNMP, as a protocol, does not care of this distinction since it does not know the concept of MIB modules. Other useful tools from Net-SNMP includes snmpget to get a specific variable, snmpwalk to discover available variables (with GETNEXT operation) from an OID and snmptranslate to translate between object names and OID. 
$ snmpget localhost IF-MIB::ifDescr.2
IF-MIB::ifDescr.2 = STRING: eth0
$ snmpwalk localhost IF-MIB::ifDescr
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1
IF-MIB::ifDescr.4 = STRING: br0
$ snmpwalk -On public localhost .1.3.6.1.2.1.2.2.1.2
.1.3.6.1.2.1.2.2.1.2.1 = STRING: lo
.1.3.6.1.2.1.2.2.1.2.2 = STRING: eth0
.1.3.6.1.2.1.2.2.1.2.3 = STRING: eth1
.1.3.6.1.2.1.2.2.1.2.4 = STRING: br0
$ snmptranslate -On IF-MIB::ifDescr.3
.1.3.6.1.2.1.2.2.1.2.3

Extending Net-SNMP ethtool -S allows us to access various statistics from a network card, like the number of octets received and transmitted or the number of collisions. While some of those statistics are defined in IF-MIB, in RMON-MIB and in EtherLike-MIB, some of them are not available and very specific. Therefore, we would like to export those statistics, unmodified, through SNMP. To the best of my knowledge, no MIB module exists for such an usage. The first task is to write one. It can be called ETHTOOL-MIB. It only contains one table indexed by the interface index and the name of the statistic. Here is the expected output: 
$ snmpwalk localhost ETHTOOL-MIB::ethtoolStat.2
ethtoolStat.2.'align_errors' = Counter64: 0
ethtoolStat.2.'broadcast' = Counter64: 25
ethtoolStat.2.'multicast' = Counter64: 345
ethtoolStat.2.'rx_errors' = Counter64: 0
ethtoolStat.2.'rx_missed' = Counter64: 0
ethtoolStat.2.'rx_packets' = Counter64: 262011
ethtoolStat.2.'tx_aborted' = Counter64: 0
ethtoolStat.2.'tx_errors' = Counter64: 0
ethtoolStat.2.'tx_multi_collisions' = Counter64: 0
ethtoolStat.2.'tx_packets' = Counter64: 296170
ethtoolStat.2.'tx_single_collisions' = Counter64: 0
ethtoolStat.2.'tx_underrun' = Counter64: 0
ethtoolStat.2.'unicast' = Counter64: 261641
I have coded various implementations of this MIB module and made them available in a GitHub repository.

With arbitrary commands The extend directive allows the agent to execute an arbitrary command and to provide its output and status through NET-SNMP-EXTEND-MIB module. If we want to export the statistics for eth0, we could add something like this in snmpd.conf, using a custom script to format the output of ethtool -S in a way that is more suitable for our use: 
# Export eth0 statistics
extend eth0-names  /full/path/to/ethtool-stats eth0 names
extend eth0-values /full/path/to/ethtool-stats eth0 values
The result can be retrieved through NET-SNMP-EXTEND-MIB:: nsExtendOutput2Table: 
$ snmpwalk localhost NET-SNMP-EXTEND-MIB::nsExtendOutput2Table
nsExtendOutLine."eth0-names".1 = STRING: tx_packets
nsExtendOutLine."eth0-names".2 = STRING: rx_packets
nsExtendOutLine."eth0-names".3 = STRING: tx_errors
nsExtendOutLine."eth0-names".4 = STRING: rx_errors
nsExtendOutLine."eth0-names".5 = STRING: rx_missed
nsExtendOutLine."eth0-names".6 = STRING: align_errors
nsExtendOutLine."eth0-names".7 = STRING: tx_single_collisions
nsExtendOutLine."eth0-names".8 = STRING: tx_multi_collisions
nsExtendOutLine."eth0-values".1 = STRING: 246309
nsExtendOutLine."eth0-values".2 = STRING: 223941
nsExtendOutLine."eth0-values".3 = STRING: 0
nsExtendOutLine."eth0-values".4 = STRING: 0
nsExtendOutLine."eth0-values".5 = STRING: 0
nsExtendOutLine."eth0-values".6 = STRING: 0
nsExtendOutLine."eth0-values".7 = STRING: 0
nsExtendOutLine."eth0-values".8 = STRING: 0
It is also possible to configure such a command using SET requests. For example, if we want to add statistics for eth2, we could issue the following command: 
$ snmpset -m +NET-SNMP-EXTEND-MIB localhost \
>    'nsExtendStatus."eth2-names"'  = createAndGo \
>    'nsExtendCommand."eth2-names"' = /full/path/to/ethtool-stats \
>    'nsExtendArgs."eth2-names"'    = 'eth2 names' \
>    'nsExtendStatus."eth2-values"'  = createAndGo \
>    'nsExtendCommand."eth2-values"' = /full/path/to/ethtool-stats \
>    'nsExtendArgs."eth2-values"'    = 'eth2 values'
nsExtendStatus."eth2-names" = INTEGER: createAndGo(4)
nsExtendCommand."eth2-names" = STRING: /full/path/to/ethtool-stats
nsExtendArgs."eth2-names" = STRING: eth2 names
nsExtendStatus."eth2-values" = INTEGER: createAndGo(4)
nsExtendCommand."eth2-values" = STRING: /full/path/to/ethtool-stats
nsExtendArgs."eth2-values" = STRING: eth2 values
$ snmpgetnext -m +NET-SNMP-EXTEND-MIB localhost \
>    'nsExtendOutLine."eth2-names"'
nsExtendOutLine."eth2-names".1 = STRING: tx_packets
This can be very convenient but also a huge security risk. You should disable this functionality or enable it only for some SNMPv3 user. Using extend does not allow to implement an arbitrary MIB module. If the cache is not disabled, extend is a good solution for various simple needs, like providing the content of a one-line file or the output of some Nagios plugin. However, exporting anything tabular, like interface statistics, is not a good fit.

With pass-through scripts Net-SNMP allows one to extend the agent by delegating some sub-tree to a script defined by the pass_persist directive. Such a script will receive requests on its standard input with a very simple line-oriented protocol described in the manual page of snmpd.conf. Here is an example of interaction: 
   PING
   PONG
   get
   .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116
   .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116
   counter64
   223941
   getnext
   .1.3.6.1.4.1.39178.100.1.1.1.2.2
   .1.3.6.1.4.1.39178.100.1.1.1.2.2.97.108.105.103.110.95.101.114
   counter64
   0
The protocol is simple enough to implement it from scratch in any scripting language. However, in Perl, you may prefer to use SNMP:: ExtensionSNMP:: PassPersist extension from S bastien Aperghis-Tramoni. You can find the complete example on Github. 
use SNMP::Extension::PassPersist;
my $extsnmp = SNMP::Extension::PassPersist->new(
    backend_collect => \&update_tree
    );
$extsnmp->run;
sub update_tree  
    my @interfaces = </sys/class/net/*>;
    foreach my $interface (@interfaces)  
        $interface =~ s/^.*\///;
        open(ETHTOOL, "ethtool -S $interface 2>/dev/null  ") or next;
        while (<ETHTOOL>)  
            /^\s+(\w+): (\d+)$/ or next;
            my $name  = $1;
            my $value = int($2);
            my $oid   = oid_compute($interface, $name);
            $extsnmp->add_oid_entry($oid, "counter", $value);
         
        close(ETHTOOL);
With Python, snmp-passpersist module, from Nicolas Agius, provides a similar interface. Look at the complete example on GitHub. pass_persist allows you to implement an arbitrary MIB module with honest performances. Here are some important things to know about this directive:
  • Unless you have a recent version of Net-SNMP, 64bit types are converted to 32bit equivalents. The counters may overflow quickly.
  • While handling a request, you are not allowed to pause (for example, to fetch a remote information): the agent would become unresponsive. If you need slow-to-get data or data from the agent, grab them in a separate thread and ensure you have them beforehand.

With AgentX protocol AgentX is a protocol defined in RFC 2741 allowing a master agent to be extended by independent sub-agents. For snmpd to become a master agent, you just need to add master agentx in snmpd.conf. Here are some of the benefits of an AgentX sub-agent over pass_persist:
  1. No configuration is needed for the master agent to accept an additional sub-agent. A sub-agent registers to the master agent the MIB modules (or part of them) it wants to take care of.
  2. A sub-agent is decoupled from the master agent. It can run with a different identity or be integrated into another daemon to export its internal metrics, send traps or allow remote configuration through SNMP.
  3. AgentX protocol can be carried over TCP. Sub-agents can therefore run on a foreign host or in a jailed environment.
  4. 64bit types are fully supported. Traps are also supported.

Perl and Python Net-SNMP provides a Perl module, NetSNMP::agent, to write a sub-agent using AgentX protocol. This module is not as convenient as the module for pass_persist described above since it does not provide a convenient layer to put required information into some datastore. Basically, it mimics the available C API. Here are a few examples of its use: There is no similar binding for Python but there is an agentx extension using ctypes. It provides a more high-level view. I did not write an example for this one but you can look at the example provided by the author.

C with new API To write a sub-agent in C, you have to choose between two API:
  1. the API inherited from UCD-SNMP also known as the traditional API;
  2. the new API developed for Net-SNMP 5.x.
With both API, an handler is registered to take care of one or several sub-trees. With the traditional API, the handler is invoked with very little information. It is then up to you to locate the appropriate variable by mapping the index part of the OID to the objects you want to export. It is quite simple for scalar objects but columnar objects with complex indexes are a pain with such an API. The new API provides various helpers. Some of them allow you to copy objects into an array or a list and let Net-SNMP library do the hard work of serving them. Another helper will provide methods to serve objects without putting them in a special structure but by providing an iterator. Some of those helpers can also help you to cache some variables. On top of this new API, Net-SNMP comes with mib2c, a tool that will help you convert a MIB module into C code: after a few questions, you will get a skeleton C code you will have to complete with the logic to retrieve real objects. A generated file also provides directions on what to do (with some magic command to have a step-by-step cookbook). I was able to implement ETHTOOL-MIB by just running it through mib2c, choosing the table container helper with cache and adding some code in ethtoolStatTable_data_access.c. You can look at the result on Github. Here is a stripped-down version2 of what has been modified: 
/* Socket for ioctl */
skfd = socket(AF_INET, SOCK_DGRAM, 0);
/* Iterate through all interfaces */
getifaddrs(&ifap);
for (ifa = ifap; ifa != NULL; ifa = ifa->ifa_next)  
    /* Grab statistics name and values */
    strcpy(ifr.ifr_name, ifa->ifa_name);
    drvinfo.cmd         = ETHTOOL_GDRVINFO;
    ifr.ifr_data        = (caddr_t) &drvinfo;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    n_stats             = drvinfo.n_stats;
    strings->cmd        = ETHTOOL_GSTRINGS;
    strings->string_set = ETH_SS_STATS;
    strings->len        = n_stats;
    ifr.ifr_data        = (caddr_t) strings;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    stats->cmd          = ETHTOOL_GSTATS;
    stats->n_stats      = n_stats;
    ifr.ifr_data        = (caddr_t) stats;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    ifIndex = if_nametoindex(ifa->ifa_name);
    /* Iterate through statistics */
    for (i = 0; i < n_stats; i++)  
        /* Declare the appropriate index */
        strncpy(ethtoolStatName,
                (char *)&strings->data[i * ETH_GSTRING_LEN],
                ETH_GSTRING_LEN);
        ethtoolStatName[sizeof(ethtoolStatName) - 1] = '\0';
        ethtoolStatName_len = strlen(ethtoolStatName);
        /* Append this new variable to the container. */
        rowreq_ctx = ethtoolStatTable_allocate_rowreq_ctx();
        ethtoolStatTable_indexes_set(rowreq_ctx,
                    ifIndex,
                    ethtoolStatName, ethtoolStatName_len);
        rowreq_ctx->data.ethtoolStat.high = stats->data[i] >> 32;
        rowreq_ctx->data.ethtoolStat.low = (uint32_t)stats->data[i];
        CONTAINER_INSERT(container, rowreq_ctx);
See? It is pretty easy. I have only enumerated all the objects I wanted to expose. I didn t even have to convert statistics names to an OID, mib2c built a ethtoolStatTable_indexes_set() for this purpose. OpenHPI project provides an extensive documentation to write a sub-agent this way. Now, on the downside, to implement a single table, I have 14 generated files whose code style is usually unfit to integrate into an existing project : some people, me included, do not like automatically generated code, all the more when it is so verbose. The solution would be to use the new API without mib2c; unfortunately, while the documentation exists, all provided examples rely exclusively on mib2c tool. Unless you are able to grasp the new API without it, I would restrict its use to two cases:
  • Writing a new module for inclusion into Net-SNMP. Since most modules are now written with the help of mib2c, there is no coding style problem.
  • Writing a standalone sub-agent for some MIB. As a standalone project, you won t run into coding style problems either.
For other uses, let s have a look at the traditional API.

C with traditional API The traditional API is way simpler and even if it is quite old, it is easier to find associated examples. Here is how to initialize the agent, again, in a stripped-down version (the complete version is on GitHub): 
static oid ethtool_oid[] =  1, 3, 6, 1, 4, 1, 39178, 100, 1 ;
static struct variable3 ethtool_vars[] =  
   1, ASN_COUNTER64, RONLY, ethtool_stat, 3,  1, 1, 2 
 ;
int main()
 
  netsnmp_enable_subagent();
  snmp_disable_log();
  snmp_enable_stderrlog();
  init_agent("ethtoolAgent");
  REGISTER_MIB("ethtoolStatTable", ethtool_vars,
               variable3, ethtool_oid);
  init_snmp("ethtoolAgent");
  while (1)
    agent_check_and_process(1);
The REGISTER_MIB() macro is used to register the provided root OID (ethtool_oid) to the master agent. It also registers handlers associated to various sub-OID through the ethtool_vars[] array. Each member of this array features the following fields:
  • magic is an integer allowing one to discriminate OID handled by the same handler. For a table, this allows to handle several columns with the same handler since they share a common index.
  • type is the type of the variable that will be returned.
  • acl tells if the object is read-only (RONLY) or read-write (RWRITE).
  • findVar is the handler associated to the object. Its task will be to find the appropriate object and return its value.
  • The two last fields are the relative OID this entry is responsible for along with its length.
The prototype of ethtool_stat() handler function is the following: 
static u_char*
ethtool_stat(struct variable *vp, oid *name, size_t *length,
             int exact, size_t *var_len, WriteMethod **write_method);
This function should return the value of the requested variable or NULL if no appropriate object was found. The value can be statically stored since it will be copied. Here is the description of the arguments:
  • struct variable *vp is the structure the handle was associated with, except that vp->name is now a full OID, not a relative one. You can access vp->magic to determine which column or which scalar to handle if this handler is common to several objects.
  • oid *name and size_t *length describe the OID requested. If you want to return another OID (for a GETNEXT request for example), you can modify them. name points to an OID array large enough to fit any OID you want to return.
  • int exact tells if the requested OID should be exactly matched (case of GET or SET) or not (GETNEXT).
  • size_t *var_len is used to tell the size of the returned object.
  • WriteMethod **write_method is only used if the variable is modifiable. When a SET request is received, the API will call the handler with exact set to 1 and expects to get a pointer to the function that will handle the write operation3.
To implement this function, you are on your own. When relying on externally fetched data, you need to handle a cache yourself. I have used a red-black tree for this purpose (with code stolen from OpenBSD). This also enables an easy implementation of the GETNEXT operation. Here are three projects using this API to provide SNMP support:
  • Keepalived is a VRRP daemon and a monitoring daemon for LVS clusters. I have added a complete SNMP support. The most interesting files are core/snmp.c, check/check_snmp.c and vrrp/vrrp_snmp.c. It features a tight integration into Keepalived event loop that should be reusable by other projects. It includes read-only support of all data structures (configuration, state and statistics), ability to send traps on events and a write support for some values.
  • lldpd is an implementation of 802.1AB (LLDP) which is a protocol allowing an equipment to advertise itself to its neighbors on Ethernet level. 802.1AB comes with a huge MIB module and many extensions. lldpd provides a fairly complete read-only implementation of this MIB module. Some tables have complex indexes. Have a look at agent.c. There is currently no write support and no traps but lldpd features privilege separation and some code allows it to still use an Unix socket to speak with the master agent.
  • Asterisk, an open source PBX, includes an AgentX sub-agent allowing one to grab some metrics. Look at res/snmp/agent.c.

Other solutions The manual page for snmpd.conf explores other ways to extend Net-SNMP agent under the section Extending agent functionality :
  • exec and sh are deprecated. Use extend instead.
  • pass is a bit simpler than pass_persist. The command is run once for each request. Such a command can easily be turned into a pass_persist script. Moreover, Net-SNMP will not answer any request while a pass command is running.
  • proxy redirects requests to another SNMP agent. Unless you have some code that can only act as a complete SNMP agent, it is better to use AgentX instead. For programs using Net-SNMP API, turning an agent into an AgentX sub-agent is just one line of code. Since a sub-agent involves less code than a complete agent, it is more efficient and more reliable.
  • smux has been superseeded by AgentX.
  • perl and dlmod allow one to load and execute external code inside the running agent. Unless you have important performance requirements, it is better to use a sub-agent instead: if some bug happens, only the sub-agent will be affected, not the master agent. A sub-agent sticks more closely to the Unix philosophy: write programs that do one thing and do it well . However, Net-SNMP FAQ says:
Implementing the module in C within the main agent (directly or via dlmod) is probably the most efficient and reliable, closely followed by embedded perl (or python) extensions. These have the advantage of minimal overheads between the code implementing the MIB module, and the agent framework, and no inter-process communication issues.
  1. Additional arguments for snmp* are usually required to specify a version and a community (for SNMPv1 and SNMPv2c) or a user (for SNMPv3). In all examples, I do not specify anything. I assume that ~/.snmp/snmp.conf contains the appropriate bits. Moreover, some outputs are truncated.
  2. No allocations, no variable declarations and no error checkings. Do not do this at home!
  3. If you want to look at an example of how to handle a SET request, look at the
    SNMP support for Keepalived. A SET operation is more complex because SNMP ensures that the
    SET operation is atomic across all provided variables. It uses a three-staged commit. The traditional API is very inefficient for this kind of operation because the object has to be located again for each step.

Michal &#268;iha&#345;: ColorHug with non English locales

Since infamous erasing of factory calibration in my ColorHug device and restoring calibration matrix, I noticed it did screen calibration wrong. However I did not find time to properly investigate the issue. Yesterdays mail from Richard was actually trigger for me so I've opened up this topic. In the end it turned out to be caused by Little CMS wrongly parsing CCMX in case you are using locales which use something else than . as decimal point. After lot of googling, I've realized there is probably no good way of parsing floats independent on current locales, so I used one of hacks I found and I think it's less intrusive - get current decimal point by printing float string using printf and then convert the string to it. I know it looks ugly, but including own implementation of strtod is also not nice and playing with locales is definitely something not thread safe to do within widely used library. Anyway I've asked upstream to merge my patches, so let's see what they think of it.

Filed under: Debian English Linux Photography Suse 8 comments Flattr this!

8 February 2012

C.J. Adams-Collier: SELinux on Wheezy

So, Collier Technologies LLC needs to pass annual audits to operate a certification authority recognized by the SoS. To this end, I m working with the fine group of developers who maintain SELinux. It seems that the configuration of Xorg that I m using while typing this here blog post does not have a policy set up for it in the Debian packages. Or if it does, I don t know enough about it to figure it out. I ve been keeping logs and publishing them here: http://www.colliertech.org/federal/nsa/ I ll update this post as progress is made. [edit 20120608T1042] It looks like loading all .pp files (except alsa) makes X run: 
cjac@foxtrot:/usr/share/selinux/default$ time sudo \
semodule -i  ls *.pp   grep -v -e 'base.pp' -e 'alsa.pp' 
real	0m24.148s
user	0m23.249s
sys	0m0.628s
I had to boot into single user mode and load the policies before switching to runlevel 2. To get the kernel args added to the grub command line, I modified /etc/default/grub to include this line: 
cjac@foxtrot:/usr/share/selinux/default$ grep -i selinux /etc/default/grub
GRUB_CMDLINE_LINUX=" selinux=1 security=selinux"
Next steps: [edit 20120208T1305] It looks like the seinfo package has not been updated in the last 18 months. 
cjac@foxtrot:/usr/src/git/debian/setools$ grep url .git/config
	url = git://git.debian.org/git/users/srivasta/debian/setools.git
cjac@foxtrot:/usr/src/git/debian/setools$ git log   head -4
commit 22a5d3e451d8a1e60a3c746466c865e63089a92a
Merge: fa238f0 149e283
Author: Manoj Srivastava <srivasta>
Date:   Tue Jul 20 23:10:06 2010 -0700
[edit 20120208T1346] Stephen tells me that the modules are persistent across re-boots. 
> What's the best way to do this at boot?
You just do it once and it remains until/unless you remove it with
semodule -r.  No need to do it on each boot.  Normally it is done when
you install the policy package, but since your policy package apparently
didn't install all modules, I'm suggesting that you do so manually.

Pietro Abate: Dependency order matters !

Recently I've discovered a subtle consequence of how the order in which dependencies are specified in debian actually matters. While re-factoring the code of dose3, I changed the order in which dependencies are considered by our sat solver (of edos-fame) . I witnessed a twofold performance loss just by randomizing how variables were presented to our sat solver. This highlights, on one hand, how our solver is strongly dependent on the structure of the problem and, on the other hand the standard practice of debian maintainers to assign an implicit priority in the disjunctive dependencies where the first is the most preferred packages (and maybe the most tested, at least dependency-wise). The basic idea of distcheck is to encode the dependencies information contained in a Packages file in CNF format and then to feed them to a sat solver to find out if a package has broken dependencies or if its dependencies are such that no matter what, it would be impossible to install this package on a user machine. Conflicts are encoded as binary clauses. So if package A conflicts with package B, I add a constraint they says "not (A and B)" , that is A and B cannot be considered together. The dependencies encoding associates to each disjunction of the depends field a clause that says "A implies B". If a package foo depends on A,B C,D , I'll add "foo implies A and B" & "foo implies C and D" . This encoding is pretty standard and it is easy to understand. The problem is how the sat solver will search for a solution to the problem "Is is possible to install package foo in an empty repository". The solver we use is very efficient and can easily deal with 100K packages or more. But in general is not very good at dealing with random CNF instances. The reason because edos-debcheck is so efficient lies in the way it exploits the structure of the sat problems. The goal of a sat solver is to find a model (that is a variable assignment list) that is compatible with the given set of constraints. So if my encoding of the debian repository is a set of constraints R, the installability problem boils down to add an additional constraint to R imposing that the variable associated to the package foo must be true, and then ask the solver to find a model to make this possible. This installation, in sat terms, would be just an array of variables that must be true in order to satisfy the given set of constraints. If you look at the logic problem as a truth table, the idea is to find a row in this table. This is the solution of your problem. Brute force of course is not an option and modern sat solvers use a number of strategies and heuristic to guide the search in the most intelligent way possible. Some of them try to learn from previous attempts, some of them, when they are lost try to pick a random variable to proceed. If we consider problems that have a lot of structure, award winning sat solver do not back track very much. By exploiting the structure of the problem, their algorithm allows them to considerably narrow down the search only to those variables that are really important to find a solution. All this long introduction was to talk about the solver that is currently used in edos-debcheck and distcheck (that is a rewrite of the edos-debcheck). So why dependency order matters ? If we consider any package, even if the policy does not specify any order in the dependencies, it's common practice to write disjunctive dependencies specifying the most probable and tested alternative first and all other, more esoteric choices later. Moreover real packages are considered *before* virtual packages. Since every developer seems be doing the same, some kind of structure might be hidden in the order in which dependencies are specified. Part of the efficiency of the the solver used in our tools is actually due to the fact that its search strategy is strongly dependent on the order in which literal are specified in each clause. Saying the package foo depends on A and B is "different" then saying it depends on B and A, even if semantically equivalent. In my tests, I found about a twofold performance loss if the order of literals is either randomized or inverted. This is clearly a specific problem related to our solvers, while other solvers might not be susceptible to such small structural changes. Sat competitions often employs some form of obfuscation strategies of well known problems with well known structures in order to make useless to encode a search strategy that exploits the specific structure of a problem. Since here we're not trying to win a sat competition, but to provide tool to solve a specific problem, we are of course very happy to exploit this structure.

4 February 2012

Paul Tagliamonte: Mapping the Ubuntu Community

In playing with some tools I ve run into at $work, I ve tried loading in some Ubuntu datasets in some fun and interesting ways. Today, I ve chosen to map all Ubuntu Members with a public lat/lon, sized by Karma. The sizes relate to if the Karma is greater then: 1: 10 2: 50 3: 100 4: 500 5: 1000 6: 2000 7: 7000 8: 15000 9: 25000 10: 50000 So, without further adieu, here re some maps! UK US EU EEU SA Globe

Christian Perrier: 2012 update 8 for Debian Installer localization

Status for D-I level 1 (core D-I files): Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.): Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader) Full 100% completeness (hall of fame) for 17 languages: Bulgarian, Czech, German, Esperanto, Spanish, Persian, French, Italian, Japanese, Kazakh, Dutch, <Polish, Portuguese,="" Russian,="" Slovak,="" Turkish="" Simplified="" Chinese="">

2 February 2012

Jamie McClelland: Servers4All... unless someone complains

On Wednesday, February 1, a new virtual server May First/People Link recently rented went offline. We contracted the virtual server through Server4All because we need their un-metered 100Mbit connection to help us handle the bandwidth for Sahara Reporters, one of the most important independent news sources for Saharan Africa. With the server offline, the web site was down as well. We scrambled to setup alternative caching servers to handle the bandwidth. When I logged into our control panel, I saw the message: This virtual server has been suspended by the administrator. Please contact support. I immediately contacted support and then received the message: 
Hello
We have received the following complaint associated with your server/service.
IP: 76.73.121.164
To prevent any further abuse we have suspended this service. In order to
resume, we request you to cooperate with our investigation as promptly as
possible. Please respond to us with the following details:
(1) What has caused the complaint
(2) What is the server used for. Purpose?
(3) How can you resolve the complaint and make sure it will not be repeated.
Depending on the nature of the complaint and your response, we will put back
the server online.  Please note, this has violated our Terms Of Service. We
expect your response within 24 hours, otherwise your account will be
terminated permanently.  Thank you
What?? What complaint?? I followed up but had to wait til the next day to get the response. 
Here is the full log,
An email advertizing the Domain Name: saharareporters.com
has been sent to the blacklist.woody.ch spamtrap.
This Domain does resolve to IP addresses one of which your are responsible:
76.73.121.164
Please investigate why this Domain has been advertized.
Attached you find the headers and reports in ARF for automatic processing.
Feedback is appreciated.
Actual listing periods:
Bounce: 1 Hour in DNS.
Whitelisted IP: Not lised in DNS.
Spam: 24 hours in DNS.
Every Hit: 14 days in evidence DB.
For any questions or Feedback, contact abuse@woody.ch
From: is intentionally set to a bit-bucket.
Kind regards
-Benoit Panizzon-
There is no attachment. I went to woody.ch and it was in German. Then tried blacklist.woody.ch, but no luck. Finally I found the Woody's World Blacklist Page. I plugged in our IP address into their checker and I got: 
Output from the Check, if empty the IP is not listed.
164.121.73.76.[name of the blacklist] being tested.
Host 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN)
Host 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN)
Host 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN)
Host 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN)
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
Host 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN)
Host 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN)
Host 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN)
Host 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN)
Host 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN)
Host 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN)
Host 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN)
164.121.73.76.blacklist.spambag.org has address 208.91.197.182
164.121.73.76.blacklist.spambag.org descriptive text "v=spf1 -all"
So, Woody's World thinks we are listed in spambag.org. I went to spambag.org and learned that the domain is for sale. I then tried blacklist.spambag.org and got the same page. This page has many links all pointing to advertisements. The "RBL List" link takes me to a page advertising "5 foods you must not eat." Next, out of curiousity, I tried Mxtoolbox. I got one hit from Barricuda. Barricuda says the reputation of the IP address is "poor". Why? According to Barricuda, the reasons could be: Keep in mind, this IP address is not used for sending email. It's just the web site. I then took a step back and re-read the complaint and noticed that it says that the domain name saharareporters.com was listed in a spam email. Hm. More concerted searching for the terms "woody spamtrap blacklist" and I found a pattern in URLs that suggested I plug in the following: http://news.scoutnet.org/rblhostlist.php?id=saharareporters.com.uri And sure enough, there was a result. In short, it was a classic Nigerian Oil scam in which the person claims to be "JAMES IBORI ex-governor of DELTA STATE oil city." The scammer acknowledges that he has been arrested, but promises lots of cash to the person who can help him. In an effort to boost their credibility, the scammer included a link to a Sahara Reporters article about the real James Ibori. And that, my friends, is enough to have one of the most prominent independent African news organization taken offline. But, Sahara Reporters shouldn't necessarily feel singled out. Sahara Reporters should feel singled out. They exist to illuminate news from Africa. They were taken offline because a series of individuals don't know the difference between a prominent independent African news service and a criminal scammer. To most of the Internet, Nigeria and email scams are synonymous. That has to change. Although Sahara Reporters is particularly vulnerable, any site hosted with Server4All can potentially be taken down. All you have to do is write a fake spam/scam email, including a link to the web site you want to be taken offline, and then send that email to: listme@blacklist.woody.ch. I'm currently following up with both Woody's World and Server4All. However, once this particular issue is resolved, we're left with a much bigger and ominous problem. If your hosting provider (or their upstream provider) takes your site offline when it receives a complaint first and then asks questions second, you have a big problem. All of our legal fights over our rights to keep content online are moot if our providers, without any legal pressure to do so, still take down our services based on spurious complaints.

31 January 2012

Andrea Veri: A few useful Puppet snippets

I ve been playing with Puppet lately both on my home network and within the Fedora s Infrastructure team and I thought some of the work I did might be useful for anyone out there being stuck with a Puppet s manifest or an ERB template. Snippet #1: Make sure the user foo is alwais created with its own home directory, password, shell, and full name. 
class users  
    users::add   "foo":
        username        => 'foo',
        comment         => 'Foo's Full Name',
        shell           => '/bin/bash',
        password_hash   => 'pwd_hash_as_you_can_see_in_/etc/shadow'
     
define users::add($username, $comment, $shell, $password_hash)  
    user   $username:
        ensure => 'present',
        home   => "/home/$ username ",
        comment => $comment,
        shell  => $shell,
        managehome => 'true',
        password => $password_hash,
Snippet #2: Make sure the user foo gets added into /etc/sudoers. 
class sudoers  
file   "/etc/sudoers":
      owner   => "root",
      group   => "root",
      mode    => "440",
      
 
augeas   "addfootosudoers":
  context => "/files/etc/sudoers",
  changes => [
    "set spec[user = 'foo']/user foo",
    "set spec[user = 'foo']/host_group/host ALL",
    "set spec[user = 'foo']/host_group/command ALL",
    "set spec[user = 'foo']/host_group/command/runas_user ALL",
  ],
Snippet #3: Make sure that openssh-server is: installed, running on Port 222 and accepting RSA authentications only. 
class openssh-server  
  package   "openssh-server":
      ensure => "installed",
   
    service   "ssh":
        ensure    => running,
        hasstatus => true,
        require   => Package["openssh-server"],
     
augeas   "sshd_config":
  context => "/files/etc/ssh/sshd_config",
    changes => [
    "set PermitRootLogin no",
    "set RSAAuthentication yes",
    "set PubkeyAuthentication yes",
    "set AuthorizedKeysFile	%h/.ssh/authorized_keys",
    "set PasswordAuthentication no",
    "set Port 222",
  ],
Snippet #4: Don t apply a specific IPTABLES rule if an host is tagged as staging in the relevant node file. On templates/iptables.erb: 
<% if environment == "production" %>
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
<% unless defined?(staging).nil? %>
-A INPUT -s X.X.X.X -j REJECT --reject-with icmp-host-prohibited
<% end -%>
# Allow unlimited traffic on eth0
-A INPUT -i eth0 -j ACCEPT
-A OUTPUT -o eth0 -j ACCEPT
# Allow unlimited traffic from trusted IP addresses
-A INPUT -s 192.168.1.1/24 -j ACCEPT
<% end -%>
On the manifest file: 
class iptables  
    package   iptables:
        ensure => installed;
     
    service   "iptables":
        ensure    => running,
        hasstatus => true,
        require   => Package["iptables"],
     
    file   "/etc/sysconfig/iptables":
        owner   => "root",
        group   => "root",
        mode    => 644,
        content => template("iptables/iptables.erb"),
        notify  => Service["iptables"],
That s all for now!

Rapha&#235;l Hertzog: Contributing to the translation of Debian

If you re not into packaging and if you asked how you could help Debian, someone probably suggested that you help to translate it. It s true that translating Debian is essential if we want to make Debian available to everybody on the world. There are many persons who are stuck as soon as they get a message in English, so it s important to aim for 100% coverage in terms of localization. Some vocabulary: localization vs internationalization Internationalization (i18n) is the work that makes it possible to translate messages in a given application. Localization (l10n) is the work of translating messages of said application. So as a translator, you ll be doing localization but some knowledge of internationalization is still useful because it will define how you re supposed to provide the translations. We ll come back to that later. Join your localization team Usually the translation work is shared among multiple translators within a localization team. Check out the Debian International page on www.debian.org to find out instructions for translators for each language. Many teams have a debian-l10n-*@lists.debian.org mailing list used for coordination, feel free to ask questions on those lists when you start (but make sure that you have read the relevant documentation before). Each team has its own workflow, so observe for a while to get used to what s happening before asking your first questions. What is there to translate? The translation of most of the software provided by Debian is not handled by Debian. The Debian translation teams only handle the translation of: Now before contributing to your first translation, I have to come back to internationalization to teach you a few things. In the above list, the projects marked with (*) do use PO files for their translation and the next sections will explain you how to work with those files. Introduction to Gettext The free software community has mostly standardized on a single internationalization infrastructure known as Gettext. With this tool, you re provided a POT file which contains all the translatable strings. It looks like this: 
# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR Software in the Public Interest, Inc.
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <email>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: dpkg 1.16.1\n"
"Report-Msgid-Bugs-To: debian-dpkg@lists.debian.org\n"
"POT-Creation-Date: 2011-09-23 03:37+0200\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <email>\n"
"Language-Team: LANGUAGE <ll>\n"
"Language: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=INTEGER; plural=EXPRESSION;\n"
#: lib/dpkg/ar.c:66
#, c-format
msgid "invalid character '%c' in archive '%.250s' member '%.16s' size"
msgstr ""
#: lib/dpkg/ar.c:81 lib/dpkg/ar.c:97 lib/dpkg/ar.c:108 lib/dpkg/ar.c:112
#: lib/dpkg/ar.c:134 utils/update-alternatives.c:1154
#, c-format
msgid "unable to write file '%s'"
msgstr ""
[ ]
The lines starting with #: are comments that indicate the source files where the (English) string is used. This can be useful if you want check the source to have more information about how the string is used. The lines starting with #, contain flags that can be important. If the fuzzy flag is set, the translated string is not used because it must be updated (or at least verified) since the original string evolved. The c-format flags indicates that the string must be a C format string, this has some implications in what s allowed in the string (in particular when it embeds conversion specifier for arguments submitted to printf-like functions). Another thing to note is that the translation of the empty string is used to store some meta-information about the translation itself. Contributing a translation as a PO file When you start a new translation, you copy that POT file to create a PO file for your own language (eg. fr.po for the French language). You replace some template values (identified with the upper case words in the POT file) and you replace all the empty strings on msgstr lines with the translation of the string that appears in the previous msgid line. The result could be something like this: 
# translation of fr.po to French
# Messages fran ais pour dpkg (Linux-GNU Debian).
msgid ""
msgstr ""
"Project-Id-Version: fr\n"
"Report-Msgid-Bugs-To: debian-dpkg@lists.debian.org\n"
"POT-Creation-Date: 2011-09-23 03:37+0200\n"
"PO-Revision-Date: 2012-01-16 07:57+0100\n"
"Last-Translator: Christian Perrier <bubulle>\n"
"Language-Team: French <debian-l10n-french>\n"
"Language: fr\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: Plural-Forms: nplurals=2; plural=n>1;\n"
"X-Generator: Lokalize 1.2\n"
#: lib/dpkg/ar.c:66
#, c-format
msgid "invalid character '%c' in archive '%.250s' member '%.16s' size"
msgstr "caract re invalide  %1$c  dans la taille du membre  %3$.16s  de l'archive  %2$.250s "
#: lib/dpkg/ar.c:81 lib/dpkg/ar.c:97 lib/dpkg/ar.c:108 lib/dpkg/ar.c:112
#: lib/dpkg/ar.c:134 utils/update-alternatives.c:1154
#, c-format
msgid "unable to write file '%s'"
msgstr "impossible d' crire le fichier  %s "
[ ]
If there s already a PO file for your language, there might still work to do: there might be strings that have not yet been translated and there might be fuzzy strings which have to be updated strings which were already translated but where the original string has been modified. There are software that can assist you to edit PO files: poedit, virtaal, lokalize, gtranslator. There are also special extensions for vim (packaged in vim-scripts) and for Emacs. Submit the translation for inclusion Once you have a complete PO file, you should submit it for inclusion. Sometimes you will have been granted commit rights to the source code repository so that you can include your translation by yourself. In the other cases, you should submit your translation with a bug report tagged l10n and someone else will include your work in the next release. Depending on the team, the workflow might require a review before the submission. In that case, you usually have to send a call for review on the coordination mailing list. Go ahead! Hopefully those explanations will be enough to get you started. There are many other things to learn but it s good to learn while practicing For example, can you find out why the French translation above changed %c in %1$c ?
Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

No comment Liked this article? Click here. My blog is Flattr-enabled.

28 January 2012

Bartosz Fe&#324;ski: dibbler 0.8.1

Can t believe its almost 7 years since my first sponsored upload of Dibbler to Debian archive. I did it cause upstream promised to maintain it and fix any bugs submitted by Debian users. Unfortunately after 3-4 years he stopped to do it. Well since I was the sponsor of it, then I should feel responsible for these packages after him. It s quite hard to do it, cause I m not using these packages at all ;) Anyway, I finally found some time and reviewed them. It wasn t easy cause in the meantime whole build process totally changed. To whom it may concern: Dibbler is an IPv6 DHCP client, relay and server. I ve just uploaded the newest upstream version. Changelog follows: dibbler (0.8.1-1) unstable; urgency=low * ACK previous NMUes thanks!
* New upstream version (Closes: #629685)
uses correct example adsress pool in examples (Closes: #544323)
corrects scope of stateless in manpage (Closes: #615165)
* Fix pending l10n issues. Debconf translations:
Danish courtesy of Joe Hansen (Closes: #597767)
Dutch courtesy of Jeroen Schot (Closes: #632628)
* Fixes handling of children processes using resolvconf (Closes: #627317)
* Doesn t conflict with other resolver configuration daemons (Closes: #627786)
* Correctly handles multiselect values from debconf (Closes: #629681)
* Debianized almost from scratch, uses new source format.
* New Standards-Version (no changes needed). Bartosz Fenski <fenio> Fri, 27 Jan 2012 14:37:13 +0100 Yep, it fixes 8 bugs, and brings the newest upstream version with huge number of new features to Debian. Enjoy!

Russ Allbery: Review: Effective Java, Second Edition

Review: Effective Java, Second Edition, by Joshua Bloch
Publisher: Addison-Wesley
Copyright: 2008
Printing: October 2009
ISBN: 0-321-35668-3
Format: Trade paperback
Pages: 319
Guy Steele, in the forward to this book, writes:
Programs, unlike spoken sentences and unlike most books and magazines, are likely to be changed over time. It's typically not enough to produce code that operates effectively and is readily understood by other persons; one must also organize the code so that it is easy to modify. There may be ten ways to write code for some task T. Of those ten ways, seven will be awkward, inefficient, or puzzling. Of the other three, which is most likely to be similar to the code needed for the task T in next year's software release?
Teaching this, as well as how to avoid being awkward, inefficient, or puzzling, is the goal of this book. Clearly communicating effective, idiomatic, and maintainable usage to a newcomer to a programming language is one of the hardest types of programming books to write. Books like this are therefore quite scarce. Most introductory texts do try to communicate some degree of basic usage, but they rarely go far beyond the syntax, and when they do that usage is rarely both well-defended and inobvious. Bloch takes the concept quite far indeed, going deep not only into the Java language but also into object-oriented software construction in general. Effective Java is modeled after Effective C++ by Scott Meyers, a book I've not read (due to the lack of need for C++ in my programming life), but which I've heard a great deal about. This means the book is organized into 78 numbered items, each of which provides specific advice and analysis about one area of Java. Examples include item 16, "Favor composition over inheritance," or item 33, "Use EnumMap instead of ordinal indexing." As you can see, they run the gamut from high-level design principles to specific coding techniques. This sort of book demands a lot of the author. Everyone has a coding style, and everyone can make usage recommendations, but the merits or lack thereof of specific recommendations are often only visible with substantial later experience. More than any other type of programming language book, this sort of usage guide must be written by a language expert with years of experience with both good and bad code in the language. This is where Effective Java shines. Joshua Bloch led the design and implementation of significant portions of the Java core libraries at Sun and is currently the chief Java architect at Google, but even without knowing that background, his expertise is obvious. Every item in this book is backed up with specific examples and justification, and Bloch quotes extensively from the Java core library to illustrate both the advantages of the techniques he describes and the problems that result when they're not followed. This is not an introductory book, which is one of the things that makes it so efficient and concise. It's a book aimed at the intermediate or advanced Java programmer and assumes you already know the language and the basic pitfalls. There are only a few items in here that would be obvious to most experienced programmers, and even there Bloch ties them back to specific issues in Java in ways that are illuminating. I would not have expected to learn something new from a chapter on a hoary old problem like avoiding float and double for precise values, but I did: Bloch discusses the available alternatives within Java and their tradeoffs and then makes useful specific recommendations. If, like me, you're an experienced programmer already but relatively new to Java, you still should not read this book first. You need a general introduction to the language and libraries and a few projects under your belt before you can appreciate it. (I personally started with Thinking in Java by Bruce Eckel and it served me well, although on several points of style Bloch disagrees with advice in Eckel's book, and I find Bloch's arguments convincing.) But I think this is one of the best possible choices for your second book on Java, in large part because Bloch will head off bad design and style decisions that you don't realize you're making and catch them before they become entrenched. I'm glad I read it as soon as I knew the language well enough to absorb it, and it's the sort of book that I'm likely to re-read sections of whenever I work on Java code related to those topics. It's not entirely obvious that you should take my advice about this sort of book, since I'm not a Java expert and don't have those years of experience with it. But I've checked the recommendation with other programmers I know who are experts, and I've never heard anything but praise for it. It's also one of the books recommended in Coders at Work, and Bloch is one of the people interviewed there, which carries a lot of weight with me. And, apart from that, any long-time programmer who cares about their craft builds an internal sense of aesthetics around what a well-written program should look like and finds themselves recognizing a similar sense in other people's code, even in languages with which they're not familiar. Bloch's recommendations and analysis feel right; one can immediately see how they improve maintainability and robustness, and some of the techniques he shows are elegant and beautiful. This is not a general programming book. It's specifically focused on the Java language, and much of it deals with specific suggestions on how to use Java's core libraries and language features. If you're not going to be writing code in Java, I can't really recommend it. But one of the things I loved about it is that, while talking about Java, Bloch also talks about object-oriented software construction: techniques for extending foreign libraries one does not control, API design, and the proper use of inheritance, among other topics.. That advice is some of the best object-oriented software design advice I've ever read. There isn't enough of it to recommend the book to people with no interest in Java, but this book has even made my C and Perl code better, and has helped me grasp the tradeoff between inheritance and composition in a deeper way than I ever had before. It's a lovely side bonus. If you're writing Java, read this book. If you're learning Java, don't read it first, but read it second, and more than once, or alongside a project where you can apply the advice. It's dense and efficient in the information that it conveys, which means there's more in a couple pages here than in thirty or forty pages of some of the sprawling introductory programming books. I did read it cover to cover, which is one of the better ways to get a sense of Bloch's more general advice on software construction, but you'll hit information overload and will want to return to it piecemeal to fully absorb it. And do get the second edition. I'm sure the first edition is available cheap used, but the additions of enums and generics to the Java language are hugely important and provide some of the most elegant and graceful techniques in the book. Rating: 9 out of 10

Christian Perrier: 2012 update 7 for Debian Installer localization

Status for D-I level 1 (core D-I files): Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.): Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader) Full 100% completeness (hall of fame) for 17 languages: Bulgarian, Czech, German, Esperanto, Spanish, Persian, French, Italian, Japanese, Kazakh, Dutch, <polish>, Portuguese, Russian, Slovak, Turkish Simplified Chinese

27 January 2012

Rapha&#235;l Hertzog: People Behind Debian: Josselin Mouette, founder of the Debian GNOME team

Josselin Mouette is one the leaders of the pkg-gnome team, he takes sound technical decisions and doesn t fear writing code to work-around upstream issues. He deserves kudos for the work he has put into packaging GNOME over the years. He can also be very sarcastic (sometimes he even enjoys participating to flamewars on debian lists), and there are quite a few topics where we have long agreed to disagree. But this kind of diversity is also what makes Debian a so interesting place Read on to learn more about the pkg-gnome team, its plans for Wheezy, Josselin s opinion on the GNOME 3 switch, and much more. Raphael: Who are you? Josselin: I am a 31 years old Linux systems engineer. I started in life with physics, which I studied at the ENS Lyon. I started a thesis on experimental and numerical models for optoelectronics, but when it became clear that research was not for me, I abandoned it and accepted a job at the CEA, which holds the largest computing center in Europe. Working on these machines has been the most awesome job ever (except for it being near Paris). After that I worked a bit on system monitoring technologies. I am married, currently living in Lyon, and working for EDF (the French historical electricity company) on scientific workstations using Debian. EDF is using Debian on more than a thousand workstations and holds the fastest Debian supercomputer in the world (200 Tflops), which makes it another obvious place for Debian developers. Raphael: How did you start contributing to Debian? Josselin: I discovered Debian in 1999 while studying at the ENS, which is one of the biggest nests of Debian developers while being a small place, it is producing almost one Debian developer per year on average. After wondering for a while what it could be useful for, hacking on a slink snapshot made me think that it was for, well, everything except for gaming. Later, in 2002, when I was working on optoelectronics computing codes, I started to package them for Debian in order to make them easier to install, for us as well as other labs over the world. I started the NM process, and it was going smoothly but also going to take time. However, at that moment, the frozen-bubble game went out and made quite some buzz. Since I knew a guy who knew the game s developer, he asked me to package it. The package found 3 sponsors in a very short time and was fast-tracked into the archive at a speed that was unseen before. After which the NM process was completed very quickly. At that time, I was a heavy WindowMaker user, but I didn t like the direction the project was taking (actually, I wonder if there was one). GNOME was starting to become attractive, but its packaging in Debian was very ineffective, with many inconsistent packages maintained by people who didn t ever talk to each other some of them didn t speak English, and some of them didn t talk at all. Together with awesome people, among which Jordi Mallach, Gustavo Noronha Silva, JHM Dassen, Ross Burton and S bastien Bacher, we started the GNOME team in 2003, introducing consistent packaging practices, and initiating synchronized uploads. Releasing a completely integrated GNOME 2.8 in sarge was a considerable achievement; proving (together with the Perl team) that a team was the best way to maintain large package sets changed the way people work on Debian.
Proving [ ] that a team was the best way to maintain large package sets changed the way people work on Debian.
Raphael: You re one of the most active contributors of the team which is packaging GNOME for Debian. What would you suggest to a new contributor who would like to help the team? Josselin: There are several ways to contact the team, but the recommended one has always been IRC. We hang on #debian-gnome on the OFTC network, so just come around and ask for us. The real question is what you want to do in the team. Of course, most new volunteers want to help packaging the latest and greatest version of GNOME into unstable as soon as possible, but unless they already have Debian background, this is not the easiest task. Since there are already people working on this, the big packages are usually waiting on dependencies. I used to direct newcomers towards bug triage, but it is a tedious task and I m now convinced that our huge bug backlog will never be dealt with. The most useful thing to do for newcomers now is probably to find a GNOME or GNOME-related package that needs improvement or is lagging behind, and simply try to work on it. You can also come and fix the bugs you find annoying. Find a patch on the GNOME bugzilla, or cook it yourself, propose it, and if it s worthy enough you ll soon get commit access.
Our huge bug backlog will never be dealt with.
At this point I feel worth mentioning that if no one answers in 10 minutes, it doesn t mean that no one will answer in 2 hours, so please stay on the channel after asking. Raphael: There s been some controversy about GNOME 3 and the direction that the project is taking. What s your personal stance on GNOME 3? And what s the position of the pkg-gnome team? Josselin: The controversy is not new to GNOME 3, but the large-scale changes made with it have put it more prominently. The criticism usually boils down to a few categories:
  1. General lack of configurability
  2. Strange design decisions
  3. Red Hat centric development
  4. Hardware requirements
  5. Change resistance
The lack of configuration options has been an ongoing criticism since GNOME 2.0 has decided to rip off most of them. Of course, when the control center was redesigned again for 3.0, there was a surge of horrified exclamations from people who missed their favorite buttons. On this topic, I fully concur with GNOME developers. The configuration option that is useful for you is not necessarily useful for someone else. Of course, sometimes developers go a bit too far, but the general direction is right. At work, we found that only a minority of users actually configure anything on their desktops: they just want something that works to launch their applications. Apple and Google have sold millions of devices by making them the simplest possible and without any configuration. Design decisions are, on the contrary, individual decisions, and each of them, while having reasons behind it, can be questioned. I remember seeing a lot of complaints when the OK and Cancel buttons were reversed in dialog boxes, something that nobody questions anymore. GNOME Shell is full of such changes; some are easy to get accustomed with, some others just make eyebrows raise. The most obvious example is the user menu in GNOME 3.2, which contains an entry to configure your Google account, but no entry to shutdown the computer. Both decisions were taken independently, each of them with (good or bad) reasons, but the result is simply ridiculous. The default configuration in Debian will contain an extension to make it a bit better, but on the whole we don t intend to diverge from the upstream design, on which a lot of good work has been done.
On the whole we don t intend to diverge from the upstream design, on which a lot of good work has been done.
Point 3 is more complex. Red Hat being the company spending the most on GNOME, it is obvious that their employees work on making things work for their distribution. An example is the recurring discussions about relying on system services that are currently only implemented by systemd. Since there is a lot of (mostly unjustified) resistance against systemd in Debian, and since it won t work on kFreeBSD anyway, someone needs to develop an alternative implementation of these services for upstart and sysvinit. Everything is in place for someone else to do the job but it has to be done, and this can be frustrating. Especially since it can also be hard to integrate changes needed for other distributions . Hardware requirements are mostly a consequence of the previous criticism: there s hardware that most distributions just don t want to bother supporting. We ve seen it in squeeze with the introduction of a hard dependency on PulseAudio. The Debian GNOME team (together with the Gentoo maintainers) made this dependency optional, carrying heavy patches, in order to cover the cases where it does not work. Now that it has gained more maturity, making this effort obsolete, the new tendency is to require 3D acceleration. For various reasons, it is not available to everyone . On this matter, the position of the Debian GNOME team has always been to support as much different configurations as possible with reasonable effort. Thanks to efforts from the incredible Vincent Untz, upstream supports a so-called fallback mode , which is the GNOME panel from 2.x with a lot of its bugs fixed. We intend to support this mode for as long as reasonably possible in Debian, possibly even after upstream ends up dropping it. However, other applications are going to require 3D because GStreamer is moving to clutter too, affecting video playback performance on non-accelerated systems . For epiphany this is not a problem; only embedded video will be affected. But for totem, this is a major issue; because of that we will probably keep totem 3.0 in wheezy. Finally, there is a natural human tendency to dislike change (I have it too), and it applies a lot to desktop users habits. Needless to say a change of such a scale as introducing GNOME Shell can trigger reactions. However, I don t think it is reasonable, because of this resistance, to keep gnome-panel 2.x in Debian. This would be a lot of work on obsolete technology, and would prevent the upcoming removal of a lot of deprecated libraries. This time is much better spent improving gnome-panel 3.x in Debian and keeping the fallback mode great. One of the change that was made in Debian was to make it easier to find, being available as GNOME Classic directly from the login manager, instead of having to find it in an obscure configuration panel. In all cases, I would recommend to actually try GNOME Shell for a few hours before ditching it. I had never been accustomed to a new environment as quickly ever before.
In all cases, I would recommend to actually try GNOME Shell for a few hours before ditching it.
Having seen several of my GDM patches reverted without a warning, I know we are not finished with carrying patches in Debian packages.
Scientific workstations are a non-trivial example, since there is a measurable effect of using 3D in the window manager on heavy 3D applications.
On the other hand, on accelerated systems, this feature should end up improving performance a lot. Raphael: What are your plans for Debian Wheezy? Josselin: The first goal of the GNOME team is, of course, to provide again a great desktop environment to work on. For wheezy it will probably be based on GNOME 3.4. There also needs to be some work on package management interfaces. Upstream bases everything on PackageKit, but it is not as featureful as the aptdaemon Ubuntu technology. If I have time, I would also like to improve HTTP proxy support, since currently it is based on a stack of terrible hacks. Raphael: If you could spend all your time on Debian, what would you work on? Josselin: Obviously I would like to make GNOME in Debian even better. That would imply working on underneath dependencies (what we now like to call plumbing) to make sure everything is working great. This would also imply working more as GNOME upstream to make it more suitable for our needs. I would also work on large-scale improvements on the distribution, like conditional recommends which I d love to see implemented , or automatic build-dependency generation. I would also work on the installer to make it better for desktops machines. The idea is to automatically install language packs, or glues between two packages when both packages are installed. Raphael: What s the biggest problem of Debian? Josselin: The obvious answer is the same as the one most people you interviewed before gave: not enough members in core teams. A lot of developers join Debian to work on a small number of pet packages, and don t necessarily want to be involved with existing teams. It is probably still not obvious enough that the primary way to start contributing to Debian is to join an existing team. But if there is one thing that is preventing Debian from gaining more momentum now, it is a completely different one: the too short support timeframe. 3 years is really not enough for corporate users. One year to migrate from one version to another is too short, and it is not possible to skip a release. It is definitely possible to change that with reasonable effort: the long-term support after 3 years doesn t have to cover the same perimeter as the short-term one. For example, we could upgrade the kernel to the version in the current stable release, and stop fixing all non-remote security holes. The important thing is to cover the most basic needs: companies are ready to take the risk of having less support if it allows skipping a version, but not the risk of having no support at all. And even more important is to say that you do something. Red Hat says they support a release for 10 years, but of course after 5 years the supported perimeter is extremely small.
3 years [of support] is really not enough for corporate users.
Long-term support will not magically fix all problems in Debian, but it will bring more corporate users into the picture. And with corporate users come paid Debian developers, who can work on critical pieces of the system. Debian was built on the synergy between individuals and companies, and in recent years perhaps as a reaction against what happened with Ubuntu we ve kind of forgot the latter. A lot of individuals have joined the project, and they are actively working, for example, on shortening the release cycle, which goes against the interest of professionals. We should embrace again such users and developers, and that means adapting to the current needs of larger entities. Raphael: You re the maintainer of python-support, a packaging helper that was competing with python-central. Both helpers are now deprecated in favor of dh_python2. Does this mean that the situation of Python in Debian is now sane? Or are there remaining problems? Josselin: dh_python2 (and the Python3 version, dh_python3) has a sane enough design. It fixes a lot of issues in python-central and also python-support, at the expense of somehow reduced functionality for developers. However, just like the previous tools, it merely works around design mistakes in the Python interpreter. For example it is not possible to split binary modules, pure-Python modules and byte-compiled modules in different directory trees, like Perl does although PEP 3147 introduces a way to do so. There is still no sane and standardized way to deal with module versions. There is no difference made between the module (which is a part of language semantics) and the file containing it (an information which depends on the implementation). Developers heavily rely on introspection features and make assumptions based on the implementation, that make it impossible to work around problems with module files. Such problems are not restricted to Python. Those who fought against Ruby gems could tell even worse stories. While introducing GObject introspection packages in Debian (they can be used in JavaScript and Python to provide modules based on GObject libraries), I was pleased to see a clear distinction between file and module, but I was again struck by the fact you are not forced to declare API versions in your Python/JS code. In all cases, there is no reliable way to detect runtime dependencies in a given Python or JavaScript file, which leaves the maintainer to declare them by hand, and of course, often be wrong about them. Add to that the fact that most errors cannot be detected before runtime. For all these reasons, and while still being fond of Python for scripts and prototyping, I ve become really skeptical of using purely interpreted languages to write real applications. Some GNOME developers are moving away from Python and JavaScript, mostly towards Vala; I can only approve of that move and hope the same happens to other projects. Raphael: Is there someone in Debian that you admire for their contributions? Of course there is the never-sleeping, never-stopping, Michael Biebl who can upload a whole GNOME release in a single week-end. But there are a lot of awesome people who make Debian something that simply works. I could talk about Cyril Brulebois from the X strike force, Julien Cristau from the release team, Sjoerd Simons for his sound advice and work on plumbing, Luca Falavigna who is so fast at processing NEW, to quote only a few of those I work with frequently. And of course, Jordi and Sam for their humor.
Thank you to Josselin for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Note that you can find older interviews on http://wiki.debian.org/PeopleBehindDebian.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook .

8 comments Liked this article? Click here. My blog is Flattr-enabled.

26 January 2012

Russ Allbery: Do symbols files make sense for C++?

I also posted this to debian-devel, which is where the discussion will be, but I wanted to post it on my journal as well so that others who may be interested will see it on Planet Debian. For those not familiar with Debian, this concerns the implications of a new feature for how Debian tracks shared library dependencies. I'm currently working on the Policy modification to document (and recommend) use of symbols instead of shlibs, but I'd only personally used symbols with C libraries. Today I decided that I should try adding a symbols file to a C++ library, particularly if I'm going to recommend everyone do it. I tried this exercise with xml-security-c, which is, I think, a reasonably typical C++ library. Not the sort of core C++ library that would sit at the center of the distribution, but a random software package that's in Debian because other things use it. The experience was rather interesting, and I ended up uploading the new version without a symbols file and continuing to just use shlibs. That's for the following reasons:
  1. The generated symbols file was HUGE. Hundreds of lines. This is a marked difference from the typical C symbols file, which is of quite manageable size. Some of that is that the library provides a lot of different classes, but some of it is that C++ just generates a lot of exported symbols. There's no way that I could do what I would do with a C library and understand those symbols, why they're there, and whether they are likely to have changed between revisions.
  2. Generating a reasonable symbols file was a pain. Generating an unreasonable symbols file that just contains all of the mangled symbols is largely mechanical and uninteresting, but that symbols file doesn't seem to me to convey useful information. So I did some scripting to translate the symbols back with c++filt, and add (c++) tags, and then try to understand what I was looking at and figure out whether I should sort the symbols list because the default sort is by mangled name, which is meaningless. This is a rather unappealing process. It's not particularly difficult, but it's very awkward and feels like it's missing vital tools.
  3. The resulting symbols file is incomprehensible to someone without strong knowledge of C++. It's full of opaque entries that don't make sense to the non-C++ programmer, which I suspect is a substantial number of people who package C++ libraries for Debian. I know enough C++ from school that I can evaluate security fixes, make simple patches, and review upstream changes, and I think that's all that should be needed to package things for Debian. But I'm deeply uncomfortable producing a symbols file on my own that contains entries for things that I know nothing about and cannot evaluate when they've last changed, like "non-virtual thunk to FooClass::~FooClass@Base".
  4. Once I had a symbols file that resulted in a successful build and that I could have uploaded, I started thinking about how I was going to maintain it. With a C program, I would change the symbols file versions when the underlying function implementation changes in a way that may not offer eqiuvalence, similar to bumping shlibs. I realized that I was going to have no idea when that happened, and the only way that I would maintain the symbols file would be to either trust upstream to maintain ABI equivalence and therefore only change the symbols file when upstream changes the SONAME, or not trust upstream to maintain ABI equivalence and therefore change all the versions with each new upstream release. That gives me exactly the same semantics as a shlibs file, so what's the point in having a symbols file?
  5. The exported symbols of the library contained many symbols that obviously weren't really from that library, but instead were artifacts of the C++ compilation process, things like instantiations of std::vector. Do those go into the symbols file? Do they change from architecture to architecture? If they disappear again, is that actually an ABI break? How do I know? It's all very mysterious, and while shlibs provides the same semantics as just ignoring this, at least I'm not then including in the package data, generated by me, things that I'm just blindly ignoring.
I came away from this experience thinking that I should revise the Policy amendment to say that symbols files are really for C libraries and for C++ libraries with either a tightly maintained symbol export list or maintained by a C++ expert, and that most C++ library maintainers should just not bother with this and use shlibs, bumping the shlibs version or not based on their impression of how good upstream is at maintaining ABI equivalence. But that feels like a result contrary to what I had previously thought was the intended direction, so I wanted to ask the Debian development community as a whole: am I missing something? Are these symbols files actually useful? Am I missing some trick to make them useful?

25 January 2012

Pietro Abate: new thinkpad x220

This summer my beloved thinkpad x301 died in a cloud of smoke. It was exactly 3 years and 20 days old while my warranty was valid only for 3 years. Now, don't tell me this is a coincidence. Anyway. After about 5 months, I finally managed to convince my employer to get me a new thinkpad, the x220. My specs includes a 128G SSD , 4G of RAM, 2.4Gz processor, camera and fingerprint reader. It's a pity that the x300 series is not in production anymore. They were light, with a solid battery and a large screen. The X1 just don't cut it. Even though it can be consider the successor of the x300, its battery life is just not enough for my needs. On the other hand, the x220 is a very nice machine : the screen is a bit smaller then the X300, but it is light, with a very powerful processor, good battery and it feels very solid. In my opinion lenovo should have packed the new hw of the x220 in the chassis of the x300, maybe with small compromise on the battery life (I got the big battery and I can squeeze almost 7 hs with a single charge) but clearly this was not a good business choice... Installing debian on this laptop is not immediate because none of the official debian installers are shipped with a recent kernel (as in 3.x series). Since with the official debian installer I cannot have neither the driver for the Ethernet card or the driver for the wirelles card, I opted to use a custom installer built by kmuto (http://kmuto.jp/debian/d-i/ ) . Using this installer the ethernet card is recognized immediately and it's easy to proceed with the installation as usual. Another option would have been to add the binary blog for the wireless chip, but apparently the deb installer supports only WEP auth, while all my access point are WPA. I didn't spend too much time on the wireless setup, so it might well be that is indeed possible to install using a WPA access point. Last time I installed a laptop, I used the automatic partition option to have lvm on top of a lucks encrypted partition, only to discover later that the default dimensions of the partitions were a bit too small for me. For example, by default the boot partition is only 256Mb. This is plenty if you want to have only one kernel image installed at each given time, but if you want more then one kernel, memtest and for example a grml rescue iso, it's easy to run out of space. So I proceed to manually partition the disc creating a boot partition of 512M, and using the rest as a luks encrypted device with lvm on top and 3 logical volumes : sys (15G), swap (4G) and home (the rest). For my daily use having 15G on my laptop for /usr, /var, etc should more the enough... Next step was to install the system. Since in recent times I got extremely pissed off with gnome 3, I've decided to dump it completely and go back to awesome. But since awesome all by itself is a bit sad, I paired it up with xfce. Everything works, except the automount, and I'm still trying to figure out how to make it work. Apparently is a consolkit problem... I'll write another post about the xfce4 + awsome setup soon... Today I've also started playing with the finger print reader. It seems working, but I haven't managed yet to use it in conjunction with pam for authentication ... more to come. And... On last closing remark : during the last 5 months I've used a dell latitude e6410 ... Gosh. I feel I'm on anther planet. The keyboard of a thinkpad give you pleasure, not pain, from 2 to 4G of RAM is a big jump and from a conventional HD to a SSD ... well... it seems I'm flyinggggg :) I've the impression my productivity just went up 50% !!! If you work with your laptop everyday get a good laptop. It is well worth the investment ...

24 January 2012

Julian Andres Klode: Managing system package selections using custom meta packages

Over the last years, I have developed a variety of metapackages for managing the package selections of the systems I administrate. The meta packages are organized like this:
jak-standard
Standard packages for all systems
jak-desktop
Standard packages for all desktop systems (GNOME 3 if possible, otherwise GNOME 2)
jak-printing
Print support
jak-devel
Development packages
jak-machine-<X>
The meta package defining the computer X
Each computer has a jak-machine-X package installed. This package is marked as manually installed, all other packages are marked as automatically installed. The machine packages have the attribute XB-Important: yes set in debian/control. This creates an Important: yes field. This field is not official, but APT recognizes it and does not remove those packages (the same field is set for the APT package by APT when building the cache, as APT should not be removed either by APT). It seems to work a bit like Essential, with the exception that non-installed packages are not installed automatically on dist-upgrade. The meta packages are created using seed files similar to Ubuntu. In contrast to Ubuntu, I m not using germinate to create the packages from the seeds, but a custom dh_germinate_lite that simply takes a seed file and creates the correct substvars. It s faster than germinate and really simplistic. It also does not handle Recommends currently. The whole result can be seen on http://anonscm.debian.org/gitweb/?p=users/jak/jak-meta.git. Maybe that s useful for some people. And if you happen to find some packages in the seeds that are deprecated, please let me know. Oh, and yes, some packages (such as the letterman one) are internal software not publically available yet [letterman is a simple GUI for creating letters using LaTeX]. While I m at it, I also built Ubuntu s version of wine1.2 for i386 squeeze. It can be found in
deb http://people.debian.org/~jak/debian/ squeeze main (it still needs a few changes to be correct though, I ll upload a jak2 build soon). I also built updated sun-java6 packages for my parents (mostly needed due to the plugin, some websites do not work with the IcedTea one), but can t share the binaries due to licensing requirements. I may push out a source repository, though, so others can build those packages themselves. I ll let you know once that s done.
Filed under: Debian, General

23 January 2012

Christian Perrier: 2012 update 6 for Debian Installer localization

When you shake the tree, things happen..:-). I sent targeted update requests for level 2 (mostly espeakup and grub2). Status for D-I level 1 (core D-I files): Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.): Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader) Full 100% completeness (hall of fame) for 14 languages: Bulgarian, Czech, German, Esperanto, Spanish, French, Italian, Japanese, Kazakh, Dutch, Portuguese, Russian, Slovak, Simplified Chinese

Next.

Previous.