Search Results: "xam"

12 September 2015

Michal Čihař: Weblate for translating everything

Weblate is not only useful for translating software, it can help in translating any content. Let's look where our users are using it. Software translation is the most usual use case. This is actually where Weblate was used for first time and still provides great support for that. As an example (and oldest project hosted in Weblate) you can look at phpMyAdmin, where Weblate also helps to keep in sync translation for different maintenance branches. It can also help you in using same terminology in command line utility and graphical one like it is done in Gammu and Wammu translations. Weblate also supports localization of mobile application. Android support was originally implemented for OsmAnd project, but now they also translate iOS application using Weblate. Another use case is for translating documentation. You need to have some infrastructure to do, but there are several free software project to help here as well. For phpMyAdmin documentation or documentation for python module Odorik the Sphinx documentation generator is used, which has support for translation of the documentation. I've recently published quick howto for this setup. Another great tool for translating documentation is Publican, it is Docbook based tool and is used for translating The Debian Administrator's Handbook. Last but not least, you can use Weblate to translate your website. For example website for Weblate itself is translated this way. Did you miss your use case for Weblate? Please let me know. PS: I've heard that one Linux distribution will start using Weblate soon.

Filed under: English phpMyAdmin SUSE Weblate 0 comments

11 September 2015

Matthias Klumpp: Deploying Limba packages: Ideas & current status

LimbaHubThe Limba project does not only have the goal to allow developers to deploy their applications directly on multiple Linux distributions while reducing duplication of shared resources, it should also make it easy for developers to build software for Limba. Limba is worth nothing without good tooling to make it fun to use. That s why I am working on that too, and I want to share some ideas of how things could work in future and which services I would like to have running. I will also show what is working today already (and that s quite something!). This time I look at things from a developer s perspective (since the last posts on Limba were more end-user centric). If you read on, you will also find a nice video of the developer workflow 1. Creating metadata and building the software To make building Limba packages as simple as possible, Limba reuses already existing metadata, like AppStream metadata to find information about the software you want to create your package for. To ensure upstreams can build their software in a clean environment, Limba makes using one as simple as possible: The limba-build CLI tool creates a clean chroot environment quickly by using an environment created by debootstrap (or a comparable tool suitable for the Linux distribution), and then using OverlayFS to have all changes to the environment done during the build process land in a separate directory. To define build instructions, limba-build uses the same YAML format TravisCI uses as well for continuous integration. So there is a chance this data is already present as well (if not, it s trivial to write). In case upstream projects don t want to use these tools, e.g. because they have well-working CI already, then all commands needed to build a Limba package can be called individually as well (ideally, building a Limba package is just one call to lipkgen). I am currently planning DeveloperIPK packages containing resources needed to develop against another Limba package. With that in place and integrated with the automatic build-environment creation, upstream developers can be sure the application they just built is built against the right libraries as present in the package they depend on. The build tool could even fetch the build-dependencies automatically from a central repository. 2. Uploading the software to a repository While everyone can set up their own Limba repository, and the limba-build repo command will help with that, there are lots of benefits in having a central place where upstream developers can upload their software to. I am currently developing a service like that, called LimbaHub . LimbaHub will contain different repositories distributors can make available to their users by default, e.g. there will be one with only free software, and one for proprietary software. It will also later allow upstreams to create private repositories, e.g. for beta-releases. 3. Security in LimbaHub Every Limba package is signed with they key of its creator anyway, so in order to get a package into LimbaHub, one needs to get their OpenPGP key accepted by the service first. Additionally, the Hub service works with a per-package permission system. This means I can e.g. allow the Mozilla release team members to upload a package with the component-ID org.mozilla.firefox.desktop or even allow those user(s) to own the whole org.mozilla.* namespace. This should prevent people hijacking other people s uploads accidentally or on purpose. 4. QA with LimbaHub LimbaHub should also act as guardian over ABI stability and general quality of the software. We could for example warn upstreams that they broke ABI without declaring that in the package information, or even reject the package then. We could validate .desktop files and AppStream metadata, or even check if a package was built using hardening flags. This should help both developers to improve their software as well as users who benefit from that effort. In case something really bad gets submitted to LimbaHub, we always have the ability to remove the package from the repositories as a last resort (which might trigger Limba to issue a warning for the user that he won t receive updates anymore). What works Limba, LimbaHub and the tools around it are developing nicely, so far no big issues have been encountered yet. That s why I made a video showing how Limba and LimbaHub work together at time: Still, there is a lot of room for improvement Limba has not yet received enough testing, and LimbaHub is merely a proof-of-concept at time. Also, lots of high-priority features are not yet implemented. LimbaHub and Limba need help! At time I am developing LimbaHub and Limba alone with only occasional contributions from others (which are amazing and highly welcomed!). So, if you like Python and Flask, and want to help developing LimbaHub, please contact me the LimbaHub software could benefit from a more experienced Python web developer than I am (and maybe having a designer look over the frontend later makes sense as well). If you are not afraid of C and GLib, and like to chase bugs or play with building Limba packages, consider helping Limba development :-)

10 September 2015

Chris Lamb: Joining strings in POSIX shell

A common programming task is to glue (or "join") items together to create a single string. For example: 
>>> ', '.join(['foo', 'bar', 'baz'])
"foo, bar, baz"
Notice that we have three items but only two commas this can be important if the tool we passing doesn't support trailing delimiters or we simply want the result to be human-readable. Unfortunately, this can be inconvenient in POSIX shell where we construct strings via explicit concatenation. A na ve solution of: 
RESULT=""
for X in foo bar baz
do
    RESULT="$ RESULT , $ X "
done
... incorrectly returns ", foo, bar, baz". We can solve this with a (cumbersome) counter or flag to only attach the delimiter when we need it: 
COUNT=0
RESULT=""
for X in foo bar baz
do
    if [ "$ COUNT " = 0 ]
    then
        RESULT="$ X "
    else
        RESULT="$ RESULT , $ X "
    fi
    COUNT=$((COUNT + 1))
done
One alternative is to use the little-known ":+" expansion modifier. Many people are familiar with ":-" for returning default values: 
$ echo $ VAR:-fallback
By contrast, the ":+" modifier inverts this logic, returning the fallback if the specified variable is actually set. This results in the elegant: 
RESULT=""
for X in foo bar baz
do
    RESULT="$ RESULT:+$ RESULT ,  $ X "
done

8 September 2015

Daniel Silverstone: Orchestration, a cry for help

Over the past few years, a plethora of orchestration frameworks have been exploding onto the scene. Many have been around for quite a while but not all have the same sort of community behind them. For example there's a very interesting option in Joey Hess' Propellor but that is hurt by needing to be able to build Propellor on all the hosts you manage. On the other hand, Ansible is able to operate without installing extra software on your target hosts, but instead it ends up very latency-bound which can cause problems when your managed hosts are "far away". I have considered CFEngine, Chef, Puppet and Salt in addition to the above mentioned options, but none of them feel quite right to me. I am looking for a way to manage a small number of hosts, at least one of which is not always online (my laptop) and all of which are essentially snowflakes whose sparkleybits I want some reasonable control over. I have a few basic requirements which I worry would be hard to meet -- I want to be able to make changes to my hosts by editing a file and committing/pushing it to a git server. I want to be able to manage a host entirely over SSH from one or more systems, ideally without having to install the orchestration software on the target host, but where if the software is present it will get used to accelerate matters. I don't want to have to install Ruby or PHP on any system in order to have orchestration, and some of the systems I wish to manage simply can't compile Haskell stuff sanely. I'm not desperately interested in learning yet more DSLs, but I appreciate that it will be necessary, but I really don't want to have to learn more than one DSL simply to run one frameworks. I don't want to have to learn strange and confusing combinations of file formats. For example, Ansible quite sensibly uses YAML for its structured data except for its host/group lists. It uses Jinja2 for its templating and looping, except for some things which it generates its own looping constructs inside its YAML. I also personally find Ansible's sportsball oriented terminology to be confusing, but that might just be me. So what I'm hoping is that someone will be able to point me at a project which combines all the wonderful features of the above, with a need to learn only one DSL and which doesn't require to be installed on the managed host but which can benefit from being so installed, is driven from git, and won't hurt my already overly burdened brain. Dear Lazyweb, pls. kthxbye.

Eriberto Mota: Debian: how to use blhc to solve hardening issues when packaging

Implementing the hardening When packaging in Debian, is very common to see the lintian messages 'hardening-no-relro' and 'hardening-no-fortify-functions' in some softwares written in C or C++. To solve these issues, we can use the 'blhc' tool (apt-get install blhc). Please, get the revision 1.11-9 of the icmpinfo package. You can get this revision from http://snapshot.debian.org or from http://eriberto.pro.br/debian/icmpinfo. As a shortcut, you can use the following command: 
$ dget -u http://eriberto.pro.br/debian/icmpinfo/icmpinfo_1.11-9.dsc
The icmpinfo 1.11-9 is almost clean for lintian (in 2015-09-07, Standards-Version 3.9.6). The only problem is: 
W: icmpinfo: hardening-no-relro usr/sbin/icmpinfo
To track the problem I will use blhc over the .build file: 
$ blhc icmpinfo_1.11-9_amd64.build
LDFLAGS missing (-Wl,-z,relro): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -o icmpinfo recvping.o print.o err.o icmpinfo.o pid.o
Note that the problem is some missing options (-Wl,-z,relro) for LDFLAGS when building icmpinfo (for newbies, in GCC, -o is used to indicate the name to be used for the final binary after the compilation). If you are using the DebHelper compat 9 (debian/compat=9) and the DebHelper 9 (debhelper >= 9 in Build-Depends field in d/control), some variables as CFLAGS, LDFLAGS, CPPFLAGS and CXXFLAGS will be automatically passed during calls to dh_auto_* programs (yes, you should use the new and reduced d/rules format - see as example the debian/rules of the icmpinfo 1.11-9; if you still have doubts, $ man dh). Now, we need discover the reason why the LDFLAGS is being changed between its generation by the Debian build system and its utilization by the upstream's source code. So, we need to check the upstream Makefile. There is in Makefile (after a 'quilt push -a', to apply all current patches): 
LDFLAGS= $(CFLAGS)
OBJECTS= recvping.o print.o err.o icmpinfo.o pid.o
TARGET = icmpinfo
$(TARGET): $(OBJECTS)
 $(CC) $(LDFLAGS) -o $@ $(OBJECTS) $(LDLIBS)
Hummm... The LDFLAGS content generated by Debian is being dropped by Makefile because it is saying that "LDFLAGS = CFLAGS content". This line is a problem because the upstream Makefile needs to take and use the CFLAGS and LDFLAGS independently. To fix the issue, you can use this patch: 
--- icmpinfo-1.11.orig/Makefile
+++ icmpinfo-1.11/Makefile
@@ -20,13 +20,13 @@ VERS = 1.11
 
 RM = rm -f
 
-LDFLAGS= $(CFLAGS)
+#LDFLAGS= $(CFLAGS)
 
 OBJECTS= recvping.o print.o err.o icmpinfo.o pid.o
 TARGET = icmpinfo
 
 $(TARGET): $(OBJECTS)
- $(CC) $(LDFLAGS) -o $@ $(OBJECTS) $(LDLIBS)
+ $(CC) $(LDFLAGS) $(CFLAGS) -o $@ $(OBJECTS) $(LDLIBS)
 
 tgz: clean
 rm -f CHECKSUMS.asc
After a 'debuild' is a fact the problem is solved and the lintian is happy. See the blhc results: 
$ blhc ../icmpinfo_1.11-9_amd64.build
$
Now, we can improve the hardening. To see the current status, we can use the 'blhc --all' command. See here: 
blhc --all ../icmpinfo_1.11-9_amd64.build
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o recvping.o recvping.c
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o print.o print.c
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o err.o err.c
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o icmpinfo.o icmpinfo.c
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o pid.o pid.c
LDFLAGS missing (-fPIE -pie -Wl,-z,now): cc -Wl,-z,relro -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -o icmpinfo recvping.o print.o err.o icmpinfo.o pid.o
Well, we know that CFLAGS and LDFLAGS are present. Now, we can force the DebHelper to pass some extra options to make hardening better. Generally, is only needed to add the following line to debian/rules: 
export DEB_BUILD_MAINT_OPTIONS = hardening=+all
See the results (after a new debuild command): 
$ blhc --all ../icmpinfo_1.11-9_amd64.build
$
More examples Let me to show other example. I will use the mac-robber 1.02-3 (however, I disabled the Makefile.patch in debian/patches/series). After a debuild, the following lintian messages are presented: 
W: mac-robber: hardening-no-relro usr/bin/mac-robber
I: mac-robber: hardening-no-fortify-functions usr/bin/mac-robber
Using blhc: 
$ blhc ../mac-robber_1.02-3_amd64.build 
CFLAGS missing (-g -O2 -fstack-protector-strong -Wformat -Werror=format-security): gcc -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
CPPFLAGS missing (-D_FORTIFY_SOURCE=2): gcc -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
LDFLAGS missing (-Wl,-z,relro): gcc -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
We need to verify what is the problem in Makefile with CFLAGS, CPPFLAGS and LDFLAGS when generating the binary 'mac-robber' (just recalling, -o mac-robber in GCC command). See: 
linux_notstatic: 
 $(CC) -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
There are no references to CFLAGS, CPPFLAGS and LDFLAGS. To solve the problem, we need patch the Makefile to make this: 
linux_notstatic: 
 $(CC) $(CFLAGS) $(LDFLAGS) $(CPPFLAGS) -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
As last example, is possible that the Makefile is overriding the content sent by DebHelper when building. See this line from a hypothetical Makefile: 
CFLAGS = -g -Wall
As you can see, the Makefile is redefining CFLAGS; consequently, it is discarding any previous content sent by DebHelper. To solve this issue, we can use the following patch: 
-CFLAGS = -g -Wall
+CFLAGS += -g -Wall
So, the content received from DebHelper will be added to '-g -Wall'. Default parameters As curiosity, to see the basic parameters created by DebHelper as hardening, use the command: 
$ dpkg-buildflags
To see the all parameters, use the command: 
$ DEB_BUILD_MAINT_OPTIONS=hardening=+all dpkg-buildflags
More information More information about the hardening can be viewed at two places: https://wiki.debian.org/Hardening https://wiki.debian.org/HardeningWalkthrough I hope this help. Enjoy!

7 September 2015

Matthias Klumpp: Update notifications in Debian Jessie

Piwik told me that people are still sharing my post about the state of GNOME-Software and update notifications in Debian Jessie. So I thought it might be useful to publish a small update on that matter: I hope that clarifies things a little. Have fun using Debian 8!
UPDATE: It appears that many people have problems with getting update notifications in GNOME on Jessie. If you are affected by this, please try the following:
  1. Open dconf-editor and navigate to org.gnome.settings-daemon.plugins.updates. Check if the key active is set to true
  2. If that doesn t help, also check if at org.gnome.settings-daemon.plugins.updates the frequency-refresh-cache value is set to a sane value (e.g. 86400)
  3. Consider increasing the priority value (if it isn t at 300 already)
  4. If all of that doesn t help: I guess some internal logic in g-s-d is preventing a cache refresh then (e.g. because I thinks it is on a expensive network connection and therefore doesn t refresh the cache automatically, or thinks it is running on battery). This is a bug. If that still happens on Debian Stretch with GNOME-Software, please report a bug against the gnome-software package. As a workaround for Jessie you can enable unconditional cache refreshing via the APT cronjob by installing apt-config-auto-update.

6 September 2015

Lunar: Reproducible builds: week 19 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Dmitry Shachnev uploaded sphinx/1.3.1-6 with improved patches from Val Lorentz. Chris Lamb submitted a patch for ibus-table which makes the output of ibus-table-createdb deterministic. Niko Tyni wrote a patch to make libmodule-build-perl linking order deterministic. Santiago Vila has been leading discussions on the best way to fix timestamps coming from Gettext POT files. Packages fixed The following 35 packages became reproducible due to changes in their build dependencies: apache-log4j2, dctrl-tools, dms, gitit, gnubik, isrcsubmit, mailutils, normaliz, oaklisp, octave-fpl, octave-specfun, octave-vrml, opencolorio, openvdb, pescetti, php-guzzlehttp, proofgeneral, pyblosxom, pyopencl, pyqi, python-expyriment, python-flask-httpauth, python-mzml, python-simpy, python-tidylib, reactive-streams, scmxx, shared-mime-info, sikuli, siproxd, srtp, tachyon, tcltk-defaults, urjtag, velvet. The following packages became reproducible after getting fixed: The package is not in yet in unstable, but linux/4.2-1~exp1 is now reproducible! Kudos to Ben Hutchings, and most fixes are already merged upstream. Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net Some bugs that prevented packages to build successfully in the remote builders have been fixed. (h01ger) Two more amd64 build jobs have been removed from the Jenkins host in favor of six more on the new remote nodes. (h01ger) The munin graphs currently looks fine, so more amd64 jobs will probably be added in the next week. diffoscope development Version 32 of diffoscope has been released on September 3rd with the following new features: It also fixes many bugs. Head over to the changelog for the full list. Version 33 was released the day after to fix a bug introduced in the packaging. Documentation update Chris Lamb blessed the SOURCE_DATE_EPOCH specification with the version number 1.0 . Lunar documented how the .file assembler directive can help with random filenames in debug symbols. Package reviews 235 reviews have been removed, 84 added and 277 updated this week. 29 new FTBFS bugs were filled by Chris Lamb, Chris West (Faux), Daniel Stender, and Niko Tyni. New issues identified this week: random_order_in_ibus_table_createdb_output, random_order_in_antlr_output, nondetermistic_link_order_in_module_build, and timestamps_in_tex_documents. Misc. Thanks to Dhole and Thomas Vincent, the talk held at DebConf15 now has subtitles! Void Linux started to merge changes to make packages produced by xbps reproducible.

4 September 2015

Julien Danjou: Data validation in Python with voluptuous

Continuing my post series on the tools I use these days in Python, this time I would like to talk about a library I really like, named voluptuous. It's no secret that most of the time, when a program receives data from the outside, it's a big deal to handle it. Indeed, most of the time your program has no guarantee that the stream is valid and that it contains what is expected. The robustness principle says you should be liberal in what you accept, though that is not always a good idea neither. Whatever policy you chose, you need to process those data and implement a policy that will work lax or not. That means that the program need to look into the data received, check that it finds everything it needs, complete what might be missing (e.g. set some default), transform some data, and maybe reject those data in the end.
Data validation The first step is to validate the data, which means checking all the fields are there and all the types are right or understandable (parseable). Voluptuous provides a single interface for all that called a Schema. 
>>> from voluptuous import Schema
>>> s = Schema( 
...   'q': str,
...   'per_page': int,
...   'page': int,
...  )
>>> s( "q": "hello" )
 'q': 'hello' 
>>> s( "q": "hello", "page": "world" )
voluptuous.MultipleInvalid: expected int for dictionary value @ data['page']
>>> s( "q": "hello", "unknown": "key" )
voluptuous.MultipleInvalid: extra keys not allowed @ data['unknown']

The argument to voluptuous.Schema should be the data structure that you expect. Voluptuous accepts any kind of data structure, so it could also be a simple string or an array of dict of array of integer. You get it. Here it's a dict with a few keys that if present should be validated as certain types. By default, Voluptuous does not raise an error if some keys are missing. However, it is invalid to have extra keys in a dict by default. If you want to allow extra keys, it is possible to specify it. 
>>> from voluptuous import Schema
>>> s = Schema( "foo": str , extra=True)
>>> s( "bar": 2 )
 "bar": 2

It is also possible to make some keys mandatory. 
>>> from voluptuous import Schema, Required
>>> s = Schema( Required("foo"): str )
>>> s( )
voluptuous.MultipleInvalid: required key not provided @ data['foo']

You can create custom data type very easily. Voluptuous data types are actually just functions that are called with one argument, the value, and that should either return the value or raise an Invalid or ValueError exception. 
>>> from voluptuous import Schema, Invalid
>>> def StringWithLength5(value):
...     if isinstance(value, str) and len(value) == 5:
...             return value
...     raise Invalid("Not a string with 5 chars")
...
>>> s = Schema(StringWithLength5)
>>> s("hello")
'hello'
>>> s("hello world")
voluptuous.MultipleInvalid: Not a string with 5 chars

Most of the time though, there is no need to create your own data types. Voluptuous provides logical operators that can, combined with a few others provided primitives such as voluptuous.Length or voluptuous.Range, create a large range of validation scheme. 
>>> from voluptuous import Schema, Length, All
>>> s = Schema(All(str, Length(min=3, max=5)))
>>> s("hello")
"hello"
>>> s("hello world")
voluptuous.MultipleInvalid: length of value must be at most 5

The voluptuous documentation has a good set of examples that you can check to have a good overview of what you can do. Data transformation
What's important to remember, is that each data type that you use is a function that is called and returns a value, if the value is considered valid. That value returned is what is actually used and returned after the schema validation: 
>>> import uuid
>>> from voluptuous import Schema
>>> def UUID(value):
...     return uuid.UUID(value)
...
>>> s = Schema( "foo": UUID )
>>> data_converted = s( "foo": "uuid?" )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']
>>> data_converted = s( "foo": "8B7BA51C-DFF5-45DD-B28C-6911A2317D1D" )
>>> data_converted
 'foo': UUID('8b7ba51c-dff5-45dd-b28c-6911a2317d1d')

By defining a custom UUID function that converts a value to a UUID, the schema converts the string passed in the data to a Python UUID object validating the format at the same time. Note a little trick here: it's not possible to use directly uuid.UUID in the schema, otherwise Voluptuous would check that the data is actually an instance of uuid.UUID: 
>>> from voluptuous import Schema
>>> s = Schema( "foo": uuid.UUID )
>>> s( "foo": "8B7BA51C-DFF5-45DD-B28C-6911A2317D1D" )
voluptuous.MultipleInvalid: expected UUID for dictionary value @ data['foo']
>>> s( "foo": uuid.uuid4() )
 'foo': UUID('60b6d6c4-e719-47a7-8e2e-b4a4a30631ed')

And that's not what is wanted here. That mechanism is really neat to transform, for example, strings to timestamps. 
>>> import datetime
>>> from voluptuous import Schema
>>> def Timestamp(value):
...     return datetime.datetime.strptime(value, "%Y-%m-%dT%H:%M:%S")
...
>>> s = Schema( "foo": Timestamp )
>>> s( "foo": '2015-03-03T12:12:12' )
 'foo': datetime.datetime(2015, 3, 3, 12, 12, 12) 
>>> s( "foo": '2015-03-03T12:12' )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']

Recursive schemas So far, Voluptuous has one limitation so far: the ability to have recursive schemas. The simplest way to circumvent it is by using another function as an indirection. 
>>> from voluptuous import Schema, Any
>>> def _MySchema(value):
...     return MySchema(value)
...
>>> from voluptuous import Any
>>> MySchema = Schema( "foo": Any("bar", _MySchema) )
>>> MySchema( "foo":  "foo": "bar" )
 'foo':  'foo': 'bar' 
>>> MySchema( "foo":  "foo": "baz" )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']['foo']

Usage in REST API
I started to use Voluptuous to validate data in a the REST API provided by Gnocchi. So far it has been a really good tool, and we've been able to create a complete REST API that is very easy to validate on the server side. I would definitely recommend it for that. It blends with any Web framework easily. One of the upside compared to solution like JSON Schema, is the ability to create or re-use your own custom data types while converting values at validation time. It is also very Pythonic, and extensible it's pretty great to use for all of that. It's also not tied to any serialization format. On the other hand, JSON Schema is language agnostic and is serializable itself as JSON. That makes it easy to be exported and provided to a consumer so it can understand the API and validate the data potentially on its side.

1 September 2015

Lunar: Reproducible builds: week 18 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Aur lien Jarno uploaded glibc/2.21-0experimental1 which will fix the issue were locales-all did not behave exactly like locales despite having it in the Provides field. Lunar rebased the pu/reproducible_builds branch for dpkg on top of the released 1.18.2. This made visible an issue with udebs and automatically generated debug packages. The summary from the meeting at DebConf15 between ftpmasters, dpkg mainatainers and reproducible builds folks has been posted to the revelant mailing lists. Packages fixed The following 70 packages became reproducible due to changes in their build dependencies: activemq-activeio, async-http-client, classworlds, clirr, compress-lzf, dbus-c++, felix-bundlerepository, felix-framework, felix-gogo-command, felix-gogo-runtime, felix-gogo-shell, felix-main, felix-shell-tui, felix-shell, findbugs-bcel, gco, gdebi, gecode, geronimo-ejb-3.2-spec, git-repair, gmetric4j, gs-collections, hawtbuf, hawtdispatch, jack-tools, jackson-dataformat-cbor, jackson-dataformat-yaml, jackson-module-jaxb-annotations, jmxetric, json-simple, kryo-serializers, lhapdf, libccrtp, libclaw, libcommoncpp2, libftdi1, libjboss-marshalling-java, libmimic, libphysfs, libxstream-java, limereg, maven-debian-helper, maven-filtering, maven-invoker, mochiweb, mongo-java-driver, mqtt-client, netty-3.9, openhft-chronicle-queue, openhft-compiler, openhft-lang, pavucontrol, plexus-ant-factory, plexus-archiver, plexus-bsh-factory, plexus-cdc, plexus-classworlds2, plexus-component-metadata, plexus-container-default, plexus-io, pytone, scolasync, sisu-ioc, snappy-java, spatial4j-0.4, tika, treeline, wss4j, xtalk, zshdb. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: Chris Lamb also noticed that binaries shipped with libsilo-bin did not work. Documentation update Chris Lamb and Ximin Luo assembled a proper specification for SOURCE_DATE_EPOCH in the hope to convince more upstreams to adopt it. Thanks to Holger it is published under a non-Debian domain name. Lunar documented easiest way to solve issues with file ordering and timestamps in tarballs that came with tar/1.28-1. Some examples on how to use SOURCE_DATE_EPOCH have been improved to support systems without GNU date. reproducible.debian.net armhf is finally being tested, which also means the remote building of Debian packages finally works! This paves the way to perform the tests on even more architectures and doing variations on CPU and date. Some packages even produce the same binary Arch:all packages on different architectures (1, 2). (h01ger) Tests for FreeBSD are finally running. (h01ger) As it seems the gcc5 transition has cooled off, we schedule sid more often than testing again on amd64. (h01ger) disorderfs has been built and installed on all build nodes (amd64 and armhf). One issue related to permissions for root and unpriviliged users needs to be solved before disorderfs can be used on reproducible.debian.net. (h01ger) strip-nondeterminism Version 0.011-1 has been released on August 29th. The new version updates dh_strip_nondeterminism to match recent changes in debhelper. (Andrew Ayer) disorderfs disorderfs, the new FUSE filesystem to ease testing of filesystem-related variations, is now almost ready to be used. Version 0.2.0 adds support for extended attributes. Since then Andrew Ayer also added support to reverse directory entries instead of shuffling them, and arbitrary padding to the number of blocks used by files. Package reviews 142 reviews have been removed, 48 added and 259 updated this week. Santiago Vila renamed the not_using_dh_builddeb issue into varying_mtimes_in_data_tar_gz_or_control_tar_gz to align better with other tag names. New issue identified this week: random_order_in_python_doit_completion. 37 FTBFS issues have been reported by Chris West (Faux) and Chris Lamb. Misc. h01ger gave a talk at FrOSCon on August 23rd. Recordings are already online. These reports are being reviewed and enhanced every week by many people hanging out on #debian-reproducible. Huge thanks!

Russ Allbery: Review: Kanban

Review: Kanban, by David J. Anderson
Publisher: Blue Hole
Copyright: 2010
ISBN: 0-9845214-0-2
Format: Trade paperback
Pages: 240
Another belated review, this time of a borrowed book. Which I can now finally return! A delay in the review of this book might be a feature if I had actually used it for, as the subtitle puts it, successful evolutionary change in my technology business. Sadly, I haven't, so it's just late. So, my background: I've done a lot of variations of traditional project management for IT projects (both development and more operational ones), both as a participant and as a project manager. (Although I've never done the latter as a full-time job, and have no desire to do so.) A while back at Stanford, my team adopted Agile, specifically Scrum, so I did a fair bit of research about Scrum including a couple of training courses. Since then, at Dropbox, I've used a few different variations on a vaguely Agile-inspired planning process, although it's not particularly consistent with any one system. I've been hearing about Kanban for a while and have friends who swear by it, but I only had a vague idea of how it worked. That seemed like a good excuse to read a book. And Anderson's book is a good one if, like me, you're looking for a basic introduction. It opens with a basic description and definition, talks about motivation and the expected benefits, and then provides a detailed description of Kanban as a system. The tone is on the theory side, written using the terminology of (I presume) management theory and operations management, areas about which I know almost nothing, but the terminology wasn't so heavy as to make the book hard to read. Anderson goes into lots of detail, and I thought he did a good job of distinguishing between basic principles, optional techniques, and variations that may be appropriate for particular environments. If you're also not familiar, the basic concept of Kanban is to organize work using an in-progress queue. It eschews time-bounded planning entirely in favor of staging work at the start of a sequence of queues and letting the people working on each queue pull from the previous queue when they're ready to take on a new task. As you might guess from that layout, Kanban was originally invented for assembly-line manufacturing (at Toyota). That was one of the problems that I had with it, or at least the presentation in this book: most of my software development doesn't involve finishing one part of something and handing it off to someone else, which made it hard to identify with the pipeline model. Anderson has clearly spent a lot of time working with large-scale programming shops, including outsourced development firms, with dedicated QA and operations roles. This is not at all how Silicon Valley agile development works, so parts of this book felt like missives from a foreign country. That said, the key point of Kanban is not the assembly line but the work limit. One of the defining characteristics of Kanban, at least as Anderson presents it, is that one does not try to estimate work and tile it based on estimates, not even to the extent that Scrum and other Agile methodologies do within the sprint. Instead, each person takes as long as they take on the things they're working on, and pulls a new task when they have free capacity. The system instead puts a limit on how many things they can have in progress at a time. The problem of pipeline stalls is dealt with both via continuous improvement and "swarming" of a problem to unblock the line (since other teams may not be able to work until the block is fixed), and with being careful about the sizing of work items (I'm used to calling them stories) that go in the front end. Predictability, which Scrum uses story sizing and team velocity analysis to try to achieve, is basically statistical. One uses a small number of buckets of sizes of stories, and the whole pipeline will finish some number of work items per unit time, with a variance. The promise made to clients and other teams is that some percentage of the work items will be finished within some time frame from when they enter the system. Most importantly, these are all descriptive properties, determined statistically after the fact, rather than planned properties worked out through story sizing and extensive team discussion. If you, like me, are pretty thoroughly sick of two-hour sprint planning meetings and endless sizing exercises, this is rather appealing. My problem with most work planning systems is that I think they overplan and put too much weight our ability to estimate how long work will take. Kanban is very appealing when viewed through that lens: it gives up on things we're bad at in favor of simple measurement and building a system with enough slack that it can handle work of various different sizes. As mentioned at the top, I haven't had a chance to try it (and I'm not sure it's a good fit with the inter-group planning methods in use at my current workplace), but I came away from this book wanting to do so. If, like me, your experience is mostly with small combined teams or individual programming work, Anderson's examples may seem rather large, rather formal, and bureaucratic. But this is a solid introduction and well worth reading if your only experience with Agile is sprint planning, story writing and sizing, and fast iterations. Rating: 7 out of 10

30 August 2015

Antonio Terceiro: DebConf15, testing debian packages, and packaging the free software web

This is my August update, and by the far the coolest thing in it is Debconf. Debconf15 I don t get tired of saying it is the best conference I ever attended. First it s a mix of meeting both new people and old friends, having the chance to chat with people whose work you admire but never had a chance to meet before. Second, it s always quality time: an informal environment, interesting and constructive presentations and discussions. This year the venue was again very nice. Another thing that was very nice was having so many kids and families. This was no coincidence, since this was the first DebConf in which there was organized childcare. As the community gets older, this a very good way of keeping those who start having kids from being alienated from the community. Of course, not being a parent yet I have no idea how actually hard is to bring small kids to a conference like DebConf. ;-) I presented two talks: There was also the now traditional Ruby BoF, where discussed the state and future of the Ruby ecosystem in Debian; and an in promptu Ruby packaging workshop where we introduced the basics of packaging in general, and Ruby packaging specifically. Besides shak, I was able to hack on a few cool things during DebConf: Miscellaneous updates

Philipp Kern: Automating the 3270 part of a Debian System z install

If you try to install Debian on System z within z/VM you might be annoyed at the various prompts it shows before it lets you access the network console via SSH. We can do better. From within CMS copy the default EXEC and default PARMFILE:

COPYFILE DEBIAN EXEC A DEBAUTO EXEC A
COPYFILE PARMFILE DEBIAN A PARMFILE DEBAUTO A

Now edit DEBAUTO EXEC A and replace the DEBIAN in 'PUNCH PARMFILE DEBIAN * (NOHEADER' with DEBAUTO. This will load the alternate kernel parameters file into the card reader, while still loading the original kernel and initrd files.

Replace PARMFILE DEBAUTO A's content with this (note the 80 character column limit):

ro locale=C
s390-netdevice/choose_networktype=qeth s390-netdevice/qeth/layer2=true
s390-netdevice/qeth/choose=0.0.fffc-0.0.fffd-0.0.fffe
netcfg/get_ipaddress=<IPADDR> netcfg/get_netmask=255.255.255.0
netcfg/get_gateway=<GW> netcfg/get_nameservers=<FIRST-DNS>
netcfg/confirm_static=true netcfg/get_hostname=debian
netcfg/get_domain=
network-console/authorized_keys_url=http://www.kern.lc/id_rsa.pub
preseed/url=http://www.kern.lc/preseed-s390x.cfg

Replace <IPADDR>, <GW>, and <FIRST-DNS> to suit your local network config. You might also need to change the netmask, which I left in for clarity about the format. Adjust the device address of your OSA network card. If it's in layer 3 mode (very likely) you should set layer2=false. Note that mixed case matters, hence you will want to SET CASE MIXED in xedit.

Then there are the two URLs that need to be changed. The authorized_keys_url file contains your SSH public key and is fetched unencrypted and unauthenticated, so be careful what networks you traverse with your request (HTTPS is not supported by debian-installer in Debian).

 preseed/url is needed for installation parameters that do not fit the parameters file - there is an upper character limit that's about two lines longer than my example. This is why this example only contains the bare minimum for the network part, everything else goes into this preseeding file. It file can optionally be protected with a MD5 checksum in preseed/url/checksum.

Both URLs need to be very short. I thought that there was a way to specify a line continuation, but in my tests I was unable to produce one. Hence it needs to fit on one line, including the key. You might want to use an IPv4 as the hostname.

To skip the initial boilerplate prompts and to skip straight to the user and disk setup you can use this as preseed.cfg:

d-i debian-installer/locale string en_US
d-i debian-installer/country string US
d-i debian-installer/language string en
d-i time/zone US/Eastern
d-i mirror/country manual
d-i mirror/http/mirror string httpredir.debian.org
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string

I'm relatively certain that the DASD disk setup part cannot be automated yet. But the other bits of the installation should be preseedable just like on non-mainframe hardware.

Dirk Eddelbuettel: RcppGSL 0.3.0

A new version of RcppGSL just arrived on CRAN. The RcppGSL package provides an interface from R to the GNU GSL using our Rcpp package. Following on the heels of an update last month we updated the package (and its vignette) further. One of the key additions concern memory management: Given that our proxy classes around the GSL vector and matrix types are real C++ object, we can monitor their scope and automagically call free() on them rather then insisting on the user doing it. This renders code much simpler as illustrated below. Dan Dillon added const correctness over a series on pull request which allows us to write more standard (and simply nicer) function interfaces. Lastly, a few new typedef declaration further simply the use of the (most common) double and int vectors and matrices. Maybe a code example will help. RcppGSL contains a full and complete example package illustrating how to write a package using the RcppGSL facilities. It contains an example of computing a column norm -- which we blogged about before when announcing an much earlier version. In its full glory, it looks like this: 
#include <RcppGSL.h>
#include <gsl/gsl_matrix.h>
#include <gsl/gsl_blas.h>
extern "C" SEXP colNorm(SEXP sM)  
  try  
        RcppGSL::matrix<double> M = sM;     // create gsl data structures from SEXP
        int k = M.ncol();
        Rcpp::NumericVector n(k);           // to store results
        for (int j = 0; j < k; j++)  
            RcppGSL::vector_view<double> colview = gsl_matrix_column (M, j);
            n[j] = gsl_blas_dnrm2(colview);
         
        M.free() ;
        return n;                           // return vector
    catch( std::exception &ex )  
        forward_exception_to_r( ex );
    catch(...)  
        ::Rf_error( "c++ exception (unknown reason)" );
   
  return R_NilValue; // -Wall
We manually translate the SEXP coming from R, manually cover the try and catch exception handling, manually free the memory etc pp. Well in the current version, the example is written as follows: 
#include <RcppGSL.h>
#include <gsl/gsl_matrix.h>
#include <gsl/gsl_blas.h>
// [[Rcpp::export]]
Rcpp::NumericVector colNorm(const RcppGSL::Matrix & G)  
    int k = G.ncol();
    Rcpp::NumericVector n(k);           // to store results
    for (int j = 0; j < k; j++)  
        RcppGSL::VectorView colview = gsl_matrix_const_column (G, j);
        n[j] = gsl_blas_dnrm2(colview);
     
    return n;                           // return vector
This takes full advantage of Rcpp Attributes automagically creating the interface and exception handler (as per the previous release), adds a const & interface, does away with the tedious and error-pronce free() and uses the shorter-typedef forms for RcppGSL::Matrix and RcppGSL::VectorViews using double variables. Now the function is short and concise and hence easier to read and maintain. The package vignette has more details on using RcppGSL. The NEWS file entries follows below:
Changes in version 0.3.0 (2015-08-30)
  • The RcppGSL matrix and vector class now keep track of object allocation and can therefore automatically free allocated object in the destructor. Explicit x.free() use is still supported.
  • The matrix and vector classes now support const reference semantics in the interfaces (thanks to PR #7 by Dan Dillon)
  • The matrix_view and vector_view classes are reorganized to better support const arguments (thanks to PR #8 and #9 by Dan Dillon)
  • Shorthand forms such as Rcpp::Matrix have been added for double and int vectors and matrices including views.
  • Examples such as fastLm can now be written in a much cleaner and shorter way as GSL objects can appear in the function signature and without requiring explicit .free() calls at the end.
  • The included examples, as well as the introductory vignette, have been updated accordingly.
Courtesy of CRANberries, a summary of changes to the most recent release is available. More information is on the RcppGSL page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Sven Hoexter: 1960 SubjectAlternativeNames on one certificate

tl;dr; You can add 1960+ SubjectAlternativeNames on one certificate and at least Firefox and Chrome are working fine with that. Internet Explorer failed but I did not investigate why. So why would you want to have close to 2K SANs on one certificate? While we're working on adopting a more dynamic development workflow at my workplace we're currently bound to a central development system. From there we serve a classic virtual hosting setup with "projectname.username.devel.ourdomain.example" mapped on "/web/username/projectname/". That is 100% dynamic with wildcard DNS entries and you can just add a new project to your folder and use it directly. All of that is served from just a single VirtualHost. Now our developers started to go through all our active projects to make them fit for serving via HTTPS. While we can verify the proper usage of https on our staging system where we've validating certificates, that's not the way you'd like to work. So someone approached me to look into a solution for our development system. Obvious choices like wildcard certificates do not work here because we've two dynamic components in the FQDN. So we would've to buy a wildcard certificate for every developer and we would've to create a VirtualHost entry for every new developer. That's expensive and we don't want all that additional work. So I started to search for documented limits on the number of SANs you can have on a certificate. The good news: there are none. The RFC does not define a limit. So much about the theory. ;) Following Ivans excellent documentation I setup an internal CA and an ugly "find ... sed ... tr ..." one-liner later I had a properly formated openssl config file to generate a CSR with all 1960 "projectname.username..." SAN combinations found on the development system. Two openssl invocations (CSR generation and signing) later I had a signed certificate with 1960 SANs on it. I imported the internal CA I created in Firefox and Chrome, and to my surprise it worked. Noteworthy: To sign with "openssl ca" without interactive prompts you've to use the "-batch" option. I'm thinking about regenerating the certificate every morning so our developers just have to create a new project directory and within 24h serving via HTTPS would be enabled. The only thing I'm currently pondering about is how to properly run the CA in a corporate Windows world. We could of course ask the Windows guys to include it for everyone but then we would've to really invest time in properly running the CA. I'd like to avoid that hassle. So I'd guess we just stick to providing the CA for those developers who need it. This all or nothing model is a constant PITA, and you really do not want to get owned via your own badly managed CA. :( Regarding Internet Explorer it jumped in my face with a strange error message that recommended to enable TLS 1.0, 1.1 and 1.2 in the options menu. Of course that's already enable. I'll try to take a look at the handshake next week, but I bet we've to accept for the moment that IE will not work with so many SANs. Would be interesting to try out Windows 10 with Spartan, but well I'm not that interested in Windows to invest more time on that front. Other TLS implementations, like Java, would be also interesting to test.

Russ Allbery: Review: Pound Foolish

Review: Pound Foolish, by Helaine Olen
Publisher: Penguin
Copyright: 2012, 2013
Printing: 2013
ISBN: 1-59184-679-X
Format: Trade paperback
Pages: 241
For at least the last six years, it's not been surprising news that the relationship between the average person and the US financial system is tense at best, and downright exploitative at worst. The collapse of the housing bubble revealed a morass of predatory lending practices, spectacularly poor (or spectacularly cynical) investment decisions, and out-of-control personal debt coupled with erosion of bankruptcy law. Despite this, there's always a second story in all discussions of the finances of the US population: the problem isn't with financial structures and products, but with us. We're too stupid, or naive, or emotional, or uninformed, or greedy, or unprincipled, or impatient. Finances are complicated, yes, but that just means we have to be more thoughtful. All of these complex financial products could have been used properly. Helaine Olen's Pound Foolish is a systemtic, biting, well-researched, and pointed counter to that second story. The short summary of this book is that it's not us. We're being set up for failure, and there is a large and lucrative industry profiting off of that failure. And many (although not all) people in that industry know exactly what they're doing. Pound Foolish is one of my favorite forms of non-fiction: long-form journalism. This is an investigative essay into the personal finance and investment industry, developed to book length. Readers of Michael Lewis will feel right at home. Olen doesn't have Lewis's skill with characterization and biography, but she makes up for it in big-picture clarity. She takes a wealth of individual facts about who is involved in personal finance, how they make money, what they recommend, and who profits, and develops it into a clear and coherent overview. If you have paid any attention to US financial issues, you'll know some of this already. Frontline has done a great job of covering administrative fees in mutual funds. Lots of people have warned about annuities. The spectacular collapse of the home mortgage is old news now. But Olen does a great job of finding the connections between these elements and adding some less familiar ones, including an insightful and damning analysis of financial literacy campaigns and the widespread belief that these problems are caused by lack of consumer understanding. I've read and watched a lot of related material, including several full-book treatments of the mortgage crisis, so I think it's telling that I never got bored in the middle of Olen's treatment. I find the deep US belief in the power of personal improvement fascinating. It feels like one of the defining characteristics of US culture, for both good and for ill. We're very good at writing compelling narratives of personal improvement, and sometimes act on them. We believe that everyone can and should improve themselves. But that comes coupled to a dislike and distrust of expertise, even when it is legitimate and earned (Hofstadter's Anti-Intellectualism in American Life develops this idea at length). And I believe we significantly overestimate the ability of individuals to act despite systems that are stacked against us, and significantly overestimate our responsibility for the inevitable results. This was the main message I took from Pound Foolish: we desperately want to believe in the myth of personal control. We want to believe that our financial troubles are something we can fix through personal education, more will power, better decisions, or just the right investment. And so, we turn to gurus like Suze Orman and buy their mix of muddled financial advice and "tough love" that completely ignores broader social factors. We're easy marks for psychologically-trained investment sellers who mix fear, pressure, and a fantasy of inside knowledge and personal control. We're fooled by a narrative of empowerment and stand by while a working retirement system (guaranteed benefit pensions) is undermined and destroyed in favor of much riskier investment schemes... riskier for us, at least, but loaded with guaranteed profits for the people who "advise" us. And we cling to financial literacy campaigns that are funded by exactly the same credit card companies who fight tooth and nail against regulations that would require they provide simple, comprehensible descriptions of loan terms. One wonders if they support them precisely because they know they don't work. Olen mentions, in passing, the Stanford marshmallow experiment, which is often used as a foundation for arguments about personal responsibility for financial outcomes, but she doesn't do a detailed critique. I wish she had, since I think it's such a good example of the theme of this book. The Stanford marshmallow experiment was a psychological experiment from the late 1960s and early 1970s in delayed gratification. Children were put in a room in front of some treat (marshmallows, cookies, pretzels) and told that they could eat it if they wished. But if they didn't eat the treat before the monitor came back, they would get two of the treat instead. Long-term follow-up studies found that the children who refrained from eating the treat and got the reward had better life outcomes along multiple metrics: SAT scores, educational attainment, and others. On the surface, this seems to support everything US culture loves to believe about the power of self-control, self-improvement, and the Protestant work ethic. People who can delay gratification and save for a future reward do better in the world. (The darker interpretation, also common since the experiment was performed on children, is that the ability to delay gratification has a genetic component, and some people are just doomed to make poor decisions due to their inability to exercise self-control.) However, I can call the traditional interpretation into question with one simple question that the experimenters appeared not to ask: under what circumstances would taking the treat immediately be the rational and best choice? One answer, of course, is when one does not trust the adult to produce the promised reward. If the adult might come back, take the treat away, and not give any treat, it's to the child's advantage to take the treat immediately. Even if the adult left the treat but wouldn't actually double it, it's to the child's advantage to take the treat immediately. The traditional interpretation assumes the child trusts the adults performing the experiment a logical assumption for those of us whose childhood experience was that adults could generally be trusted and that promised rewards would materialize. If the child instead came from a chaotic family where adults weren't reliable, or just one where frequent unexpected money problems meant that promised treats often didn't materialize, the most logical assumption may be much different. One has to ask if such a background may have more to do with the measured long-term life outcomes than the child's unwillingness to trust in a future reward. And this is one of the major themes of this book. Problems the personal finance industry attributes to our personal shortcomings (which they're happy to take our money to remedy) are often systematic, or at least largely outside of our control. We may already be making the most logical choices given our personal situations. We're in worse financial shape because we're making less money. Our retirements are in danger because our retirement systems were dismantled and replaced with risky and expensive alternatives. And where problems are attributed to our poor choices, one can find entire industries that focus on undermining our ability to make good choices: scaring us, confusing us, hiding vital information, and exploiting known weaknesses of human psychology to route our money to them. These are not problems that can be solved by watching Suze Orman yell at us to stop buying things. These are systematic social problems that demand a sweeping discussion about regulation, automatic savings systems, and social insurance programs to spread risk and minimize the weaknesses of human psychology. Exactly the kind of discussion that the personal finance industry doesn't want us to have. Those who are well-read in these topics probably won't find a lot new here. Those who aren't in the US will shake their heads at some of the ways that the US fails its citizens, although many of Olen's points apply even to countries with stronger social safety nets. But if you're interested in solid long-form journalism on this topic, backed by lots of data on just how badly a focus on personal accountability is working for us, I recommend Pound Foolish. Rating: 8 out of 10

29 August 2015

Tassia Camoes Araujo: Report from the MicroDebconf Bras lia 2015

This was an event organized due to a coincidental meeting of a few DD s in the city of Brasilia on May 31st 2015. What a good thing when we can mix vacations, friends and Debian ;-)

Group photo

We called it Micro due to its short duration and planning phase, to be fair with other Mini DebConfs that take a lot more of organization. We also ended up having a translation sprint inside the event that attracted contributors from other cities. Our main goal was to boost the local community and bring new contributors to Debian. And we definitely made it! The meeting happened at University of Brasilia (UnB Gama). It started with a short presentation where each DD and Debian contributor presented their involvement with Debian and plans for the hacking session. This was an invitation for new contributors to choose the activities they were willing to engage, taking advantage of being guided by more experienced people. Then we moved to smaller rooms where participants were split in different groups to work on each track: packaging, translation and community/contribution. We all came together later for the keysigning party. Some of the highlights of the day:
  • ~40 participants, from which ~10 were already engaged in the Debian community
  • hands-on packaging tutorial
  • 4 new packages uploaded
  • from the 6 brazilian names annouced as new contributors in the DPN just after the meeting, 4 were among us in Bras lia
  • hands-on translation tutorial
  • newbie translators paired with more experienced ones, numerous translations committed
  • discussion about chalenges of migrating debianArt to Noosfero
  • initial setup of Collab.Debian (with Noosfero), aiming to facilitate contributions of users to the Debian project (this platform was offcially released at DC15 lightining talks (46:00))
  • first keysigning party for many of the participants
  • first time some longterm Brazilian contributors had the change to meet in person
For more details of what happened, you can read our full report. The MicroDebconf wouldn t be possible without the support of prof. Paulo Meirelles from UnB Gama and all the LAPPIS team for the local organization and students mobilization. We also need to thank to Debian donnors, who covered the travel costs of one of our contributors. Last but not least, thanks to our participants and the large Brazilian community who is giving a good example of team work. A similar meeting happened in July during the Free Software International Forum (FISL) and another one is already planned to happen in October as part of the LatinoWare. I hope I can join those folks again in the near future!

19 August 2015

Russell Coker: The Purpose of a Code of Conduct

On a private mailing list there have been some recent discussions about a Code of Conduct which demonstrate some great misunderstandings. The misunderstandings don t seem particular to that list so it s worthy of a blog post. Also people tend to think more about what they do when their actions will be exposed to a wider audience so hopefully people who read this post will think before they respond. Jokes The first discussion concerned the issue of making jokes . When dealing with the treatment of other people (particularly minority groups) the issue of jokes is a common one. It s fairly common for people in positions of power to make jokes about people with less power and then complain if someone disapproves. The more extreme examples of this concern hate words which are strongly associated with violence, one of the most common is a word used to describe gay men which has often been associated with significant violence and murder. Men who are straight and who conform to the stereotypes of straight men don t have much to fear from that word while men who aren t straight will associate it with a death threat and tend not to find any amusement in it. Most minority groups have words that are known to be associated with hate crimes. When such words are used they usually send a signal that the minority groups in question aren t welcome. The exception is when the words are used by other members of the group in question. For example if I was walking past a biker bar and heard someone call out geek or nerd I would be a little nervous (even though geeks/nerds have faced much less violence than most minority groups). But at a Linux conference my reaction would be very different. As a general rule you shouldn t use any word that has a history of being used to attack any minority group other than one that you are a member of, so black rappers get to use a word that was historically used by white slave-owners but because I m white I don t get to sing along to their music. As an aside we had a discussion about such rap lyrics on the Linux Users of Victoria mailing list some time ago, hopefully most people think I m stating the obvious here but some people need a clear explanation. One thing that people should consider jokes is the issue of punching-down vs punching-up [1] (there are many posts about this topic, I linked to the first Google hit which seems quite good). The basic concept is that making jokes about more powerful people or organisations is brave while making jokes about less powerful people is cowardly and serves to continue the exclusion of marginalised people. When I raised this issue in the mailing list discussion a group of men immediately complained that they might be bullied by lots of less powerful people making jokes about them. One problem here is that powerful people tend to be very thin skinned due to the fact that people are usually nice to them. While the imaginary scenario of less powerful people making jokes about rich white men might be unpleasant if it happened in person, it wouldn t compare to the experience of less powerful people who are the target of repeated jokes in addition to all manner of other bad treatment. Another problem is that the impact of a joke depends on the power of the person who makes it, EG if your boss makes a joke about you then you have to work on your CV, if a colleague or subordinate makes a joke then you can often ignore it. Who does a Code of Conduct Protect One member of the mailing list wrote a long and very earnest message about his belief that the CoC was designed to protect him from off-topic discussions. He analysed the results of a CoC on that basis and determined that it had failed due to the number of off-topic messages on the mailing lists he subscribes to. Being so self-centered is strongly correlated with being in a position of power, he seems to sincerely believe that everything should be about him, that he is entitled to all manner of protection and that any rule which doesn t protect him is worthless. I believe that the purpose of all laws and regulations should be to protect those who are less powerful, the more powerful people can usually protect themselves. The benefit that powerful people receive from being part of a system that is based on rules is that organisations (clubs, societies, companies, governments, etc) can become larger and achieve greater things if people can trust in the system. When minority groups are discouraged from contributing and when people need to be concerned about protecting themselves from attack the scope of an organisation is reduced. When there is a certain minimum standard of treatment that people can expect then they will be more willing to contribute and more able to concentrate on their contributions when they don t expect to be attacked. The Public Interest When an organisation declares itself to be acting in the public interest (EG by including Public Interest in the name of the organisation) I think that we should expect even better treatment of minority groups. One might argue that a corporation should protect members of minority groups for the sole purpose of making more money (it has been proven that more diverse groups produce better quality work). But an organisation that s in the Public Interest should be expected to go way beyond that and protect members of minority groups as a matter of principle. When an organisation is declared to be operating in the Public Interest I believe that anyone who s so unable to control their bigotry that they can t refrain from being bigoted on the mailing lists should not be a member.
Related posts:
  1. Perfect Code vs Quite Good Code Some years ago I worked on a project where software...
  2. The Purpose of Planet Debian An issue that causes ongoing discussion is what is the...
  3. WTF Let s write all the code twice There is an interesting web site WorseThanFailure.com (with the slogan...

18 August 2015

Enrico Zini: standup-comedy-notes

My semi serious stand up comedy notes Video Disclaimers
Someone has said that it requires less mental effort to condemn than to think. (Emma Goldman, on several things including mailing list flamewars)
Fascinating A da's "Dogging" song. Look for "dogging etiquette" for more examples of code of conducts. Just don't take your computer for repair immediately afterwards . Introduction
Every daring attempt to make a great change in existing conditions, every lofty vision of new possibilities for the human race, has been labeled Utopian. (Emma Goldman, on the Debian Social Contract)
I am going to talk about many topics that we all know have so much in common:
  • Anarchism
  • Poliamory
  • BDSM
  • and Free Software
They are all, after all:
  • People
  • Consensually
  • Doing Things
  • Together
BDSM
A person is no less a slave because they are allowed to choose a new master once in a term of years. (Lysander Spooner about proprietary cloud service providers)
If you thought you've seen it all with recursive acronyms, here's a chain acronym: Bondage Discipline, Dominance Submission, Sado Masochism. Why I think BDSM is interesting: not (just) because of whips, but for having a lot of awareness about power releationships. Why should one accept from a coworker a level of abuse that would be considered a hard limit when negotiating with a trusted dom? The BDSM Free Software definition: "I refuse to be bound by software I cannot negotiate with". YKINMKBYKIOK (Your Kink Is Not My Kink But Your Kink Is Okay) is a nice example of dealing with diversity, and it also definitely solves the emacs vs vi debate. Comfort zones, safewords, traffic light flow control, safety. "No means no", unless there has been a long discussion first, and a safeword is in place, in which case "Fuzzy purple unicorn" means "no" "No means no", and if someone insists after a "no", it becomes harassment. "No means no" is a precondition for being able to say "yes": http://pervocracy.blogspot.de/2011/03/no-and-no-and-no-and-yes.html Aftercare! Aftercare! Release parties! High fives! Solidarity after flamewars or votes! Poliamory
If love does not know how to give and take without restrictions, it is not love, but a transaction that never fails to lay stress on a plus and a minus. (Emma Goldman, on volunteer projects)
Polyamory is the practice, desire, or acceptance of intimate relationships that are not exclusive with respect to other sexual or intimate relationships, with knowledge and consent of everyone involved. Compersion, n: the feeling you get when someone else also takes good care of one of your packages. We currently allow only one value in the Maintainer field: * takeover is traumatic, because values can only be replaced * if values could be added instead, and removed when they don't make sense anymore... What is your definition of love? My current one is: my world is better with you in it. Relationship anarchy is the practice of forming relationships which are not bound by rules aside from what the people involved mutually agree on. How do you call a relationship that is bound by rules that the people involved do not agree on? From discussions after the talk New Relationship Energy, the excitement when you start to maintain a new package, and the risk of been carried away by the excitement and neglecting all the other ones. Consent
Anarchism, to me, means not only the denial of authority, not only a new economy, but a revision of the principles of morality. It means the development of the individual as well as the assertion of the individual. It means self-responsibility, and not leader worship. (Voltairine de Cleyre about trusting lintian warnings)
You need to know what you are doing, and what situation you're putting yourself into. You need to know that the person asking a question really is able to accept any answer, and take it seriously. You need to feel that you have alternatives. Be selfish when you ask, honest when you reply, and when others reply, take them seriously. If any of this doesn't stand, I find it hard to trust that we are in a consensual situation. When is one supposed to learn about consent?
  • I see little consensuality in standard education.
  • I see little consensuality at work.
Consent explained with tea. Practical advice
Anarchism has but one infallible, unchangeable motto, Freedom. Freedom to discover any truth, freedom to develop, to live naturally and fully. (Lucy Parsons about the DFSG)
Relationship advice and work advice have a lot in common: Relationship advice from 99 ways to ruin an open source project Online participation advice from How to Screw Up Your Relationship (and make everyone miserable while you re at it) Packaging advice from BDSM Basics: 20 Unsolicited Tips for New Dominants Advice about joining a new community from Advice to a newbie submissive about dominants Dear Debian, and dear everyone contributing to it: my world is better with you in it. I love you all :* <3

17 August 2015

Iustin Pop: Nikkor 300mm f/4E tests and happy dogs!

I recently got a new lens, which I'm still trying to learn how to properly use . The new lens is the Nikon 300m f/4E PF ED VR (wow, that's a mouthful of acronyms ) and it's as good as I expected, except that it seems that it's too good for the person using it (me). Physically, the lens is indeed very light and very easy to hand-hold; it's barely heavier and longer than the 70-300mm f/4-5.6 zoom, although a bit more bulky. I was fearing something "bigger" and heavier for a fixed-aperture lens, even after hearing all the good reports about it, but it was better than I expected. I was also slightly concerned, this being a prime lens, about finding the subject fast, especially if smaller or far away. It turned out that if the lens is focused at somewhat the right distance (not at the opposite end), it is not a real problem. So, as the first test/learning exercise, I did a couple of hour long walks outdoor, photographing random subjects. First day I used the lens by itself, on the second day I added the Nikon TC-14E(III) tele-converter, for a 420mm focal length. All pictures below except the Huskies are non-cropped, just down-sampled (they have the original field of view); the Husky pictures were cropped. In all cases, camera was set to semi-manual mode, mostly at 1/1000s or 1/2000s, various apertures between f/4 and f/8; this means wide open or stopped down one (when with the tele-converter) or two stops. Semi-manual as auto-ISO was on. Lens focus limiter on, VR either active or sport (to help with the viewfinder image mostly, at this speed), camera set to AF-C and group mode most of the time. Also, all pictures were shot hand-held, with bad technique - until somebody stopped me, turns out he was a photographer, and showed me how to properly hold a telephoto lens; thanks, whoever you were! The "random subjects" parts worked only somewhat, as (at first) the subjects were not too interesting; air-plane flying high above - check, fast focus (although it is easy to focus on a plane in clear sky), or close, static subjects: Airplane in the sky Test of close focusing Birds flying around - almost check, fast focus (both with just the lens and with the TC), although I learned that shooting flying birds while panning needs 1/2000s - my initial attempt at 1/1000s was not enough to eliminate either panning blur or wing-tip blur (both depend on the speed of the respective action); in any case, initial focus acquisition is speedy enough for me. A couple of examples (first without, second with the TC; in the second one, I was unprepared when the I saw the bird from the corner of my eye, and managed to bring up the camera and take exactly one photo before the bird was gone): Not enough speed Too much blur Another attempt was to focus on dogs jumping in the water. Here the problem was different: focusing on the correct object! The splashing around meant that there were lots of water droplets both in front and behind the dog, the dog being partially obscured by them. I don't know what AF settings would have helped here (maybe 3D tracking pre-focused on the black dog)? I had a low success rate on the first attempt, so I'll have to try again. The lens did correctly track the wrong thing, though . Examples: Almost focused Wrong focus Once the dog caught its target, the lens focused well enough, of course, but not perfect; it might need some AF fine-tuning or photographer upgrade : Not quite perfect focus At one point, I was lucky to have a dog running towards me; focus tracking on a subject approaching the camera is a good test for the lens (and camera) auto-focus performance. I was pleasantly surprised, given my failure to track left-to-right flying birds (which should be easier!): Run! Not yet tired! In any case, I was walking and thinking on how to improve on my technique, when I saw in the distance a pair of dogs (Husky's, I believe) which were very active. This case was relatively easy for the camera and lens, since they were moving only left-to-right, instead of forward-backward, so the focus distance was mostly constant, and I just had to pan. Full gallery here, two examples: Jump! Play over, going home These two dogs had an education value beyond the "how is the lens performing"; they were so fast, active (they kept going at it non-stop for a good number of minutes) and playful that they made my day brighter . So, that concludes my first test of this lens and the 1.4 tele-converter. Conclusion: equipment good, need to upgrade the photographer!

Lars Wirzenius: On the cost analysis of dependencies

A question that I'm asked repeatedly recently is why I chose not to use an existing library for serialising data structures in Obnam. This blog post is the answer. Obnam is a backup program, and it needs to store various data about files. This includes stat(2) information about each file in the live data, as well as data Obnam needs to keep track of everything. At run-time, Obnam keeps this data in memory data structures, such as Python dicts. For storage, these data structures need to be converted, serialised, to and from streams of bytes. The are a variety of libraries for doing this, designed for different purposes and with their own constraints and pitfalls. Python's standard library comes with the cPickle library, for example, but its serialisation format is not guaranteed to be compatible with any other version of Python. For Obnam, I need something that will last a long time. I do not want to have to deal with a library changing its serialisation format, as that would mean either that Obnam can't handle old backups, or that I need to start maintaining the old version of the library. A way to look at this is that any dependencies your software have a cost, and that cost should be smaller than the benefit you get from them. For example, Obnam depends on the paramiko library to implement the SSH protocol. This library has some cost, and I've run into one or two bugs in it that have been rather unfortunate. However, the benefit it brings is huge: I don't have to implement SSH myself. I'm happy to have Obnam depend on paramiko. For the serialisation thing, I wrote my own library, after a small about of research into existing ones. Research time is a cost, too. Mine is somewhat Obnam specific, in that it can make some assumptions about the data to be serialised, and this allows a simpler library. A generic library would have to handle a number of special cases that mine can ignore. It took me less than an hour to write this twice. I first wrote a quick prototype and a little microbenchmark to see if my approach would be feasible. Then I deleted that code, and started from scratch, TDD style, to make sure the code was reliable. The cost of writing my own serialisation code was less than the cost of finding, let alone evaluating existing libraries. It may be that my own library turns out to be inadequate. Then, and only then, is when I start researching other libraries. Until then, I'll avoid the cost of research to find a suitable library, the cost of learning the chosen one, the cost of integrating it into Obnam, the cost to porters of Obnam of dealing with a new dependency, and the risk of the library changing in ways that are unsuitable for Obnam. Obviously, writing your own code has costs, too. Designing and implementing a library is a cost, as is maintaining it (debugging, changes in requirements, etc). Write your own or use existing code? It's a cost/benefit analysis. There's no clear one answer that's always correct.

Next.

Previous.