Search Results: "spa"

8 September 2015

Eriberto Mota: Debian: how to use blhc to solve hardening issues when packaging

Implementing the hardening When packaging in Debian, is very common to see the lintian messages 'hardening-no-relro' and 'hardening-no-fortify-functions' in some softwares written in C or C++. To solve these issues, we can use the 'blhc' tool (apt-get install blhc). Please, get the revision 1.11-9 of the icmpinfo package. You can get this revision from http://snapshot.debian.org or from http://eriberto.pro.br/debian/icmpinfo. As a shortcut, you can use the following command: 
$ dget -u http://eriberto.pro.br/debian/icmpinfo/icmpinfo_1.11-9.dsc
The icmpinfo 1.11-9 is almost clean for lintian (in 2015-09-07, Standards-Version 3.9.6). The only problem is: 
W: icmpinfo: hardening-no-relro usr/sbin/icmpinfo
To track the problem I will use blhc over the .build file: 
$ blhc icmpinfo_1.11-9_amd64.build
LDFLAGS missing (-Wl,-z,relro): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -o icmpinfo recvping.o print.o err.o icmpinfo.o pid.o
Note that the problem is some missing options (-Wl,-z,relro) for LDFLAGS when building icmpinfo (for newbies, in GCC, -o is used to indicate the name to be used for the final binary after the compilation). If you are using the DebHelper compat 9 (debian/compat=9) and the DebHelper 9 (debhelper >= 9 in Build-Depends field in d/control), some variables as CFLAGS, LDFLAGS, CPPFLAGS and CXXFLAGS will be automatically passed during calls to dh_auto_* programs (yes, you should use the new and reduced d/rules format - see as example the debian/rules of the icmpinfo 1.11-9; if you still have doubts, $ man dh). Now, we need discover the reason why the LDFLAGS is being changed between its generation by the Debian build system and its utilization by the upstream's source code. So, we need to check the upstream Makefile. There is in Makefile (after a 'quilt push -a', to apply all current patches): 
LDFLAGS= $(CFLAGS)
OBJECTS= recvping.o print.o err.o icmpinfo.o pid.o
TARGET = icmpinfo
$(TARGET): $(OBJECTS)
 $(CC) $(LDFLAGS) -o $@ $(OBJECTS) $(LDLIBS)
Hummm... The LDFLAGS content generated by Debian is being dropped by Makefile because it is saying that "LDFLAGS = CFLAGS content". This line is a problem because the upstream Makefile needs to take and use the CFLAGS and LDFLAGS independently. To fix the issue, you can use this patch: 
--- icmpinfo-1.11.orig/Makefile
+++ icmpinfo-1.11/Makefile
@@ -20,13 +20,13 @@ VERS = 1.11
 
 RM = rm -f
 
-LDFLAGS= $(CFLAGS)
+#LDFLAGS= $(CFLAGS)
 
 OBJECTS= recvping.o print.o err.o icmpinfo.o pid.o
 TARGET = icmpinfo
 
 $(TARGET): $(OBJECTS)
- $(CC) $(LDFLAGS) -o $@ $(OBJECTS) $(LDLIBS)
+ $(CC) $(LDFLAGS) $(CFLAGS) -o $@ $(OBJECTS) $(LDLIBS)
 
 tgz: clean
 rm -f CHECKSUMS.asc
After a 'debuild' is a fact the problem is solved and the lintian is happy. See the blhc results: 
$ blhc ../icmpinfo_1.11-9_amd64.build
$
Now, we can improve the hardening. To see the current status, we can use the 'blhc --all' command. See here: 
blhc --all ../icmpinfo_1.11-9_amd64.build
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o recvping.o recvping.c
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o print.o print.c
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o err.o err.c
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o icmpinfo.o icmpinfo.c
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o pid.o pid.c
LDFLAGS missing (-fPIE -pie -Wl,-z,now): cc -Wl,-z,relro -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -o icmpinfo recvping.o print.o err.o icmpinfo.o pid.o
Well, we know that CFLAGS and LDFLAGS are present. Now, we can force the DebHelper to pass some extra options to make hardening better. Generally, is only needed to add the following line to debian/rules: 
export DEB_BUILD_MAINT_OPTIONS = hardening=+all
See the results (after a new debuild command): 
$ blhc --all ../icmpinfo_1.11-9_amd64.build
$
More examples Let me to show other example. I will use the mac-robber 1.02-3 (however, I disabled the Makefile.patch in debian/patches/series). After a debuild, the following lintian messages are presented: 
W: mac-robber: hardening-no-relro usr/bin/mac-robber
I: mac-robber: hardening-no-fortify-functions usr/bin/mac-robber
Using blhc: 
$ blhc ../mac-robber_1.02-3_amd64.build 
CFLAGS missing (-g -O2 -fstack-protector-strong -Wformat -Werror=format-security): gcc -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
CPPFLAGS missing (-D_FORTIFY_SOURCE=2): gcc -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
LDFLAGS missing (-Wl,-z,relro): gcc -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
We need to verify what is the problem in Makefile with CFLAGS, CPPFLAGS and LDFLAGS when generating the binary 'mac-robber' (just recalling, -o mac-robber in GCC command). See: 
linux_notstatic: 
 $(CC) -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
There are no references to CFLAGS, CPPFLAGS and LDFLAGS. To solve the problem, we need patch the Makefile to make this: 
linux_notstatic: 
 $(CC) $(CFLAGS) $(LDFLAGS) $(CPPFLAGS) -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
As last example, is possible that the Makefile is overriding the content sent by DebHelper when building. See this line from a hypothetical Makefile: 
CFLAGS = -g -Wall
As you can see, the Makefile is redefining CFLAGS; consequently, it is discarding any previous content sent by DebHelper. To solve this issue, we can use the following patch: 
-CFLAGS = -g -Wall
+CFLAGS += -g -Wall
So, the content received from DebHelper will be added to '-g -Wall'. Default parameters As curiosity, to see the basic parameters created by DebHelper as hardening, use the command: 
$ dpkg-buildflags
To see the all parameters, use the command: 
$ DEB_BUILD_MAINT_OPTIONS=hardening=+all dpkg-buildflags
More information More information about the hardening can be viewed at two places: https://wiki.debian.org/Hardening https://wiki.debian.org/HardeningWalkthrough I hope this help. Enjoy!

7 September 2015

Ben Armstrong: Hike at Blomidon Park: Late Summer, 2015

I had the wonderful privilege to go camping and hiking with my kids scouting group, the Pathfinders of Tantallon SDA church. The day started with a quick trip to Pugwash with one of the leaders to bring back some chairs to their school, and then we headed back out to Blomidon to meet up with the group. Click the photo below to start the slideshow.
The road trip started early to fetch some chairsThe road trip started early to fetch some chairs click to start
The smudgy truck windows make an interesting filter Still an hour or more away from our first stop More funky filtering, this time with trees participating Wentworth valley taken over the cluttered dash We disturbed a great blue heron s breakfast at Wallace The beach at Pugwash SDA Camp where we loaded the chairs Trucking along past Truro After dropping off chairs, finally approaching Blomidon Getting very close to Blomidon Some bikers out to enjoy the views Interesting white berries Interesting red berries My first up close look at the point with my hiking buddy, Dave, on the first day An experimental panorama. Not sure I have the knack for keeping the horizon straight. We must bring the group out here tomorrow! Fast ringneck snake! Hard to get a clear shot Pre-dawn over the campground The first blush of coming dawn The Moon and Venus just before dawn Evergreens surrounding our camp site, pre-dawn Seems I m still the only one up Half of the tents on the spacious group site Half of the tents on the spacious group site My daughter, the artist My two youngest and their best friend Some relaxing down time after breakfast Not sure who said what, but apparently they were hilarious. :) The smoke was a bit much for my eldest Geoff entertaining the troops Dave making breakfast Dave making breakfast Breakfast just wrapping up Relaxing while we finish breakfast Looks like that needs some tweaking A bit too smoky The whole group The whole group Just goofing around Who s winning? Enjoying the last embers of the breakfast fire before heading to Jodrey Trail I admire this young lady s great eye for photography She has some sweet gear A lot of old hardwoods out here Dave did this hike with me yesterday Excellent hiking buddy! Words can t describe how much more stunning these views are in person All the cameras came out Got to get that perfect shot! A tree clinging to the eroding ground above the sheer cliff A lookoff on Jodrey Trail A lookoff on Jodrey Trail Lining up her shot /a> A lookoff on Jodrey Trail A lookoff on Jodrey Trail A lookoff on Jodrey Trail A fern with sharply serrated sturdy leaves I m not familiar with Breaking camp at group site 404 One final chance to enjoy the view from the park entrance before heading home
facebooktwittergoogle_plusredditpinterestlinkedinmail

5 September 2015

Lisandro Damián Nicanor Pérez Meyer: Important Akonadi fix in today's Debian Jessie's update (aka 8.2)

Todays Debian Jessie's update brings a fix in Akonadi that you certainly want in your system.

There was a bug in Akonadi that made it leak files. And if you use Kmail you will certainly want to keep reading: most of us who tested it before pushing it to testing (and now to stable) removed more than 4 GiB of useless data from our homes.

The bug that makes Akonadi leak files gets solved with the latest stable update (and has been in testing for a couple of months already). But you need to purge the leaked files. It's pretty easy: with your normal user account just run:

akonadictl fsck

That's all. After a while you will get back a lot of disk space. Note that you don't need the Akonadi fix in order to run this tool and recover your space. The fix just makes sure this won't happen again.

Niels Thykier: I accidentally dak

So, yesterday, I unbroke dak twice even! It is of course slightly less awesome that one of the broken parts was in code written by yours truly. Anyhow: Unbreaking the dak auto-decrufter You may remember the auto-decrufter, which I added to dak. As a safety measure, it bails out when in doubt about which removal breaks what package. Turns out it was often in doubt, because the code had a bug. Of course, nothing that could not be solved with a patch. Thanks to Ansgar for merging this. :) Unbreaking dak generate-releases As a part of migrating apt-file to use APTs new acquire system (from experimental), I learned APT really likes having checksums for everything. Now including checksums for both the compressed file and the uncompressed file. Sadly, dak had optimised out the uncompressed checksums for Contents files, but even after removing that optimisation (and Ganneff unbreaking my dinstall breakage) some Contents files still did not have an checksum for the uncompressed Contents file. After some sophisticated debugging (read: printf-debugging ), I finally discovered the issue and submitted a patch. Thanks to Ansgar and Ganneff for merging (and fixing my dinstall breakage).
Filed under: Debian

4 September 2015

Julien Danjou: Data validation in Python with voluptuous

Continuing my post series on the tools I use these days in Python, this time I would like to talk about a library I really like, named voluptuous. It's no secret that most of the time, when a program receives data from the outside, it's a big deal to handle it. Indeed, most of the time your program has no guarantee that the stream is valid and that it contains what is expected. The robustness principle says you should be liberal in what you accept, though that is not always a good idea neither. Whatever policy you chose, you need to process those data and implement a policy that will work lax or not. That means that the program need to look into the data received, check that it finds everything it needs, complete what might be missing (e.g. set some default), transform some data, and maybe reject those data in the end.
Data validation The first step is to validate the data, which means checking all the fields are there and all the types are right or understandable (parseable). Voluptuous provides a single interface for all that called a Schema. 
>>> from voluptuous import Schema
>>> s = Schema( 
...   'q': str,
...   'per_page': int,
...   'page': int,
...  )
>>> s( "q": "hello" )
 'q': 'hello' 
>>> s( "q": "hello", "page": "world" )
voluptuous.MultipleInvalid: expected int for dictionary value @ data['page']
>>> s( "q": "hello", "unknown": "key" )
voluptuous.MultipleInvalid: extra keys not allowed @ data['unknown']

The argument to voluptuous.Schema should be the data structure that you expect. Voluptuous accepts any kind of data structure, so it could also be a simple string or an array of dict of array of integer. You get it. Here it's a dict with a few keys that if present should be validated as certain types. By default, Voluptuous does not raise an error if some keys are missing. However, it is invalid to have extra keys in a dict by default. If you want to allow extra keys, it is possible to specify it. 
>>> from voluptuous import Schema
>>> s = Schema( "foo": str , extra=True)
>>> s( "bar": 2 )
 "bar": 2

It is also possible to make some keys mandatory. 
>>> from voluptuous import Schema, Required
>>> s = Schema( Required("foo"): str )
>>> s( )
voluptuous.MultipleInvalid: required key not provided @ data['foo']

You can create custom data type very easily. Voluptuous data types are actually just functions that are called with one argument, the value, and that should either return the value or raise an Invalid or ValueError exception. 
>>> from voluptuous import Schema, Invalid
>>> def StringWithLength5(value):
...     if isinstance(value, str) and len(value) == 5:
...             return value
...     raise Invalid("Not a string with 5 chars")
...
>>> s = Schema(StringWithLength5)
>>> s("hello")
'hello'
>>> s("hello world")
voluptuous.MultipleInvalid: Not a string with 5 chars

Most of the time though, there is no need to create your own data types. Voluptuous provides logical operators that can, combined with a few others provided primitives such as voluptuous.Length or voluptuous.Range, create a large range of validation scheme. 
>>> from voluptuous import Schema, Length, All
>>> s = Schema(All(str, Length(min=3, max=5)))
>>> s("hello")
"hello"
>>> s("hello world")
voluptuous.MultipleInvalid: length of value must be at most 5

The voluptuous documentation has a good set of examples that you can check to have a good overview of what you can do. Data transformation
What's important to remember, is that each data type that you use is a function that is called and returns a value, if the value is considered valid. That value returned is what is actually used and returned after the schema validation: 
>>> import uuid
>>> from voluptuous import Schema
>>> def UUID(value):
...     return uuid.UUID(value)
...
>>> s = Schema( "foo": UUID )
>>> data_converted = s( "foo": "uuid?" )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']
>>> data_converted = s( "foo": "8B7BA51C-DFF5-45DD-B28C-6911A2317D1D" )
>>> data_converted
 'foo': UUID('8b7ba51c-dff5-45dd-b28c-6911a2317d1d')

By defining a custom UUID function that converts a value to a UUID, the schema converts the string passed in the data to a Python UUID object validating the format at the same time. Note a little trick here: it's not possible to use directly uuid.UUID in the schema, otherwise Voluptuous would check that the data is actually an instance of uuid.UUID: 
>>> from voluptuous import Schema
>>> s = Schema( "foo": uuid.UUID )
>>> s( "foo": "8B7BA51C-DFF5-45DD-B28C-6911A2317D1D" )
voluptuous.MultipleInvalid: expected UUID for dictionary value @ data['foo']
>>> s( "foo": uuid.uuid4() )
 'foo': UUID('60b6d6c4-e719-47a7-8e2e-b4a4a30631ed')

And that's not what is wanted here. That mechanism is really neat to transform, for example, strings to timestamps. 
>>> import datetime
>>> from voluptuous import Schema
>>> def Timestamp(value):
...     return datetime.datetime.strptime(value, "%Y-%m-%dT%H:%M:%S")
...
>>> s = Schema( "foo": Timestamp )
>>> s( "foo": '2015-03-03T12:12:12' )
 'foo': datetime.datetime(2015, 3, 3, 12, 12, 12) 
>>> s( "foo": '2015-03-03T12:12' )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']

Recursive schemas So far, Voluptuous has one limitation so far: the ability to have recursive schemas. The simplest way to circumvent it is by using another function as an indirection. 
>>> from voluptuous import Schema, Any
>>> def _MySchema(value):
...     return MySchema(value)
...
>>> from voluptuous import Any
>>> MySchema = Schema( "foo": Any("bar", _MySchema) )
>>> MySchema( "foo":  "foo": "bar" )
 'foo':  'foo': 'bar' 
>>> MySchema( "foo":  "foo": "baz" )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']['foo']

Usage in REST API
I started to use Voluptuous to validate data in a the REST API provided by Gnocchi. So far it has been a really good tool, and we've been able to create a complete REST API that is very easy to validate on the server side. I would definitely recommend it for that. It blends with any Web framework easily. One of the upside compared to solution like JSON Schema, is the ability to create or re-use your own custom data types while converting values at validation time. It is also very Pythonic, and extensible it's pretty great to use for all of that. It's also not tied to any serialization format. On the other hand, JSON Schema is language agnostic and is serializable itself as JSON. That makes it easy to be exported and provided to a consumer so it can understand the API and validate the data potentially on its side.

Norbert Preining: How to rename a local OfflineIMAP managed folder

Since quite some time I am using OfflineIMAP to keep my mailbox status from various servers locally available on my machine. I did more or less follow the excellent guide A unix style mail setup, but didn t use folder name translation right from the beginning. So I got stuck with four folders named INBOX: acc1/INBOX, acc2/INBOX, etc. This wouldn t be a big problem, but when using Mutt with the folder sidebar patch (as available in the mutt-patched package in Debian), there is the problem that only the INBOX part is shown. offlineimap So I decided to rename all the local folders to the scheme described in the above guide. At the end I want to have a folder layout as follows: 
  Maildir/acc1/acc1
  Maildir/acc1/acc1.folder1
  Maildir/acc1/acc1.folder2
where the acc1 related to the INBOX on the remote account acc1, and the others are folders on the remote end. Unfortunately, renaming is not supported in offlineimap, it would simply download everything again at best! So I checked what needs to be done, and it turned out it is not as complicated as I thought. The necessary steps are: To make this more specific, let us assume I want to reorganize the account acc1 in this way, and there are two folders synced. That is, before we start the Maildir folder contains: 
Maildir/acc1/INBOX
Maildir/acc1/folder1
Maildir/acc1/folder2
and we want to achieve that the layout is 
Maildir/acc1/acc1
Maildir/acc1/acc1.folder1
Maildir/acc1/acc1.folder2
Due to the fact that both the LocalStatus files and the FolderValidity files are naned in the same way as the folders, the following trivial script does all the necessary renaming for this simple case: 
for d in ~/Maildir/acc1 ~/.offlineimap/Account-acc1/LocalStatus ~/.offlineimap/Repository-acc1local/FolderValidity ; do
  mv $d/INBOX $d/acc1
  mv $d/folder1 $d/acc1.folder1
  mv $d/folder2 $d/acc1.folder2
done
Finally, the correct renaming has to be done by adding nametrans rules to the helper script. The functions oimaptransfolder_acc1 and localtransfolder_acc1 as defined in the above tutorial can be used as is, and need only be added to the respective Repository sections in the configuration file. After having done the necessary renaming, what remains is to give it a good test. Best way to do it is with 
offlineimap --dry-run -o -a acc1
which runs the syncronization in dry-run mode, so nothing is actually copied, and only once and only for acc1. If during this test run nothing strange appears, like many many emails are deleted on either end, then you are fine. Enjoy.

Dirk Eddelbuettel: RcppArmadillo 0.5.500.2.0

armadillo image Once again time for the monthly upstream Armadillo update -- version 5.500.2 was released earlier today by Conrad. And a new and matching RcppArmadillo release 0.5.500.2.0 in already on CRAN and will go to Debian shortly. Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. This release contains mostly bug fixes and some internal code refactoring:
Changes in RcppArmadillo version 0.5.500.2.0 (2015-09-03)
  • Upgraded to Armadillo 5.500.2 ("Molotov Cocktail")
    • expanded object constructors and generators to handle size() based specification of dimensions
    • faster handling of submatrix rows
    • faster clamp()
    • fixes for handling sparse matrices
Courtesy of CRANberries, there is also a diffstat report for the most recent CRAN release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

2 September 2015

Luca Falavigna: Resource control with systemd

I m receiving more requests for upload accounts to the Deb-o-Matic servers lately (yay!), but that means the resources need to be monitored and shared between the build daemons to prevent server lockups. My servers are running systemd, so I decided to give systemd.resource-control a try. My goal was to assign lower CPU shares to the build processes (debomatic itself, sbuild, and all the related tools), in order to avoid blocking other important system services from being spawned when necessary. I created a new slice, and set a lower CPU share weight:
$ cat /etc/systemd/system/debomatic.slice
[Slice]
CPUAccounting=true
CPUShares=512
$
 Then, I assigned the slice to the service unit file controlling debomatic daemons by adding the Slice=debomatic.slice option under the Service directive. That was not enough, though, as some processes were assigned to the user slice instead, which groups all the processes spawned by users:
systemd-cgls This is probably because schroot spawns a login shell, and systemd considers it belonging to a different process group. So, I had to launch the command systemctl set-property user.slice CPUShares=512, so all processes belonging to the user.slice will receive the same share of the debomatic ones. I consider this a workaround, I m open to suggestions how to properly solve this issue :) I ll try to explore more options in the coming days, so I can improve my knowledge of systemd a little bit more :)

1 September 2015

Lunar: Reproducible builds: week 18 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Aur lien Jarno uploaded glibc/2.21-0experimental1 which will fix the issue were locales-all did not behave exactly like locales despite having it in the Provides field. Lunar rebased the pu/reproducible_builds branch for dpkg on top of the released 1.18.2. This made visible an issue with udebs and automatically generated debug packages. The summary from the meeting at DebConf15 between ftpmasters, dpkg mainatainers and reproducible builds folks has been posted to the revelant mailing lists. Packages fixed The following 70 packages became reproducible due to changes in their build dependencies: activemq-activeio, async-http-client, classworlds, clirr, compress-lzf, dbus-c++, felix-bundlerepository, felix-framework, felix-gogo-command, felix-gogo-runtime, felix-gogo-shell, felix-main, felix-shell-tui, felix-shell, findbugs-bcel, gco, gdebi, gecode, geronimo-ejb-3.2-spec, git-repair, gmetric4j, gs-collections, hawtbuf, hawtdispatch, jack-tools, jackson-dataformat-cbor, jackson-dataformat-yaml, jackson-module-jaxb-annotations, jmxetric, json-simple, kryo-serializers, lhapdf, libccrtp, libclaw, libcommoncpp2, libftdi1, libjboss-marshalling-java, libmimic, libphysfs, libxstream-java, limereg, maven-debian-helper, maven-filtering, maven-invoker, mochiweb, mongo-java-driver, mqtt-client, netty-3.9, openhft-chronicle-queue, openhft-compiler, openhft-lang, pavucontrol, plexus-ant-factory, plexus-archiver, plexus-bsh-factory, plexus-cdc, plexus-classworlds2, plexus-component-metadata, plexus-container-default, plexus-io, pytone, scolasync, sisu-ioc, snappy-java, spatial4j-0.4, tika, treeline, wss4j, xtalk, zshdb. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: Chris Lamb also noticed that binaries shipped with libsilo-bin did not work. Documentation update Chris Lamb and Ximin Luo assembled a proper specification for SOURCE_DATE_EPOCH in the hope to convince more upstreams to adopt it. Thanks to Holger it is published under a non-Debian domain name. Lunar documented easiest way to solve issues with file ordering and timestamps in tarballs that came with tar/1.28-1. Some examples on how to use SOURCE_DATE_EPOCH have been improved to support systems without GNU date. reproducible.debian.net armhf is finally being tested, which also means the remote building of Debian packages finally works! This paves the way to perform the tests on even more architectures and doing variations on CPU and date. Some packages even produce the same binary Arch:all packages on different architectures (1, 2). (h01ger) Tests for FreeBSD are finally running. (h01ger) As it seems the gcc5 transition has cooled off, we schedule sid more often than testing again on amd64. (h01ger) disorderfs has been built and installed on all build nodes (amd64 and armhf). One issue related to permissions for root and unpriviliged users needs to be solved before disorderfs can be used on reproducible.debian.net. (h01ger) strip-nondeterminism Version 0.011-1 has been released on August 29th. The new version updates dh_strip_nondeterminism to match recent changes in debhelper. (Andrew Ayer) disorderfs disorderfs, the new FUSE filesystem to ease testing of filesystem-related variations, is now almost ready to be used. Version 0.2.0 adds support for extended attributes. Since then Andrew Ayer also added support to reverse directory entries instead of shuffling them, and arbitrary padding to the number of blocks used by files. Package reviews 142 reviews have been removed, 48 added and 259 updated this week. Santiago Vila renamed the not_using_dh_builddeb issue into varying_mtimes_in_data_tar_gz_or_control_tar_gz to align better with other tag names. New issue identified this week: random_order_in_python_doit_completion. 37 FTBFS issues have been reported by Chris West (Faux) and Chris Lamb. Misc. h01ger gave a talk at FrOSCon on August 23rd. Recordings are already online. These reports are being reviewed and enhanced every week by many people hanging out on #debian-reproducible. Huge thanks!

31 August 2015

Matthew Garrett: Working with the kernel keyring

The Linux kernel keyring is effectively a mechanism to allow shoving blobs of data into the kernel and then setting access controls on them. It's convenient for a couple of reasons: the first is that these blobs are available to the kernel itself (so it can use them for things like NFSv4 authentication or module signing keys), and the second is that once they're locked down there's no way for even root to modify them.

But there's a corner case that can be somewhat confusing here, and it's one that I managed to crash into multiple times when I was implementing some code that works with this. Keys can be "possessed" by a process, and have permissions that are granted to the possessor orthogonally to any permissions granted to the user or group that owns the key. This is important because it allows for the creation of keyrings that are only visible to specific processes - if my userspace keyring manager is using the kernel keyring as a backing store for decrypted material, I don't want any arbitrary process running as me to be able to obtain those keys[1]. As described in keyrings(7), keyrings exist at the session, process and thread levels of granularity.

This is absolutely fine in the normal case, but gets confusing when you start using sudo. sudo by default doesn't create a new login session - when you're working with sudo, you're still working with key posession that's tied to the original user. This makes sense when you consider that you often want applications you run with sudo to have access to the keys that you own, but it becomes a pain when you're trying to work with keys that need to be accessible to a user no matter whether that user owns the login session or not.

I spent a while talking to David Howells about this and he explained the easiest way to handle this. If you do something like the following:
$ sudo keyctl add user testkey testdata @u
a new key will be created and added to UID 0's user keyring (indicated by @u). This is possible because the keyring defaults to 0x3f3f0000 permissions, giving both the possessor and the user read/write access to the keyring. But if you then try to do something like:
$ sudo keyctl setperm 678913344 0x3f3f0000
where 678913344 is the ID of the key we created in the previous command, you'll get permission denied. This is because the default permissions on a key are 0x3f010000, meaning that the possessor has permission to do anything to the key but the user only has permission to view its attributes. The cause of this confusion is that although we have permission to write to UID 0's keyring (because the permissions are 0x3f3f0000), we don't possess it - the only permissions we have for this key are the user ones, and the default state for user permissions on new keys only gives us permission to view the attributes, not change them.

But! There's a way around this. If we instead do:
$ sudo keyctl add user testkey testdata @s
then the key is added to the current session keyring (@s). Because the session keyring belongs to us, we possess any keys within it and so we have permission to modify the permissions further. We can then do:
$ sudo keyctl setperm 678913344 0x3f3f0000
and it works. Hurrah! Except that if we log in as root, we'll be part of another session and won't be able to see that key. Boo. So, after setting the permissions, we should:
$ sudo keyctl link 678913344 @u
which ties it to UID 0's user keyring. Someone who logs in as root will then be able to see the key, as will any processes running as root via sudo. But we probably also want to remove it from the unprivileged user's session keyring, because that's readable/writable by the unprivileged user - they'd be able to revoke the key from underneath us!
$ sudo keyctl unlink 678913344 @s
will achieve this, and now the key is configured appropriately - UID 0 can read, modify and delete the key, other users can't.

This is part of our ongoing work at CoreOS to make rkt more secure. Moving the signing keys into the kernel is the first step towards rkt no longer having to trust the local writable filesystem[2]. Once keys have been enrolled the keyring can be locked down - rkt will then refuse to run any images unless they're signed with one of these keys, and even root will be unable to alter them.

[1] (obviously it should also be impossible to ptrace() my userspace keyring manager)
[2] Part of our Secure Boot work has been the integration of dm-verity into CoreOS. Once deployed this will mean that the /usr partition is cryptographically verified by the kernel at runtime, making it impossible for anybody to modify it underneath the kernel. / remains writable in order to permit local configuration and to act as a data store, and right now rkt stores its trusted keys there.

comment count unavailable comments

Martín Ferrari: Romania

It's been over 2 years since I decided to start a new, nomadic life. I had the idea of blogging about this experience as it happened, but not only I am incredibly lazy when it comes to writing, most of the time I have been too busy just enjoying this lifestyle! The TL;DR version of these last 2 years: And now, I am in Cluj-Napoca, Romania.
View from my window
View from my window
2 comments

Russ Allbery: Review: Through Struggle, The Stars

Review: Through Struggle, The Stars, by John J. Lumpkin
Series: Human Reach #1
Publisher: John J. Lumpkin
Copyright: July 2011
ISBN: 1-4611-9544-6
Format: Kindle
Pages: 429
Never let it be said that I don't read military SF. However, it can be said that I read books and then get hellishly busy and don't review them for months. So we'll see if I can remember this well enough to review it properly. In Lumpkin's future world, mankind has spread to the stars using gate technology, colonizing multiple worlds. However, unlike most science fiction futures of this type, it's not all about the United States, or even the United States and Russia. The great human powers are China and Japan, with the United States relegated to a distant third. The US mostly maintains its independence from either, and joins the other lesser nations and UN coalitions to try to pick up a few colonies of its own. That's the context in which Neil and Rand join the armed services: the former as a pilot in training, and the latter as an army grunt. This is military SF, so of course a war breaks out. But it's a war between Japan and China: improved starship technology and the most sophisticated manufacturing in the world against a much larger economy with more resources and a larger starting military. For reasons that are not immediately clear, and become a plot point later on, the United States president immediately takes an aggressive tone towards China and pushes the country towards joining the war on the side of Japan. Most of this book is told from Neil's perspective, following his military career during the outbreak of war. His plans to become a pilot get derailed as he gets entangled with US intelligence agents (and a bad supervisor). The US forces are not in a good place against China, struggling when they get into ship-to-ship combat, and Neil's ship goes on a covert mission to attempt to complicate the war with political factors. Meanwhile, Rand tries to help fight off a Chinese invasion of one of the rare US colony worlds. Through Struggle, The Stars does not move quickly. It's over 400 pages, and it's a bit surprising how little happens. What it does instead is focus on how the military world and the war feels to Neil: the psychological experience of wanting to serve your country but questioning its decisions, the struggle of working with people who aren't always competent but who you can't just make go away, the complexities of choosing a career path when all the choices are fraught with politics that you didn't expect to be involved in, and the sheer amount of luck and random events involved in the progression of one's career. I found this surprisingly engrossing despite the slow pace, mostly because of how true to life it feels. War is not a never-ending set of battles. Life in a military ship has moments when everything is happening far too quickly, but other moments when not much happens for a long time. Lumpkin does a great job of reflecting that. Unfortunately, I thought there were two significant flaws, one of which means I probably won't seek out further books in this series. First, one middle portion of the book switches away from Neil to follow Rand instead. The first part of that involves the details of fighting orbiting ships with ground-based lasers, which was moderately interesting. (All the technological details of space combat are interesting and thoughtfully handled, although I'm not the sort of reader who will notice more subtle flaws. But David Weber this isn't, thankfully.) But then it turns into a fairly generic armed resistance story, which I found rather boring. It also ties into the second and more serious flaw: the villain. The overall story is constructed such that it doesn't need a personal villain. It's about the intersection of the military and politics, and a war that may be ill-conceived but that is being fought anyway. That's plenty of conflict for the story, at least in my opinion. But Lumpkin chose to introduce a specific, named Chinese character in the villain role, and the characterization is... well. After he's humiliated early in the story by the protagonists, Li Xiao develops an obsession with killing them, for his honor, and then pursues them throughout the book in ways that are sometimes destructive to the Chinese war efforts. It's badly unrealistic compared to the tone of realism taken by the rest of the story. Even if someone did become this deranged, it's bizarre that a professional military (and China's forces are otherwise portrayed as fairly professional) would allow this. Li reads like pure caricature, and despite being moderately competent apart from his inexplicable (but constantly repeated) obsession, is cast in a mustache-twirling role of personal villainy. This is weirdly out of place in the novel, and started irritating me enough that it took me out of the story. Through Struggle, The Stars is the first book of a series, and does not resolve much by the end of the novel. That plus its length makes the story somewhat unsatisfying. I liked Neil's development, and liked him as a character, and those who like the details of combat mixed with front-lines speculation about politics may enjoy this. But a badly-simplified mustache-twirling victim and some extended, uninteresting bits mar the book enough that I probably won't seek out the sequels. Followed by The Desert of Stars. Rating: 6 out of 10

30 August 2015

Antonio Terceiro: DebConf15, testing debian packages, and packaging the free software web

This is my August update, and by the far the coolest thing in it is Debconf. Debconf15 I don t get tired of saying it is the best conference I ever attended. First it s a mix of meeting both new people and old friends, having the chance to chat with people whose work you admire but never had a chance to meet before. Second, it s always quality time: an informal environment, interesting and constructive presentations and discussions. This year the venue was again very nice. Another thing that was very nice was having so many kids and families. This was no coincidence, since this was the first DebConf in which there was organized childcare. As the community gets older, this a very good way of keeping those who start having kids from being alienated from the community. Of course, not being a parent yet I have no idea how actually hard is to bring small kids to a conference like DebConf. ;-) I presented two talks: There was also the now traditional Ruby BoF, where discussed the state and future of the Ruby ecosystem in Debian; and an in promptu Ruby packaging workshop where we introduced the basics of packaging in general, and Ruby packaging specifically. Besides shak, I was able to hack on a few cool things during DebConf: Miscellaneous updates

DebConf team: DebConf15: Farewell, and thanks for all the Fisch (Posted by DebConf Team)

A week ago, we concluded our biggest DebConf ever! It was a huge success. Handwritten feedback note We are overwhelmed by the positive feedback, for which we re very grateful. We want to thank you all for participating in the talks; speakers and audience alike, in person or live over the global Internet it wouldn t be the fantastic DebConf experience without you! Many of our events were recorded and streamed live, and are now available for viewing, as are the slides and photos. To share a sense of the scale of what all of us accomplished together, we ve compiled a few statistics: Our very own designer Valessio Brito made a lovely video of impressions and images of the conference.
We re collecting impressions from attendees as well as links to press articles, including Linux Weekly News coverage of specific sessions of DebConf. If you find something not yet included, please help us by adding links to the wiki.
DebConf15 group photo (by Aigars Mahinovs)
We tried a few new ideas this year, including a larger number of invited and featured speakers than ever before. On the Open Weekend, some of our sponsors presented their career opportunities at our job fair, which was very well attended. And a diverse selection of entertainment options provided the necessary breaks and ample opportunity for socialising. On the last Friday, the Oscar-winning documentary Citizenfour was screened, with some introductory remarks by Jacob Appelbaum and a remote address by its director, Laura Poitras, and followed by a long Q&A session by Jacob. DebConf15 was also the first DebConf with organised childcare (including a Teckids workshop for kids of age 8-16), which our DPL Neil McGovern standardised for the future: it s a thing now, he said. The participants used the week before the conference for intensive work, sprints and workshops, and throughout the main conference, significant progress was made on Debian and Free Software. Possibly the most visible was the endeavour to provide reproducible builds, but the planning of the next stable release stretch received no less attention. Groups like the Perl team, the diversity outreach programme and even DebConf organisation spent much time together discussing next steps and goals, and hundreds of commits were made to the archive, as well as bugs closed. DebConf15 was an amazing conference, it brought together hundreds of people, some oldtimers as well as plenty of new contributors, and we all had a great time, learning and collaborating with each other, says Margarita Manterola of the organiser team, and continues: The whole team worked really hard, and we are all very satisfied with the outcome. Another organiser, Martin Krafft adds: We mainly provided the infrastructure and space. A lot of what happened during the two weeks was thanks to our attendees. And that s what makes DebConf be DebConf. Photo of hostel staff wearing DebConf15 staff t-shirts (by Martin Krafft) Our organisation was greatly supported by the staff of the conference venue, the Jugendherberge Heidelberg International, who didn t take very long to identify with our diverse group, and who left no wishes untried. The venue itself was wonderfully spacious and never seemed too full as people spread naturally across the various conference rooms, the many open areas, the beergarden, the outside hacklabs and the lawn. The network installed specifically for our conference in collaboration with the nearby university, the neighbouring zoo, and the youth hostel provided us with a 1 Gbps upstream link, which we managed to almost saturate. The connection will stay in place, leaving the youth hostel as one with possibly the fastest Internet connection in the state. And the kitchen catered high-quality food to all attendees and their special requirements. Regional beer and wine, as well as local specialities, were provided at the bistro. DebConf exists to bring people together, which includes paying for travel, food and accomodation for people who could not otherwise attend. We would never have been able to achieve what we did without the support of our generous sponsors, especially our Platinum Sponsor Hewlett-Packard. Thank you very much. See you next year in Cape Town, South Africa!
The DebConf16 logo with white background

Philipp Kern: Automating the 3270 part of a Debian System z install

If you try to install Debian on System z within z/VM you might be annoyed at the various prompts it shows before it lets you access the network console via SSH. We can do better. From within CMS copy the default EXEC and default PARMFILE:

COPYFILE DEBIAN EXEC A DEBAUTO EXEC A
COPYFILE PARMFILE DEBIAN A PARMFILE DEBAUTO A

Now edit DEBAUTO EXEC A and replace the DEBIAN in 'PUNCH PARMFILE DEBIAN * (NOHEADER' with DEBAUTO. This will load the alternate kernel parameters file into the card reader, while still loading the original kernel and initrd files.

Replace PARMFILE DEBAUTO A's content with this (note the 80 character column limit):

ro locale=C
s390-netdevice/choose_networktype=qeth s390-netdevice/qeth/layer2=true
s390-netdevice/qeth/choose=0.0.fffc-0.0.fffd-0.0.fffe
netcfg/get_ipaddress=<IPADDR> netcfg/get_netmask=255.255.255.0
netcfg/get_gateway=<GW> netcfg/get_nameservers=<FIRST-DNS>
netcfg/confirm_static=true netcfg/get_hostname=debian
netcfg/get_domain=
network-console/authorized_keys_url=http://www.kern.lc/id_rsa.pub
preseed/url=http://www.kern.lc/preseed-s390x.cfg

Replace <IPADDR>, <GW>, and <FIRST-DNS> to suit your local network config. You might also need to change the netmask, which I left in for clarity about the format. Adjust the device address of your OSA network card. If it's in layer 3 mode (very likely) you should set layer2=false. Note that mixed case matters, hence you will want to SET CASE MIXED in xedit.

Then there are the two URLs that need to be changed. The authorized_keys_url file contains your SSH public key and is fetched unencrypted and unauthenticated, so be careful what networks you traverse with your request (HTTPS is not supported by debian-installer in Debian).

 preseed/url is needed for installation parameters that do not fit the parameters file - there is an upper character limit that's about two lines longer than my example. This is why this example only contains the bare minimum for the network part, everything else goes into this preseeding file. It file can optionally be protected with a MD5 checksum in preseed/url/checksum.

Both URLs need to be very short. I thought that there was a way to specify a line continuation, but in my tests I was unable to produce one. Hence it needs to fit on one line, including the key. You might want to use an IPv4 as the hostname.

To skip the initial boilerplate prompts and to skip straight to the user and disk setup you can use this as preseed.cfg:

d-i debian-installer/locale string en_US
d-i debian-installer/country string US
d-i debian-installer/language string en
d-i time/zone US/Eastern
d-i mirror/country manual
d-i mirror/http/mirror string httpredir.debian.org
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string

I'm relatively certain that the DASD disk setup part cannot be automated yet. But the other bits of the installation should be preseedable just like on non-mainframe hardware.

Dirk Eddelbuettel: RcppGSL 0.3.0

A new version of RcppGSL just arrived on CRAN. The RcppGSL package provides an interface from R to the GNU GSL using our Rcpp package. Following on the heels of an update last month we updated the package (and its vignette) further. One of the key additions concern memory management: Given that our proxy classes around the GSL vector and matrix types are real C++ object, we can monitor their scope and automagically call free() on them rather then insisting on the user doing it. This renders code much simpler as illustrated below. Dan Dillon added const correctness over a series on pull request which allows us to write more standard (and simply nicer) function interfaces. Lastly, a few new typedef declaration further simply the use of the (most common) double and int vectors and matrices. Maybe a code example will help. RcppGSL contains a full and complete example package illustrating how to write a package using the RcppGSL facilities. It contains an example of computing a column norm -- which we blogged about before when announcing an much earlier version. In its full glory, it looks like this: 
#include <RcppGSL.h>
#include <gsl/gsl_matrix.h>
#include <gsl/gsl_blas.h>
extern "C" SEXP colNorm(SEXP sM)  
  try  
        RcppGSL::matrix<double> M = sM;     // create gsl data structures from SEXP
        int k = M.ncol();
        Rcpp::NumericVector n(k);           // to store results
        for (int j = 0; j < k; j++)  
            RcppGSL::vector_view<double> colview = gsl_matrix_column (M, j);
            n[j] = gsl_blas_dnrm2(colview);
         
        M.free() ;
        return n;                           // return vector
    catch( std::exception &ex )  
        forward_exception_to_r( ex );
    catch(...)  
        ::Rf_error( "c++ exception (unknown reason)" );
   
  return R_NilValue; // -Wall
We manually translate the SEXP coming from R, manually cover the try and catch exception handling, manually free the memory etc pp. Well in the current version, the example is written as follows: 
#include <RcppGSL.h>
#include <gsl/gsl_matrix.h>
#include <gsl/gsl_blas.h>
// [[Rcpp::export]]
Rcpp::NumericVector colNorm(const RcppGSL::Matrix & G)  
    int k = G.ncol();
    Rcpp::NumericVector n(k);           // to store results
    for (int j = 0; j < k; j++)  
        RcppGSL::VectorView colview = gsl_matrix_const_column (G, j);
        n[j] = gsl_blas_dnrm2(colview);
     
    return n;                           // return vector
This takes full advantage of Rcpp Attributes automagically creating the interface and exception handler (as per the previous release), adds a const & interface, does away with the tedious and error-pronce free() and uses the shorter-typedef forms for RcppGSL::Matrix and RcppGSL::VectorViews using double variables. Now the function is short and concise and hence easier to read and maintain. The package vignette has more details on using RcppGSL. The NEWS file entries follows below:
Changes in version 0.3.0 (2015-08-30)
  • The RcppGSL matrix and vector class now keep track of object allocation and can therefore automatically free allocated object in the destructor. Explicit x.free() use is still supported.
  • The matrix and vector classes now support const reference semantics in the interfaces (thanks to PR #7 by Dan Dillon)
  • The matrix_view and vector_view classes are reorganized to better support const arguments (thanks to PR #8 and #9 by Dan Dillon)
  • Shorthand forms such as Rcpp::Matrix have been added for double and int vectors and matrices including views.
  • Examples such as fastLm can now be written in a much cleaner and shorter way as GSL objects can appear in the function signature and without requiring explicit .free() calls at the end.
  • The included examples, as well as the introductory vignette, have been updated accordingly.
Courtesy of CRANberries, a summary of changes to the most recent release is available. More information is on the RcppGSL page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Sven Hoexter: 1960 SubjectAlternativeNames on one certificate

tl;dr; You can add 1960+ SubjectAlternativeNames on one certificate and at least Firefox and Chrome are working fine with that. Internet Explorer failed but I did not investigate why. So why would you want to have close to 2K SANs on one certificate? While we're working on adopting a more dynamic development workflow at my workplace we're currently bound to a central development system. From there we serve a classic virtual hosting setup with "projectname.username.devel.ourdomain.example" mapped on "/web/username/projectname/". That is 100% dynamic with wildcard DNS entries and you can just add a new project to your folder and use it directly. All of that is served from just a single VirtualHost. Now our developers started to go through all our active projects to make them fit for serving via HTTPS. While we can verify the proper usage of https on our staging system where we've validating certificates, that's not the way you'd like to work. So someone approached me to look into a solution for our development system. Obvious choices like wildcard certificates do not work here because we've two dynamic components in the FQDN. So we would've to buy a wildcard certificate for every developer and we would've to create a VirtualHost entry for every new developer. That's expensive and we don't want all that additional work. So I started to search for documented limits on the number of SANs you can have on a certificate. The good news: there are none. The RFC does not define a limit. So much about the theory. ;) Following Ivans excellent documentation I setup an internal CA and an ugly "find ... sed ... tr ..." one-liner later I had a properly formated openssl config file to generate a CSR with all 1960 "projectname.username..." SAN combinations found on the development system. Two openssl invocations (CSR generation and signing) later I had a signed certificate with 1960 SANs on it. I imported the internal CA I created in Firefox and Chrome, and to my surprise it worked. Noteworthy: To sign with "openssl ca" without interactive prompts you've to use the "-batch" option. I'm thinking about regenerating the certificate every morning so our developers just have to create a new project directory and within 24h serving via HTTPS would be enabled. The only thing I'm currently pondering about is how to properly run the CA in a corporate Windows world. We could of course ask the Windows guys to include it for everyone but then we would've to really invest time in properly running the CA. I'd like to avoid that hassle. So I'd guess we just stick to providing the CA for those developers who need it. This all or nothing model is a constant PITA, and you really do not want to get owned via your own badly managed CA. :( Regarding Internet Explorer it jumped in my face with a strange error message that recommended to enable TLS 1.0, 1.1 and 1.2 in the options menu. Of course that's already enable. I'll try to take a look at the handshake next week, but I bet we've to accept for the moment that IE will not work with so many SANs. Would be interesting to try out Windows 10 with Spartan, but well I'm not that interested in Windows to invest more time on that front. Other TLS implementations, like Java, would be also interesting to test.

29 August 2015

Norbert Preining: Kobo Glo and GloHD firmware 3.17.3 mega update (KSM, nickel patch, ssh, fonts)

I have updated my mega-update for Kobo to the latest firmware 3.17.3. Additionally, I have not built (and tested) updates for both Mark4 hardware (Glo) and Mark6 hardware (GloHD). Please see the previous post for details on what is included. The only difference that is important is the update to KSM (Kobo Start Menu) version 8, which is still in testing phase (thus a warning: the layout and setup of KSM8 might change till release). This is an important update as all version up to V7 could create database corruptions (which I have seen several times!) when used with Calibre and the Kepub driver. Kobo Logo Other things that are included are as usual: Metazoa firmware patches for the Glo (non HD) version I have activated the compact layout patch; koreader, pbchess, coolreader, the ssh part of kobohack, custom dictionaries support, and some side-loaded fonts. Again, for details please see the previous post You can check for database corruption by selecting tools - nickel diverse.msh - db chk integrity.sh in the Kobo Start Menu. If it returns ok, then all is fine. Otherwise you might see problems. I solved the corruption of my database by first dumping the database to an sql file, and reloading it into a new database. Assuming that you have the file KoboReader.sqlite, what I did is: 
$ sqlite3  KoboReader.sqlite 
SQLite version 3.8.11.1 2015-07-29 20:00:57
Enter ".help" for usage hints.
sqlite> PRAGMA integrity_check;
*** in database main ***
Page 5237: btreeInitPage() returns error code 11
On tree page 889 cell 1: 2nd reference to page 5237
Page 4913 is never used
Page 5009 is never used
Error: database disk image is malformed
sqlite> .output foo.sql
sqlite> .dump
sqlite> .quit
$ sqlite3 KoboReader.sqlite-NEW
SQLite version 3.8.11.1 2015-07-29 20:00:57
Enter ".help" for usage hints.
sqlite> .read foo.sql
sqlite> .quit
The first part shows that the database is corrupted. Fortunately dumping succeeded and then reloading it into a new database, too. Finally I replaced (after backup) the sqlite on the device with the new database. Download Mark6 Kobo GloHD firmware: Kobo 3.17.3 for GloHD Mega update: Kobo-3.17.3-combined/Mark6/KoboRoot.tgz Mark4 Kobo Glo, Auro HD firmware: Kobo 3.17.3 for Glo and AuroHD Mega update: Kobo-3.17.3-combined/Mark4/KoboRoot.tgz Enjoy.

28 August 2015

Dimitri John Ledkov: Go enjoy Python3

Given a string, get a truncated string of length up to 12.

The task is ambiguous, as it doesn't say anything about whether or not 12 should include terminating null character or not. None the less, let's see how one would achieve this in various languages.
Let's start with python3 

import sys
print(sys.argv[1][:12])
Simple enough, in essence given first argument, print it up to length 12. As an added this also deals with unicode correctly that is if passed arg is , it will correctly print . (note these are just random Unicode strings to me, no idea what they stand for).

In C things are slightly more verbose, but in essence, I am going to use strncpy function: 

#include <stdio.h>
#include <string.h>
void main(int argc, char *argv[])  
 char res[12];
 strncpy(res,argv[1],12);
 printf("%s\n",res);
This treats things as byte-array instead of unicode, thus for unicode test it will end up printing just . But it is still simple enough.
Finally we have Go 
package main

import "os"
import "fmt"
import "math"

func main()  
 fmt.Printf("%s\n", os.Args[1][:int(math.Min(12, float64(len(os.Args[1]))))])
This similarly treats argument as a byte array, and one needs to cast the argument to a rune to get unicode string handling. But there are quite a few caveats. One cannot take out of bounds slices. Thus a na ve os.Args[1][:12] can result in a runtime panic that slice bounds are out of range. Or if a string is known at compile time, a compile time error. Hence one needs to calculate length, and do a min comparison. And there lies the next caveat, math.Min() is only defined for float64 type, and slice indexes can only be integers and thus we end up writing ]))))])...

12 points for python3, 8 points for C, and Go receives nul points Eurovision style.

EDIT: Andreas R ssland and James Hunt are full of win. Both suggesting fmt.Printf("%.12s\n", os.Args[1]) for go. I like that a lot, as it gives simplicity & readability without compromising the default safety against out of bounds access. Hence the scores are now: 14 points for Go, 12 points for python3 and 8 points for C.

EDIT2: I was pointed out much better C implementation by Keith Thompson - http://pastebin.com/5i7rFmMQ in essence it uses strncat() which has much better null termination semantics. And Ben posted a C implementation which handles wide characters http://www.decadent.org.uk/ben/blog/truncating-a-string-in-c.html. I regret to inform you that this blog post got syndicated onto hacker news and has now become the top viewed post on my blog of all time, overnight. In retrospect, I regret awarding points at the end of the blog post, as that's just was merely an expression of opinion and is highly subjective measure. But this problem statement did originate from me reviewing go code that did "if/then/else" comparison and got it wrong to truncate a string and I thought surely one can just do [:12] which has lead me down the rabbit hole of discovering a lot about Go; it's compile and runtime out of bounds access safeguards; lack of universal Min() function; runes vs strings handling and so on. I'm only a beginner go programmer and I am very sorry for wasting everyone's time on this. I guess people didn't have much to do on a Throwback Thursday.

The postings on this site are my own and don't necessarily represent Intel s positions, strategies, or opinions.

Lucas Nussbaum: DebConf 15

I attended DebConf 15 last week. After being on semi-vacation from Debian for the last few months, recovering after the end of my second DPL term, it was great to be active again, talk to many people, and go back to doing technical work. Unfortunately, I caught the debbug quite early in the week, so I was not able to make it as intense as I wanted, but it was great nevertheless. I still managed to do quite a lot:
  • I rewrote a core part of UDD, which will make it easier to monitor data importer scripts and reduce the cron-spam
  • with DSA members, I worked on finding a suitable workaround for the storage performance issues that have been plaguing UDD for the last few months. fsyncs() will now longer hang for 15 minutes, yay!
  • I added a DUCK importer to UDD, and added that information to the Debian Maintainer Dashboard
  • I worked a bit on cleaning up the status of my packages, including digging into a strange texlive issue (that showed up in developers-reference), that is now fixed in unstable
  • I worked a bit on improving git-buildpackage documentation (more to come in that area)
  • Last but not least, I played Mao for the first time in years, and it was a lot of fun. (even if my brain is still slowly recovering)
DC15 was a great DebConf, probably one of the two bests I ve attended so far. I m now looking forward to DC16 in Cape Town!

Next.

Previous.