Search Results: "kcr"

30 October 2021

Dirk Eddelbuettel: dang 0.0.15: Small Correction

A bug-fix release of the dang package arrived at CRAN a little while ago. The dang package regroups a few functions of mine that had no other home as for example lsos() from a StackOverflow question from 2009 (!!), the overbought/oversold price band plotter from an older blog post, the market monitor from the last release as well the checkCRANStatus() function tweeted about by Tim Taylor. This release corrects a small mistake wrapping extern "C" ... around code that is not actually C but C++ which g++ kept silent about yet clang++ complains about, correctly. So CRAN asked me to correct this, which this version does. The NEWS entry follows.

Changes in version 0.0.15 (2021-10-26)
  • Corrected scope of 'extern "C"' declaration to the actually C-callable function (noticed by clang++, change requested by CRAN)

Courtesy of my CRANberries, there is a comparison to the previous release. For questions or comments use the issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

18 October 2021

Dirk Eddelbuettel: dang 0.0.14: Several Updates

A new release of the dang package arrived at CRAN a couple of hours ago, exactly eight months after the previous release. The dang package regroups a few functions of mine that had no other home as for example lsos() from a StackOverflow question from 2009 (!!), the overbought/oversold price band plotter from an older blog post, the market monitor from the last release as well the checkCRANStatus() function recently tweeted about by Tim Taylor. This release regroups a few small edits to several functions, adds a sample function for character encoding reading and conversion using a library already used by R (hence look Ma, no new depends ), adds a weekday helper, and a sample usage (computing rolling min/max values) of a new simple vector class added to tidyCpp (and the function and class need to get another blog post or study ), and an experimental git sha1sum and date marker (as I am not the fan of autogenerated binaries from repos as opposed to marked released meaning: we may see different binary release with the same version number). The full NEWS entry follows.

Changes in version 0.0.14 (2021-10-17)
  • Updated continuous integration to run on Linux only.
  • Edited checkNonAscii.cpp for readability.
  • More robust title display in intradayMarketMonitor.R.
  • New C++-based function to read and convert encoding via the R-supplied iconv library, noted a potential variability.
  • New function wday returning day of the week as integer.
  • The signature to as.data.table was standardized.
  • A new function rollMinMax was added illustrating use of the NumVec class from tidyCpp.
  • The configure script can record the last commit date and sha1 to automate timestamping builds, but not activated in this release.
  • checkCRANStatus() now works correctly for single-package lookups (Jordan Mark Barbone in #4).

Courtesy of my CRANberries, there is a comparison to the previous release. For questions or comments use the issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

1 January 2021

Keith Packard: kgames

Reviving Very Old X Code I've taken the week between Christmas and New Year's off this year. I didn't really have anything serious planned, just taking a break from the usual routine. As often happens, I got sucked into doing a project when I received this simple bug report Debian Bug #974011
I have been researching old terminal and X games recently, and realized
that much of the code from 'xmille' originated from the terminal game
'mille', which is part of bsdgames.
...
[The copyright and license information] has been stripped out of all
code in the xmille distribution.  Also, none of the included materials
give credit to the original author, Ken Arnold.
The reason the 'xmille' source is missing copyright and license information from the 'mille' files is that they were copied in before that information was added upstream. Xmille forked from Mille around 1987 or so. I wrote the UI parts for the system I had at the time, which was running X10R4. A very basic port to X11 was done at some point, and that's what Debian has in the archive today. At some point in the 90s, I ported Xmille to the Athena widget set, including several custom widgets in an Xaw extension library, Xkw. It's a lot better than the version in Debian, including displaying the cards correctly (the Debian version has some pretty bad color issues). Here's what the current Debian version looks like: Fixing The Bug To fix the missing copyright and license information, I imported the mille source code into the "latest" Xaw-based version. The updated mille code had a number of bug fixes and improvements, along with the copyright information. That should have been sufficient to resolve the issue and I could have constructed a suitable source package from whatever bits were needed and and uploaded that as a replacement 'xmille' package. However, at some later point, I had actually merged xmille into a larger package, 'kgames', which also included a number of other games, including Reversi, Dominoes, Cribbage and ten Solitaire/Patience variants. (as an aside, those last ten games formed the basis for my Patience Palm Pilot application, which seems to have inspired an Android App of the same name...) So began my yak shaving holiday. Building Kgames in 2020 Ok, so getting this old source code running should be easy, right? It's just a bunch of C code designed in the 80s and 90s to work on VAXen and their kin. How hard could it be?
  1. Everything was a 32-bit computer back then; pointers and ints were both 32 bits, so you could cast them with wild abandon and cause no problems. Today, testing revealed segfaults in some corners of the code.
  2. It's K&R C code. Remember that the first version of ANSI-C didn't come out until 1989, and it was years later that we could reliably expect to find an ANSI compiler with a random Unix box.
  3. It's X11 code. Fortunately (?), X11 hasn't changed since these applications were written, so at least that part still works just fine. Imagine trying to build Windows or Mac OS code from the early 90's on a modern OS...
I decided to dig in and add prototypes everywhere; that found a lot of pointer/int casting issues, as well as several lurking bugs where the code was just plain broken. After a day or so, I had things building and running and was no longer hitting crashes. Kgames 1.0 uploaded to Debian New Queue With that done, I decided I could at least upload the working bits to the Debian archive and close the bug reported above. kgames 1.0-2 may eventually get into unstable, presumably once the Debian FTP team realizes just how important fixing this bug is. Or something. Here's what xmille looks like in this version: And here's my favorite solitaire variant too: But They Look So Old Yeah, Xaw applications have a rustic appearance which may appeal to some, but for people with higher resolution monitors and well seasoned eyesight, squinting at the tiny images and text makes it difficult to enjoy these games today. How hard could it be to update them to use larger cards and scalable fonts? Xkw version 2.0 I decided to dig in and start hacking the code, starting by adding new widgets to the Xkw library that used cairo for drawing instead of core X calls. Fortunately, the needs of the games were pretty limited, so I only needed to implement a handful of widgets: The other Xkw widgets all got their rendering switched to using cairo, plus using double buffering to make updates look better. SVG Playing Cards Looking on wikimedia, I found a page referencing a large number of playing cards in SVG form That led me to Adrian Kennard's playing card web site that let me customize and download a deck of cards, licensed using the CC0 Public Domain license. With these cards, I set about rewriting the Xkw playing card widget, stripping out three different versions of bitmap playing cards and replacing them with just these new SVG versions. SVG Xmille Cards Ok, so getting regular playing cards was good, but the original goal was to update Xmille, and that has cards hand drawn by me. I could just use those images, import them into cairo and let it scale them to suit on the screen. I decided to experiment with inkscape's bitmap tracing code to see what it could do with them. First, I had to get them into a format that inkscape could parse. That turned out to be a bit tricky; the original format is as a set of X bitmap layers; each layer painting a single color. I ended up hacking the Xmille source code to generate the images using X, then fetching them with XGetImage and walking them to construct XPM format files which could then be fed into the portable bitmap tools to create PNG files that inkscape could handle. The resulting images have a certain charm: I did replace the text in the images to make it readable, otherwise these are untouched from what inkscape generated. The Results Remember that all of these are applications built using the venerable X toolkit; there are still some non-antialiased graphics visible as the shaped buttons use the X Shape extension. But, all rendering is now done with cairo, so it's all anti-aliased and all scalable. Here's what Xmille looks like after the upgrades: And here's spider: Once kgames 1.0 reaches Debian unstable, I'll upload these new versions.

30 May 2017

Keith Packard: DRM-lease-3

DRM leasing part three (Vulkan) With the kernel APIs off for review, and the X RandR bits looking like they're in reasonable shape, I finally found some time to sit down and figure out how I wanted to integrate this into Vulkan. Avoiding two DRM file descriptors Given that a DRM lease is represented by a DRM master file descriptor, we want to use that for all of the operations in the driver, including rendering and mode setting. Using the vulkan driver render node and the lease master node together would require passing buffer objects between the kernel contexts using even more file descriptors. The Mesa Vulkan drivers open the device nodes while enumerating devices, not when they are created. This seems a bit early to me, but it makes sure that the devices being enumerated are actually available for use, and not just present in the system. To replace the render node fd with the lease master fd means hooking into the system early enough that the enumeration code can see the lease fd. And that means creating an instance extension as the instance gets created before devices are enumerated. The VK_KEITHP_kms_display instance extension This simple instance extension provides the necessary hooks to get the lease information from the application down into the driver before the DRM node is opened. In the first implementation, I added a function that could be called before the devices were enumerated to save the information in the Vulkan loader. That worked, but required quite a bit of study of the Vulkan loader and its XML description of the full Vulkan API. Mark Young suggested that a simpler plan would be to chain the information into the VkInstanceCreateInfo pNext field; with no new APIs added to Vulkan, there shouldn't be any need to change the Vulkan loader -- the device driver would advertise the new instance extension and the application could find it. That would have worked great, except the Vulkan loader 'helpfully' elides all instance extensions it doesn't know about before returning the list to the application. I'd say this was a bug and should be fixed, but for now, I've gone ahead and added the few necessary definitions to the loader to make it work. In the application, it's a simple matter of searching for this extension, constructing the VkKmsDisplayInfoKEITHP structure, chaining that into the VkInstanceCreateInfo pNext list and passing that in to the vkCreateInstance call.
typedef struct VkKmsDisplayInfoKEITHP  
    VkStructureType         sType;  /* VK_STRUCTURE_TYPE_KMS_DISPLAY_INFO_KEITHP */
    const void*             pNext;
    int                     fd;
    uint32_t                crtc_id;
    uint32_t                *connector_ids;
    int                     connector_count;
    drmModeModeInfoPtr      mode;
  VkKmsDisplayInfoKEITHP;
As you can see, this includes the master file descriptor along with all of the information necessary to set the desired video mode using the specified resources. The driver just walks the pNext list from the VkInstanceCreateInfo structure looking for any provided VkKmsDisplayInfoKEITHP structure and pulls the data out. To avoid questions about file descriptor lifetimes, the driver dup's the provided fd. The application is expected to close their copy at a suitable time. The VK_KHR_display extension Vulkan already has an API for directly accessing the raw device, including code for exposing video modes and everything. As tempting as it may be to just go do something simpler, there's a lot to be said for using existing APIs. This extension doesn't provide any direct APIs for acquiring display resources, relying on the VK_EXT_acquire_xlib_display extension for that part. And that takes a VkPhysicalDisplay parameter, which is only available after the device is opened, which is why I created the VK_KEITHP_kms_display extension instead of just using the VK_EXT_acquire_xlib_display extension -- we can't increase the capabilities of the render node opened by the driver, and we don't want to keep two file descriptors around. With the information provided by the VK_KEITHP_kms_display extension, we can implement all of the VK_KHR_display extension APIs, including enumerating planes and modes and creating the necessary display surface. Of course, there's only one plane and one mode, so some of the implementation is pretty simplistic. The big piece of work was to create the swap chain structure and associated frame buffers. A working example I've taken the 'cube' example from the Vulkan loader and hacked it up to use XCB to construct a DRM lease, the VK_KEITHP_kms_display extension to pass that lease into the Vulkan driver. The existing support for the VK_KHR_display extension "just worked", which was pretty satisfying. It's a bit of a mess I'm not satisfied with the mesa code at this point; there's a bunch of code in the radeon driver which should be in the vulkan WSI bits, and the vulkan WSI bits should probably not have the KMS interfaces wired in. I'll ask around and see what other Mesa developers think I should do before restructuring it further; I'll probably have to rewrite it all at least one more time before it's ready to upstream. Seeing the code I'll be cleaning the code up a bit before sending it out for review, but it's already visible in my own repositories:

17 October 2015

Simon Richter: Key Transition

So, since several years I've had a second gpg key, 4096R/6AABE354. Several of you have already signed it, and I've been using it in Debian for some time already, but I've not announced it more widely yet, and I occasionally still get mail encrypted to the old key (which remains valid and usable, but it's 1024R). Of course, I've also made a formal transition statement:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
OpenPGP Key Transition Statement for Simon Richter
I have created a new OpenPGP key and will be transitioning away from
my old key.  The old key has not been compromised and will continue to
be valid for some time, but I prefer all future correspondence to be
encrypted to the new key, and will be making signatures with the new
key going forward.
I would like this new key to be re-integrated into the web of trust.
This message is signed by both keys to certify the transition.  My new
and old keys are signed by each other.  If you have signed my old key,
I would appreciate signatures on my new key as well, provided that
your signing policy permits that without re-authenticating me.
The old key, which I am transitioning away from, is:
pub   1024D/5706A4B4 2002-02-26
      Key fingerprint = 040E B5F7 84F1 4FBC CEAD  ADC6 18A0 CC8D 5706 A4B4
The new key, to which I am transitioning, is:
pub   4096R/6AABE354 2009-11-19
      Key fingerprint = 9C43 2534 95E4 DCA8 3794  5F5B EBF6 7A84 6AAB E354
The entire key may be downloaded from: http://www.simonrichter.eu/simon.asc
To fetch the full new key from a public key server using GnuPG, run:
  gpg --keyserver keys.gnupg.net --recv-key 6AABE354
If you already know my old key, you can now verify that the new key is
signed by the old one:
  gpg --check-sigs 6AABE354
If you are satisfied that you've got the right key, and the User IDs
match what you expect, I would appreciate it if you would sign my key:
  gpg --sign-key 6AABE354
You can upload your signatures to a public keyserver directly:
  gpg --keyserver keys.gnupg.net --send-key 6AABE354
Or email sr@simonrichter.eu (possibly encrypted) the output from:
  gpg --armor --export 6AABE354
If you'd like any further verification or have any questions about the
transition please contact me directly.
To verify the integrity of this statement:
  wget -q -O- http://www.simonrichter.eu/key-transition-2015-03-09.txt   gpg --verify
   Simon
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJU/bbEAAoJEH69OHuwmQgRWOIH/AogHxhhVO5Tp5FFGpBFwljf
NzKTPBExMhZ/8trAzYybOWFv3Bx4AGdWkYfDnxP6oxQJOXVq4KL6ZcPPuIZuZ6fZ
bu0XHdPMU89u0TymR/WJENRCOcydRBe/lZs+zdJbKQtEZ+on4uNXxHpUiZPi1xxM
ggSdVBKn2PlCBcYih40S9Oo/rM9uBmYcFavX7JMouBSzgX78cVoIcY6zPRmHoq4k
TkGKfvHeSu+wbzWRmDwu/PFHRA4TKNvR6oeO+Et1yk454zjrHMXineBILRvvMCIA
t54pV6n+XzOUmtXcKnkIGltK+ZaJSV6am0swtx84RaevVXrknIQE8NvlA4MNgguI
nAQBAQIABgUCVP23tAAKCRDSx966V9+/u3j4BACVHifAcO86jAc5dn+4OkFZFhV1
l3MKIolL+E7Q7Ox+vJunGJJuGnOnazUK+64yDGZ2JxNJ4QNWD1FOs/Ng2gm82Vin
ArBtyp1ZGWUa+349X+1qarUQF9qAaUXDZjFp5Hzh/o6KC4t3eECxcb41og3LUTQD
VuG2KWNXYBe5P5ak9Q==
=o61r
-----END PGP SIGNATURE-----

22 June 2014

Simon Josefsson: OpenPGP Key Transition Statement

I have created a new OpenPGP key 54265e8c and will be transitioning away from my old key. If you have signed my old key, I would appreciate signatures on my new key as well. I have created a transition statement that can be downloaded from https://josefsson.org/key-transition-2014-06-22.txt. Below is the signed statement.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
OpenPGP Key Transition Statement for Simon Josefsson
I have created a new OpenPGP key and will be transitioning away from
my old key.  The old key has not been compromised and will continue to
be valid for some time, but I prefer all future correspondence to be
encrypted to the new key, and will be making signatures with the new
key going forward.
I would like this new key to be re-integrated into the web of trust.
This message is signed by both keys to certify the transition.  My new
and old keys are signed by each other.  If you have signed my old key,
I would appreciate signatures on my new key as well, provided that
your signing policy permits that without re-authenticating me.
The old key, which I am transitioning away from, is:
pub   1280R/B565716F 2002-05-05
      Key fingerprint = 0424 D4EE 81A0 E3D1 19C6  F835 EDA2 1E94 B565 716F
The new key, to which I am transitioning, is:
pub   3744R/54265E8C 2014-06-22
      Key fingerprint = 9AA9 BDB1 1BB1 B99A 2128  5A33 0664 A769 5426 5E8C
The entire key may be downloaded from: https://josefsson.org/54265e8c.txt
To fetch the full new key from a public key server using GnuPG, run:
  gpg --keyserver keys.gnupg.net --recv-key 54265e8c
If you already know my old key, you can now verify that the new key is
signed by the old one:
  gpg --check-sigs 54265e8c
If you are satisfied that you've got the right key, and the User IDs
match what you expect, I would appreciate it if you would sign my key:
  gpg --sign-key 54265e8c
You can upload your signatures to a public keyserver directly:
  gpg --keyserver keys.gnupg.net --send-key 54265e8c
Or email simon@josefsson.org (possibly encrypted) the output from:
  gpg --armor --export 54265e8c
If you'd like any further verification or have any questions about the
transition please contact me directly.
To verify the integrity of this statement:
  wget -q -O- https://josefsson.org/key-transition-2014-06-22.txt gpg --verify
/Simon
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
iLwEAQEKAAYFAlOnV+AACgkQ7aIelLVlcW89XgUAljJgYfReyR9/bU+Om6UHUttt
CAOgSRqdcQSQ2hT69vzuhb/bc8CslIQcBtGqTgxDFsxEFhbm5zKn+tSzy5MHNHqt
MsqHcZjlYuYVhMXDhka+cfyhtd9zIxjVE5vk8v+GqEGoh8DGYq0vPy3VfvcSz5Z3
MSUpSj8gN00jlU1z4nad3maEq0ApvsLr8EsLZmtxF5TNFvzJ8mmwY+gHBGHjVYkB
8AQBAQoABgUCU6dX4AAKCRAGZKdpVCZejD1eDp46XGL2puMp0le2OF75WIUW8xqf
TMiZeB99ruk3P/jvuLnGPP2J5o7SIKE50FkMEss0yvxi6jBlHk+cJeKWGXVjBpxU
0QHq063NU+kjbMYwDfi5ZxXqaKeYODJm8Xmfh3d7lRaWF5rUOosR8nC/OROSrhg4
TjlAbvbxpQsls/JPbbporK2gbAtMlzJPD8zC8z/dT+t0qjlce8fADugblVW3bACC
Kl53X4XpojzNd/U19tSXkIBdNY/GVJqci+iruiJ1WGARF9ocnIXVuNXsfyt7UGq4
UiM/AeDVzI76v1QnE8WpsmSXzi2zXe3VahUPhOU2nPDoL53ggiVsTY3TwilvQLfX
Av/74PIaEtCi1g23YeojQlpdYzcWfnE+tUyTSNwPIBzyzHvFAHNg1Pg0KKUALsD9
P7EjrMuz63z2276EBKX8++9GnQQNCNfdHSuX4WGrBx2YgmOOqRdllMKz6pVMZdJO
V+gXbCMx0D5G7v50oB58Mb5NOgIoOnh3IQhJ7LkLwmcdG39yCdpU+92XbAW73elV
kmM8i0wsj5kDUU2ys32Gj2HnsVpbnh3Fvm9fjFJRbbQL/FxNAjzNcHe4cF3g8hTb
YVJJlzhmHGvd7HvXysJJaa0=
=ZaqY
-----END PGP SIGNATURE-----
flattr this!

5 January 2013

Olly Betts: Empty Strings in C++

Gaurav (one of Xapian's GSoC students this year) queried my advice that it's better not to explicitly initialise a std::string in C++ like this:
std::string s = "";
but instead to write:
std::string s;
It could be argued that the former makes it clearer that the string is initialised (since C++ does inherit C's behaviour of not implicitly initialising variables of fundamental types, such as int and double). But objects aren't left uninitialised - the default constructor gets called (or if there isn't one, the code won't compile). The downside is that you get quite a lot more code from the first version than the second. Perhaps compilers will grow smarter and in future both the above will compile to the same code, but that's not true today. Here's a simple bit of test code:
#include <string>
std::string f()  
#ifdef EXPLICIT_INIT
    std::string s = "";
#else
    std::string s;
#endif
    return s;
 
And I'll compile it with GCC (g++ (Debian 4.7.2-4) 4.7.2) on x86-64 to produce assembler output:
$ g++ -O2 -S s.cc -o s1.s
$ g++ -DEXPLICIT_INIT -O2 -S s.cc -o s2.s
$ diff s1.s s2.s
1a2,4
>   .section        .rodata.str1.1,"aMS",@progbits,1
> .LC0:
>   .string ""
9,10c12,25
<   movq    %rdi, %rax
<   movq    $_ZNSs4_Rep20_S_empty_rep_storageE+24, (%rdi)
---
>   pushq   %rbx
>   .cfi_def_cfa_offset 16
>   .cfi_offset 3, -16
>   movl    $.LC0, %esi
>   movq    %rdi, %rbx
>   subq    $16, %rsp
>   .cfi_def_cfa_offset 32
>   leaq    15(%rsp), %rdx
>   call    _ZNSsC1EPKcRKSaIcE
>   addq    $16, %rsp
>   .cfi_def_cfa_offset 16
>   movq    %rbx, %rax
>   popq    %rbx
>   .cfi_def_cfa_offset 8
(As an aside, I wouldn't recommend spending a lot of time digging into the assembler your compiler produces, but if there's more than one equivalent way to write code for a common case (as here) and you want to know which is most efficient, it can be informative to see what code is produced for each). You don't really need to know x86-64 assembler to grasp what's happening above, which is lucky, since I don't really know x86-64 assembler. In the first hunk, we get an empty literal string being added to the rodata (read-only data) section; and in the second, instead of two instructions which copy a standard empty string representation, we get much more elaborate code, including a function call to _ZNSsC1EPKcRKSaIcE - the C++ mangled name for the std::string from const char * constructor:
$ c++filt _ZNSsC1EPKcRKSaIcE
std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)
I also tested the example above with clang (Debian clang version 3.0-6 (tags/RELEASE_30/final) (based on LLVM 3.0)) and the resultant code is fairly similar for each case. The overhead for doing this once isn't going to matter, but if it happens every time you declare a std::string variable, the cumulative effects may be measurable. And as well as taking longer to execute, the larger code will cause greater CPU cache pressure.

6 August 2012

Mike Hommey: Building a Linux kernel module without the exact kernel headers

Imagine you have a Linux kernel image for an Android phone, but you don t have the corresponding source, nor do you have the corresponding kernel headers. Imagine that kernel has module support (fortunately), and that you d like to build a module for it to load. There are several good reasons why you can t just build a new kernel from source and be done with it (e.g. the resulting kernel lacks support for important hardware, like the LCD or touchscreen). With the ever-changing Linux kernel ABI, and the lack of source and headers, you d think you re pretty much in a dead-end. As a matter of fact, if you build a kernel module against different kernel headers, the module will fail to load with errors depending on how different they are. It can complain about bad signatures, bad version or other different things. But more on that later. Configuring a kernel The first thing is to find a kernel source for something close enough to the kernel image you have. That s probably the trickiest part with getting a proper configuration. Start from the version number you can read from /proc/version. If, like me, you re targeting an Android device, try Android kernels from Code Aurora, Linaro, Cyanogen or Android, whichever is closest to what is in your phone. In my case, it was msm-3.0 kernel. Note you don t necessarily need the exact same version. A minor version difference is still likely to work. I ve been using a 3.0.21 source, which the kernel image was 3.0.8. Don t however try e.g. using a 3.1 kernel source when the kernel you have is 3.0.x. If the kernel image you have is kind enough to provide a /proc/config.gz file, you can start from there, otherwise, you can try starting from the default configuration, but you need to be extra careful, then (although I won t detail using the default configuration because I was fortunate enough that I didn t have to, there will be some details further below as to why a proper configuration is important). Assuming arm-eabi-gcc is in your PATH, and that you have a shell opened in the kernel source directory, you need to start by configuring the kernel and install headers and scripts:
$ mkdir build
$ gunzip -c config.gz > build/.config # Or whatever you need to prepare a .config
$ make silentoldconfig prepare headers_install scripts ARCH=arm CROSS_COMPILE=arm-eabi- O=build KERNELRELEASE= adb shell uname -r 
The silentoldconfig target is likely to ask you some questions about whether you want to enable some things. You may want to opt for the default, but that may also not work properly. You may use something different for KERNELRELEASE, but it needs to match the exact kernel version you ll be loading the module from. A simple module To create a dummy module, you need to create two files: a source file, and a Makefile. Place the following content in a hello.c file, in some dedicated directory:
#include <linux/module.h>       /* Needed by all modules */
#include <linux/kernel.h>       /* Needed for KERN_INFO */
#include <linux/init.h>         /* Needed for the macros */
static int __init hello_start(void)
 
  printk(KERN_INFO "Hello world\n");
  return 0;
 
static void __exit hello_end(void)
 
  printk(KERN_INFO "Goodbye world\n");
 
module_init(hello_start);
module_exit(hello_end);
Place the following content in a Makefile under the same directory:
obj-m = hello.o
Building such a module is pretty straightforward, but at this point, it won t work yet. Let me enter some details first. The building of a module When you normally build the above module, the kernel build system creates a hello.mod.c file, which content can create several kind of problems:
MODULE_INFO(vermagic, VERMAGIC_STRING);
VERMAGIC_STRING is derived from the UTS_RELEASE macro defined in include/generated/utsrelease.h, generated by the kernel build system. By default, its value is derived from the actual kernel version, and git repository status. This is what setting KERNELRELEASE when configuring the kernel above modified. If VERMAGIC_STRING doesn t match the kernel version, loading the module will lead to the following kind of message in dmesg:
hello: version magic '3.0.21-perf-ge728813-00399-gd5fa0c9' should be '3.0.8-perf'
Then, there s the module definition.
struct module __this_module
__attribute__((section(".gnu.linkonce.this_module"))) =  
 .name = KBUILD_MODNAME,
 .init = init_module,
#ifdef CONFIG_MODULE_UNLOAD
 .exit = cleanup_module,
#endif
 .arch = MODULE_ARCH_INIT,
 ;
In itself, this looks benign, but the struct module, defined in include/linux/module.h comes with an unpleasant surprise:
struct module
 
        (...)
#ifdef CONFIG_UNUSED_SYMBOLS
        (...)
#endif
        (...)
        /* Startup function. */
        int (*init)(void);
        (...)
#ifdef CONFIG_GENERIC_BUG
        (...)
#endif
#ifdef CONFIG_KALLSYMS
        (...)
#endif
        (...)
(... plenty more ifdefs ...)
#ifdef CONFIG_MODULE_UNLOAD
        (...)
        /* Destruction function. */
        void (*exit)(void);
        (...)
#endif
        (...)
 
This means for the init pointer to be at the right place, CONFIG_UNUSED_SYMBOLS needs to be defined according to what the kernel image uses. And for the exit pointer, it s CONFIG_GENERIC_BUG, CONFIG_KALLSYMS, CONFIG_SMP, CONFIG_TRACEPOINTS, CONFIG_JUMP_LABEL, CONFIG_TRACING, CONFIG_EVENT_TRACING, CONFIG_FTRACE_MCOUNT_RECORD and CONFIG_MODULE_UNLOAD. Start to understand why you re supposed to use the exact kernel headers matching your kernel? Then, the symbol version definitions:
static const struct modversion_info ____versions[]
__used
__attribute__((section("__versions"))) =  
	  0xsomehex, "module_layout"  ,
	  0xsomehex, "__aeabi_unwind_cpp_pr0"  ,
	  0xsomehex, "printk"  ,
 ;
These come from the Module.symvers file you get with your kernel headers. Each entry represents a symbol the module requires, and what signature it is expected to have. The first symbol, module_layout, varies depending on what struct module looks like, i.e. depending on which of the config options mentioned above are enabled. The second, __aeabi_unwind_cpp_pr0, is an ARM ABI specific function, and the last, is for our printk function calls. The signature for each function symbol may vary depending on the kernel code for that function, and the compiler used to compile the kernel. This means that if you have a kernel you built from source, modules built for that kernel, and rebuild the kernel after modifying e.g. the printk function, even in a compatible way, the modules you built initially won t load with the new kernel. So, if we were to build a kernel from the hopefully close enough source code, with the hopefully close enough configuration, chances are we wouldn t get the same signatures as the binary kernel we have, and it would complain as follows, when loading our module:
hello: disagrees about version of symbol symbol_name
Which means we need a proper Module.symvers corresponding to the binary kernel, which, at the moment, we don t have. Inspecting the kernel Conveniently, since the kernel has to do these verifications when loading modules, it actually contains a list of the symbols it exports, and the corresponding signatures. When the kernel loads a module, it goes through all the symbols the module requires, in order to find them in its own symbol table (or other modules symbol table when the module uses symbols from other modules), and check the corresponding signature. The kernel uses the following function to search in its symbol table (in kernel/module.c):
bool each_symbol_section(bool (*fn)(const struct symsearch *arr,
                                    struct module *owner,
                                    void *data),
                         void *data)
 
        struct module *mod;
        static const struct symsearch arr[] =  
                  __start___ksymtab, __stop___ksymtab, __start___kcrctab,
                  NOT_GPL_ONLY, false  ,
                  __start___ksymtab_gpl, __stop___ksymtab_gpl,
                  __start___kcrctab_gpl,
                  GPL_ONLY, false  ,
                  __start___ksymtab_gpl_future, __stop___ksymtab_gpl_future,
                  __start___kcrctab_gpl_future,
                  WILL_BE_GPL_ONLY, false  ,
#ifdef CONFIG_UNUSED_SYMBOLS
                  __start___ksymtab_unused, __stop___ksymtab_unused,
                  __start___kcrctab_unused,
                  NOT_GPL_ONLY, true  ,
                  __start___ksymtab_unused_gpl, __stop___ksymtab_unused_gpl,
                  __start___kcrctab_unused_gpl,
                  GPL_ONLY, true  ,
#endif
         ;
        if (each_symbol_in_section(arr, ARRAY_SIZE(arr), NULL, fn, data))
                return true;
        (...)
The struct used in this function is defined in include/linux/module.h as follows:
struct symsearch  
        const struct kernel_symbol *start, *stop;
        const unsigned long *crcs;
        enum  
                NOT_GPL_ONLY,
                GPL_ONLY,
                WILL_BE_GPL_ONLY,
          licence;
        bool unused;
 ;
Note: this kernel code hasn t changed significantly in the past four years. What we have above is three (or five, when CONFIG_UNUSED_SYMBOLS is defined) entries, each of which contains the start of a symbol table, the end of that symbol table, the start of the corresponding signature table, and two flags. The data is static and constant, which means it will appear as is in the kernel binary. By scanning the kernel for three consecutive sequences of three pointers within the kernel address space followed by two integers with the values from the definitions in each_symbol_section, we can deduce the location of the symbol and signature tables, and regenerate a Module.symvers from the kernel binary. Unfortunately, most kernels these days are compressed (zImage), so a simple search is not possible. A compressed kernel is actually a small bootstrap binary followed by a compressed stream. It is possible to scan the kernel zImage to look for the compressed stream, and decompress it from there. I wrote a script to do decompression and extraction of the symbols info automatically. It should work on any recent kernel, provided it is not relocatable and you know the base address where it is loaded. It takes options for the number of bits and endianness of the architecture, but defaults to values suitable for ARM. The base address, however, always needs to be provided. It can be found, on ARM kernels, in dmesg:
$ adb shell dmesg   grep "\.init"
<5>[01-01 00:00:00.000] [0: swapper]      .init : 0xc0008000 - 0xc0037000   ( 188 kB)
The base address in the example above is 0xc0008000. If like me you re interested in loading the module on an Android device, then what you have as a binary kernel is probably a complete boot image. A boot image contains other things besides the kernel, so you can t use it directly with the script. Except if the kernel in that boot image is compressed, in which case the part of the script that looks for the compressed image will find it anyways. If the kernel is not compressed, you can use the unbootimg program as outlined in this old post of mine to get the kernel image out of your boot image. Once you have the kernel image, the script can be invoked as follows:
$ python extract-symvers.py -B 0xc0008000 kernel-filename > Module.symvers
Symbols and signature info could also be extracted from binary modules, but I was not interested in that information so the script doesn t handle that. Building our module Now that we have a proper Module.symvers for the kernel we want to load our module in, we can finally build the module: (again, assuming arm-eabi-gcc is in your PATH, and that you have a shell opened in the kernel source directory)
$ cp /path/to/Module.symvers build/
$ make M=/path/to/module/source ARCH=arm CROSS_COMPILE=arm-eabi- O=build modules
And that s it. You can now copy the resulting hello.ko onto the device and load it. and enjoy
$ adb shell
# insmod hello.ko
# dmesg   grep insmod
<6>[mm-dd hh:mm:ss.xxx] [id: insmod]Hello world
# lsmod
hello 586 0 - Live 0xbf008000 (P)
# rmmod hello
# dmesg   grep rmmod
<6>[mm-dd hh:mm:ss.xxx] [id: rmmod]Goodbye world

12 March 2012

Francois Marier: Creating a FreeDOS bootable USB stick to upgrade BIOS

I have an old motherboard that requires creating a DOS boot floppy in order to upgrade its BIOS. Fortunately, it's not too hard to do this with FreeDOS and a USB stick.

The instructions below are based on an FDos wiki article.

Downloading the dependenciesThe first step is to download the required files from your motherboard manufacturer:
and then install the tools you'll need:
apt-get install makebootfat syslinux

Preparing the "floppy" imageStart by collecting all of the files you need to install FreeDOS on the USB stick:
cd /tmp

wget http://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.0/pkgs/commandx.zip
wget http://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.0/pkgs/kernels.zip
wget http://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.0/pkgs/substx.zip
wget http://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.0/pkgs/unstablx.zip

for ZIP in *.zip; do unzip $ZIP; done

cp ./source/ukernel/boot/fat16.bin .
cp ./source/ukernel/boot/fat12.bin .
cp ./source/ukernel/boot/fat32lba.bin .

cp /usr/lib/syslinux/mbr.bin .
and then create a directory for the files that will end up in the root directory of the "floppy":
mkdir /tmp/fs-root
cp ./bin/command.com /tmp/fs-root/
cp ./bin/kernel.sys /tmp/fs-root/
and copy the BIOS image and update program into that same directory (/tmp/fs-root/).

Creating a bootable USB stickPlug in a FAT-formatted USB stick and look for the device it uses (/dev/sdb in the example below).

Finally, run makebootfat:
/usr/bin/makebootfat -o /dev/sdb -E 255 -1 fat12.bin -2 fat16.bin -3 fat32lba.bin -m mbr.bin /tmp/fs-root

19 February 2011

Kees Cook: ptracing siblings

In Ubuntu, the use of ptrace is restricted. The default allowed relationship between the debugger and the debuggee is that parents are allowed to ptrace their descendants. This means that running gdb /some/program and strace /some/program Just Works. Using gdb s attach and strace s -p options need CAP_SYS_PTRACE, care of sudo. The next most common use-case was that of crash handlers needing to do a live ptrace of a crashing program (in the rare case of Apport being insufficient). For example, KDE applications have a segfault handler that calls out to kdeinit and requests that the crash handling process be started on it, and then sits in a loop waiting to be attached to. While kdeinit is the parent of both the crashing program (debuggee) and the crash handling program (debugger), the debugger cannot attach to the debugee since they are siblings, not parent/descendant. To solve this, a prctl() call was added so that the debugee could declare who s descendants were going to attach to it. KDE patched their segfault handler to make the prctl() and everything Just Works again. Breakpad, the crash handler for Firefox and Chromium, was updated to do effectively the same thing, though they had to add code to pass the process id back to the debuggee since they didn t have it handy like KDE. Another use-case was Wine, where for emulation to work correctly, they needed to allow all Wine processes to ptrace each other to correctly emulate Windows. For this, they just declared that all descendants of the wine-server could debug a given Wine process, there-by confining their ptrace festival to just Wine programs. One of the remaining use-cases is that of a debugging IDE that doesn t directly use ptrace itself. For example, qtcreator will launch a program and then later attach to it by launching gdb and using the attach command. This looks a lot like the crash handler use-case, except that the debuggee doesn t have any idea that it is running under an IDE. A simple solution for this is to have the IDE run its programs with the LD_PRELOAD environment variable aimed at a short library that just calls prctl() with the parent process id, and suddenly the IDE and its descendants (i.e. gdb) can debug the program all day long. I ve got an example of this preloadable library written. If it turns out this is generally useful for IDEs, I could package it up like fakeroot and faketime.

20 November 2010

Petter Reinholdtsen: Lenny->Squeeze upgrades, apt vs aptitude with the Gnome and KDE desktop

I'm still running upgrade testing of the Lenny Gnome and KDE Desktop, but have not had time to spend on reporting the status. Here is a short update based on a test I ran 20101118. I still do not know what a correct migration should look like, so I report any differences between apt and aptitude and hope someone else can see if anything should be changed. This is for Gnome: Installed using apt-get, missing with aptitude
apache2.2-bin aptdaemon at-spi baobab binfmt-support browser-plugin-gnash cheese-common cli-common cpp-4.3 cups-pk-helper dmz-cursor-theme empathy empathy-common finger freedesktop-sound-theme freeglut3 gconf-defaults-service gdm-themes gedit-plugins geoclue geoclue-hostip geoclue-localnet geoclue-manual geoclue-yahoo gnash gnash-common gnome gnome-backgrounds gnome-cards-data gnome-codec-install gnome-core gnome-desktop-environment gnome-disk-utility gnome-screenshot gnome-search-tool gnome-session-canberra gnome-spell gnome-system-log gnome-themes-extras gnome-themes-more gnome-user-share gs-common gstreamer0.10-fluendo-mp3 gstreamer0.10-tools gtk2-engines gtk2-engines-pixbuf gtk2-engines-smooth hal-info hamster-applet libapache2-mod-dnssd libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libart2.0-cil libatspi1.0-0 libboost-date-time1.42.0 libboost-python1.42.0 libboost-thread1.42.0 libchamplain-0.4-0 libchamplain-gtk-0.4-0 libcheese-gtk18 libclutter-gtk-0.10-0 libcryptui0 libcupsys2 libdiscid0 libeel2-data libelf1 libepc-1.0-2 libepc-common libepc-ui-1.0-2 libfreerdp-plugins-standard libfreerdp0 libgail-common libgconf2.0-cil libgdata-common libgdata7 libgdl-1-common libgdu-gtk0 libgee2 libgeoclue0 libgexiv2-0 libgif4 libglade2.0-cil libglib2.0-cil libgmime2.4-cil libgnome-vfs2.0-cil libgnome2.24-cil libgnomepanel2.24-cil libgnomeprint2.2-data libgnomeprintui2.2-common libgnomevfs2-bin libgpod-common libgpod4 libgtk2.0-cil libgtkglext1 libgtksourceview-common libgtksourceview2.0-common libmono-addins-gui0.2-cil libmono-addins0.2-cil libmono-cairo2.0-cil libmono-corlib2.0-cil libmono-i18n-west2.0-cil libmono-posix2.0-cil libmono-security2.0-cil libmono-sharpzip2.84-cil libmono-system2.0-cil libmtp8 libmusicbrainz3-6 libndesk-dbus-glib1.0-cil libndesk-dbus1.0-cil libopal3.6.8 libpolkit-gtk-1-0 libpt-1.10.10-plugins-alsa libpt-1.10.10-plugins-v4l libpt2.6.7 libpython2.6 librpm1 librpmio1 libsdl1.2debian libservlet2.4-java libsrtp0 libssh-4 libtelepathy-farsight0 libtelepathy-glib0 libtidy-0.99-0 libxalan2-java libxerces2-java media-player-info mesa-utils mono-2.0-gac mono-gac mono-runtime nautilus-sendto nautilus-sendto-empathy openoffice.org-writer2latex openssl-blacklist p7zip p7zip-full pkg-config python-4suite-xml python-aptdaemon python-aptdaemon-gtk python-axiom python-beautifulsoup python-bugbuddy python-clientform python-coherence python-configobj python-crypto python-cupshelpers python-cupsutils python-eggtrayicon python-elementtree python-epsilon python-evolution python-feedparser python-gdata python-gdbm python-gst0.10 python-gtkglext1 python-gtkmozembed python-gtksourceview2 python-httplib2 python-louie python-mako python-markupsafe python-mechanize python-nevow python-notify python-opengl python-openssl python-pam python-pkg-resources python-pyasn1 python-pysqlite2 python-rdflib python-serial python-tagpy python-twisted-bin python-twisted-conch python-twisted-core python-twisted-web python-utidylib python-webkit python-xdg python-zope.interface remmina remmina-plugin-data remmina-plugin-rdp remmina-plugin-vnc rhythmbox-plugin-cdrecorder rhythmbox-plugins rpm-common rpm2cpio seahorse-plugins shotwell software-center svgalibg1 system-config-printer-udev telepathy-gabble telepathy-mission-control-5 telepathy-salut tomboy totem totem-coherence totem-mozilla totem-plugins transmission-common xdg-user-dirs xdg-user-dirs-gtk xserver-xephyr zip
Installed using apt-get, removed with aptitude
arj bluez-utils cheese dhcdbd djvulibre-desktop ekiga eog epiphany-extensions epiphany-gecko evolution-exchange fast-user-switch-applet file-roller gcalctool gconf-editor gdm gedit gedit-common gnome-app-install gnome-games gnome-games-data gnome-nettool gnome-system-tools gnome-themes gnome-utils gnome-vfs-obexftp gnome-volume-manager gnuchess gucharmap guile-1.8-libs hal libavahi-compat-libdnssd1 libavahi-core5 libavahi-ui0 libbind9-50 libbluetooth2 libcamel1.2-11 libcdio7 libcucul0 libcurl3 libdirectfb-1.0-0 libdmx1 libdvdread3 libedata-cal1.2-6 libedataserver1.2-9 libeel2-2.20 libepc-1.0-1 libepc-ui-1.0-1 libexchange-storage1.2-3 libfaad0 libgadu3 libgalago3 libgd2-noxpm libgda3-3 libgda3-common libggz2 libggzcore9 libggzmod4 libgksu1.2-0 libgksuui1.0-1 libgmyth0 libgnome-desktop-2 libgnome-pilot2 libgnomecups1.0-1 libgnomeprint2.2-0 libgnomeprintui2.2-0 libgpod3 libgraphviz4 libgtk-vnc-1.0-0 libgtkhtml2-0 libgtksourceview1.0-0 libgtksourceview2.0-0 libgucharmap6 libhesiod0 libicu38 libisccc50 libisccfg50 libiw29 libjaxp1.3-java-gcj libkpathsea4 liblircclient0 libltdl3 liblwres50 libmagick++10 libmagick10 libmalaga7 libmozjs1d libmpfr1ldbl libmtp7 libmysqlclient15off libnautilus-burn4 libneon27 libnm-glib0 libnm-util0 libopal-2.2 libosp5 libparted1.8-10 libpisock9 libpisync1 libpoppler-glib3 libpoppler3 libpt-1.10.10 libraw1394-8 libsdl1.2debian-alsa libsensors3 libsexy2 libsmbios2 libsoup2.2-8 libspeexdsp1 libssh2-1 libsuitesparse-3.1.0 libsvga1 libswfdec-0.6-90 libtalloc1 libtotem-plparser10 libtrackerclient0 libvoikko1 libxalan2-java-gcj libxerces2-java-gcj libxklavier12 libxtrap6 libxxf86misc1 libzephyr3 mysql-common rhythmbox seahorse sound-juicer swfdec-gnome system-config-printer totem-common totem-gstreamer transmission-gtk vinagre vino w3c-dtd-xhtml wodim
Installed using aptitude, missing with apt-get
gstreamer0.10-gnomevfs
Installed using aptitude, removed with apt-get
[nothing]
This is for KDE: Installed using apt-get, missing with aptitude
autopoint bomber bovo cantor cantor-backend-kalgebra cpp-4.3 dcoprss edict espeak espeak-data eyesapplet fifteenapplet finger gettext ghostscript-x git gnome-audio gnugo granatier gs-common gstreamer0.10-pulseaudio indi kaddressbook-plugins kalgebra kalzium-data kanjidic kapman kate-plugins kblocks kbreakout kbstate kde-icons-mono kdeaccessibility kdeaddons-kfile-plugins kdeadmin-kfile-plugins kdeartwork-misc kdeartwork-theme-window kdeedu kdeedu-data kdeedu-kvtml-data kdegames kdegames-card-data kdegames-mahjongg-data kdegraphics-kfile-plugins kdelirc kdemultimedia-kfile-plugins kdenetwork-kfile-plugins kdepim-kfile-plugins kdepim-kio-plugins kdessh kdetoys kdewebdev kdiamond kdnssd kfilereplace kfourinline kgeography-data kigo killbots kiriki klettres-data kmoon kmrml knewsticker-scripts kollision kpf krosspython ksirk ksmserver ksquares kstars-data ksudoku kubrick kweather libasound2-plugins libboost-python1.42.0 libcfitsio3 libconvert-binhex-perl libcrypt-ssleay-perl libdb4.6++ libdjvulibre-text libdotconf1.0 liberror-perl libespeak1 libfinance-quote-perl libgail-common libgsl0ldbl libhtml-parser-perl libhtml-tableextract-perl libhtml-tagset-perl libhtml-tree-perl libio-stringy-perl libkdeedu4 libkdegames5 libkiten4 libkpathsea5 libkrossui4 libmailtools-perl libmime-tools-perl libnews-nntpclient-perl libopenbabel3 libportaudio2 libpulse-browse0 libservlet2.4-java libspeechd2 libtiff-tools libtimedate-perl libunistring0 liburi-perl libwww-perl libxalan2-java libxerces2-java lirc luatex marble networkstatus noatun-plugins openoffice.org-writer2latex palapeli palapeli-data parley parley-data poster psutils pulseaudio pulseaudio-esound-compat pulseaudio-module-x11 pulseaudio-utils quanta-data rocs rsync speech-dispatcher step svgalibg1 texlive-binaries texlive-luatex ttf-sazanami-gothic
Installed using apt-get, removed with aptitude
amor artsbuilder atlantik atlantikdesigner blinken bluez-utils cvs dhcdbd djvulibre-desktop imlib-base imlib11 kalzium kanagram kandy kasteroids katomic kbackgammon kbattleship kblackbox kbounce kbruch kcron kdat kdemultimedia-kappfinder-data kdeprint kdict kdvi kedit keduca kenolaba kfax kfaxview kfouleggs kgeography kghostview kgoldrunner khangman khexedit kiconedit kig kimagemapeditor kitchensync kiten kjumpingcube klatin klettres klickety klines klinkstatus kmag kmahjongg kmailcvt kmenuedit kmid kmilo kmines kmousetool kmouth kmplot knetwalk kodo kolf kommander konquest kooka kpager kpat kpdf kpercentage kpilot kpoker kpovmodeler krec kregexpeditor kreversi ksame ksayit kshisen ksig ksim ksirc ksirtet ksmiletris ksnake ksokoban kspaceduel kstars ksvg ksysv kteatime ktip ktnef ktouch ktron kttsd ktuberling kturtle ktux kuickshow kverbos kview kviewshell kvoctrain kwifimanager kwin kwin4 kwordquiz kworldclock kxsldbg libakode2 libarts1-akode libarts1-audiofile libarts1-mpeglib libarts1-xine libavahi-compat-libdnssd1 libavahi-core5 libavc1394-0 libbind9-50 libbluetooth2 libboost-python1.34.1 libcucul0 libcurl3 libcvsservice0 libdirectfb-1.0-0 libdjvulibre21 libdvdread3 libfaad0 libfreebob0 libgd2-noxpm libgraphviz4 libgsmme1c2a libgtkhtml2-0 libicu38 libiec61883-0 libindex0 libisccc50 libisccfg50 libiw29 libjaxp1.3-java-gcj libk3b3 libkcal2b libkcddb1 libkdeedu3 libkdegames1 libkdepim1a libkgantt0 libkleopatra1 libkmime2 libkpathsea4 libkpimexchange1 libkpimidentities1 libkscan1 libksieve0 libktnef1 liblockdev1 libltdl3 liblwres50 libmagick10 libmimelib1c2a libmodplug0c2 libmozjs1d libmpcdec3 libmpfr1ldbl libneon27 libnm-util0 libopensync0 libpisock9 libpoppler-glib3 libpoppler-qt2 libpoppler3 libraw1394-8 librss1 libsensors3 libsmbios2 libssh2-1 libsuitesparse-3.1.0 libswfdec-0.6-90 libtalloc1 libxalan2-java-gcj libxerces2-java-gcj libxtrap6 lskat mpeglib network-manager-kde noatun pmount tex-common texlive-base texlive-common texlive-doc-base texlive-fonts-recommended tidy ttf-dustin ttf-kochi-gothic ttf-sjfonts
Installed using aptitude, missing with apt-get
dolphin kde-core kde-plasma-desktop kde-standard kde-window-manager kdeartwork kdebase kdebase-apps kdebase-workspace kdebase-workspace-bin kdebase-workspace-data kdeutils kscreensaver kscreensaver-xsavers libgle3 libkonq5 libkonq5-templates libnetpbm10 netpbm plasma-widget-folderview plasma-widget-networkmanagement xscreensaver-data-extra xscreensaver-gl xscreensaver-gl-extra xscreensaver-screensaver-bsod
Installed using aptitude, removed with apt-get
kdebase-bin konq-plugins konqueror

22 September 2010

Gunnar Wolf: Damage control: Cleaning up compromised SSH keys

This morning, my laptop was stolen from my parked car while I was jogging. I do not want to make a big deal out of it. Still, even though I am sure it was not targetted at my data (three other people at least were reporting similar facts in the same area), and the laptop's disk will probably just be reformatted, I am trying to limit the possible impact of my cryptographic identification being in somebody else's hands. GPG makes it easy: I had on that machine just my old 1024D key, so it is just matter of generating a revocation certificate. I have done that, and uploaded it to the SKS keyservers - Anyway, here is my revocation certificate:
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: A revocation certificate should follow
iHIEIBEIADIFAkyaOZwrHQJBIGNvbXB1dGVyIGNvbnRhaW5pbmcgdGhpcyBrZXkg
d2FzIHN0b2xlbgAKCRDYDvNai7UnrzWAAKC34eF76JQjxrZqSjNwcC0dU/5VbACg
gMIMmYg91Sl3y8KsZXdGj/rV7UE=
=rdlT
-----END PGP PUBLIC KEY BLOCK-----
But What worries me more is access to the computers my ssh key works for. Yes, the ssh key uses a nontrivial passphrase, but still SSH keys cannot be revoked (and this makes sense, as SSH should not add the delay, or potential impossibility, to check with a remote infrastructure whenever you want to start a session). So, I generated a new key (and stored it at ~/.ssh/id_rsa.new / ~/.ssh/id_rsa.new.pub) and came up with this snippet:
  1. $ OLDKEY=xyHywJuHD3nsfLh03G1TqUEBKSj6NlzMfB1T759haoAQ
  2. $ for host in $(cat .ssh/known_hosts cut -f 1 -d \ cut -f 1 -d ,
  3. sort uniq); do
  4. echo == $host
  5. ssh-copy-id -i .ssh/id_rsa.new.pub $host &&
  6. ssh $host "perl -n -i -e 'next if /$OLDKEY/;print' .ssh/authorized_keys"
  7. done
Points about it you might scratch your head about: Caveats: Oh, by the way: If you noticed me knocking on your SSH ports... please disregard. Possibly at some point I connected to that machine to do something, or it landed in my .ssh/known_hosts for some reason. I currently have 144 hosts registered. I am sure I triggered at least one raised eyebrow. And I will do it from a couple of different computers, to make it less probable that I miss some I have never connected from while at the particular computer I am sitting at right now. So... Any ideas on how to make this better?

13 June 2010

Petter Reinholdtsen: Lenny->Squeeze upgrades, removals by apt and aptitude

My testing of Debian upgrades from Lenny to Squeeze continues, and I've finally made the upgrade logs available from http://people.skolelinux.org/pere/debian-upgrade-testing/. I am now testing dist-upgrade of Gnome and KDE in a chroot using both apt and aptitude, and found their differences interesting. This time I will only focus on their removal plans. After installing a Gnome desktop and the laptop task, apt-get wants to remove 72 packages when dist-upgrading from Lenny to Squeeze. The surprising part is that it want to remove xorg and all xserver-xorg-video* drivers. Clearly not a good choice, but I am not sure why. When asking aptitude to do the same, it want to remove 129 packages, but most of them are library packages I suspect are no longer needed. Both of them want to remove bluetooth packages, which I do not know. Perhaps these bluetooth packages are obsolete? For KDE, apt-get want to remove 82 packages, among them kdebase which seem like a bad idea and xorg the same way as with Gnome. Asking aptitude for the same, it wants to remove 192 packages, none which are too surprising. I guess the removal of xorg during upgrades should be investigated and avoided, and perhaps others as well. Here are the complete list of planned removals. The complete logs is available from the URL above. Note if you want to repeat these tests, that the upgrade test for kde+apt-get hung in the tasksel setup because of dpkg asking conffile questions. No idea why. I worked around it by using 'echo >> /proc/pidofdpkg/fd/0' to tell dpkg to continue. apt-get gnome 72
bluez-gnome cupsddk-drivers deskbar-applet gnome gnome-desktop-environment gnome-network-admin gtkhtml3.14 iceweasel-gnome-support libavcodec51 libdatrie0 libgdl-1-0 libgnomekbd2 libgnomekbdui2 libmetacity0 libslab0 libxcb-xlib0 nautilus-cd-burner python-gnome2-desktop python-gnome2-extras serpentine swfdec-mozilla update-manager xorg xserver-xorg xserver-xorg-core xserver-xorg-input-all xserver-xorg-input-evdev xserver-xorg-input-kbd xserver-xorg-input-mouse xserver-xorg-input-synaptics xserver-xorg-input-wacom xserver-xorg-video-all xserver-xorg-video-apm xserver-xorg-video-ark xserver-xorg-video-ati xserver-xorg-video-chips xserver-xorg-video-cirrus xserver-xorg-video-cyrix xserver-xorg-video-dummy xserver-xorg-video-fbdev xserver-xorg-video-glint xserver-xorg-video-i128 xserver-xorg-video-i740 xserver-xorg-video-imstt xserver-xorg-video-intel xserver-xorg-video-mach64 xserver-xorg-video-mga xserver-xorg-video-neomagic xserver-xorg-video-nsc xserver-xorg-video-nv xserver-xorg-video-openchrome xserver-xorg-video-r128 xserver-xorg-video-radeon xserver-xorg-video-radeonhd xserver-xorg-video-rendition xserver-xorg-video-s3 xserver-xorg-video-s3virge xserver-xorg-video-savage xserver-xorg-video-siliconmotion xserver-xorg-video-sis xserver-xorg-video-sisusb xserver-xorg-video-tdfx xserver-xorg-video-tga xserver-xorg-video-trident xserver-xorg-video-tseng xserver-xorg-video-v4l xserver-xorg-video-vesa xserver-xorg-video-vga xserver-xorg-video-vmware xserver-xorg-video-voodoo xulrunner-1.9 xulrunner-1.9-gnome-support aptitude gnome 129
bluez-gnome bluez-utils cpp-4.3 cupsddk-drivers dhcdbd djvulibre-desktop finger gnome-app-install gnome-mount gnome-network-admin gnome-spell gnome-vfs-obexftp gnome-volume-manager gstreamer0.10-gnomevfs gtkhtml3.14 libao2 libavahi-compat-libdnssd1 libavahi-core5 libavcodec51 libbluetooth2 libcamel1.2-11 libcdio7 libcucul0 libcupsys2 libcurl3 libdatrie0 libdirectfb-1.0-0 libdvdread3 libedataserver1.2-9 libeel2-2.20 libeel2-data libepc-1.0-1 libepc-ui-1.0-1 libfaad0 libgail-common libgd2-noxpm libgda3-3 libgda3-common libgdl-1-0 libgdl-1-common libggz2 libggzcore9 libggzmod4 libgksu1.2-0 libgksuui1.0-1 libgmyth0 libgnomecups1.0-1 libgnomekbd2 libgnomekbdui2 libgnomeprint2.2-0 libgnomeprint2.2-data libgnomeprintui2.2-0 libgnomeprintui2.2-common libgnomevfs2-bin libgpod3 libgraphviz4 libgtkhtml2-0 libgtksourceview-common libgtksourceview1.0-0 libgucharmap6 libhesiod0 libicu38 libiw29 libkpathsea4 libltdl3 libmagick++10 libmagick10 libmalaga7 libmetacity0 libmtp7 libmysqlclient15off libnautilus-burn4 libneon27 libnm-glib0 libnm-util0 libopal-2.2 libosp5 libparted1.8-10 libpoppler-glib3 libpoppler3 libpt-1.10.10 libpt-1.10.10-plugins-alsa libpt-1.10.10-plugins-v4l libraw1394-8 libsensors3 libslab0 libsmbios2 libsoup2.2-8 libssh2-1 libsuitesparse-3.1.0 libswfdec-0.6-90 libtalloc1 libtotem-plparser10 libtrackerclient0 libxalan2-java libxalan2-java-gcj libxcb-xlib0 libxerces2-java libxerces2-java-gcj libxklavier12 libxtrap6 libxxf86misc1 libzephyr3 mysql-common nautilus-cd-burner openoffice.org-writer2latex openssl-blacklist p7zip python-4suite-xml python-eggtrayicon python-gnome2-desktop python-gnome2-extras python-gtkhtml2 python-gtkmozembed python-numeric python-sexy serpentine svgalibg1 swfdec-gnome swfdec-mozilla totem-gstreamer update-manager wodim xserver-xorg-video-cyrix xserver-xorg-video-imstt xserver-xorg-video-nsc xserver-xorg-video-v4l xserver-xorg-video-vga zip apt-get kde 82
cupsddk-drivers karm kaudiocreator kcoloredit kcontrol kde kde-core kdeaddons kdeartwork kdebase kdebase-bin kdebase-bin-kde3 kdebase-kio-plugins kdesktop kdeutils khelpcenter kicker kicker-applets knewsticker kolourpaint konq-plugins konqueror korn kpersonalizer kscreensaver ksplash libavcodec51 libdatrie0 libkiten1 libxcb-xlib0 quanta superkaramba texlive-base-bin xorg xserver-xorg xserver-xorg-core xserver-xorg-input-all xserver-xorg-input-evdev xserver-xorg-input-kbd xserver-xorg-input-mouse xserver-xorg-input-synaptics xserver-xorg-input-wacom xserver-xorg-video-all xserver-xorg-video-apm xserver-xorg-video-ark xserver-xorg-video-ati xserver-xorg-video-chips xserver-xorg-video-cirrus xserver-xorg-video-cyrix xserver-xorg-video-dummy xserver-xorg-video-fbdev xserver-xorg-video-glint xserver-xorg-video-i128 xserver-xorg-video-i740 xserver-xorg-video-imstt xserver-xorg-video-intel xserver-xorg-video-mach64 xserver-xorg-video-mga xserver-xorg-video-neomagic xserver-xorg-video-nsc xserver-xorg-video-nv xserver-xorg-video-openchrome xserver-xorg-video-r128 xserver-xorg-video-radeon xserver-xorg-video-radeonhd xserver-xorg-video-rendition xserver-xorg-video-s3 xserver-xorg-video-s3virge xserver-xorg-video-savage xserver-xorg-video-siliconmotion xserver-xorg-video-sis xserver-xorg-video-sisusb xserver-xorg-video-tdfx xserver-xorg-video-tga xserver-xorg-video-trident xserver-xorg-video-tseng xserver-xorg-video-v4l xserver-xorg-video-vesa xserver-xorg-video-vga xserver-xorg-video-vmware xserver-xorg-video-voodoo xulrunner-1.9 aptitude kde 192
bluez-utils cpp-4.3 cupsddk-drivers cvs dcoprss dhcdbd djvulibre-desktop dosfstools eyesapplet fifteenapplet finger gettext ghostscript-x imlib-base imlib11 indi kandy karm kasteroids kaudiocreator kbackgammon kbstate kcoloredit kcontrol kcron kdat kdeadmin-kfile-plugins kdeartwork-misc kdeartwork-theme-window kdebase-bin-kde3 kdebase-kio-plugins kdeedu-data kdegraphics-kfile-plugins kdelirc kdemultimedia-kappfinder-data kdemultimedia-kfile-plugins kdenetwork-kfile-plugins kdepim-kfile-plugins kdepim-kio-plugins kdeprint kdesktop kdessh kdict kdnssd kdvi kedit keduca kenolaba kfax kfaxview kfouleggs kghostview khelpcenter khexedit kiconedit kitchensync klatin klickety kmailcvt kmenuedit kmid kmilo kmoon kmrml kodo kolourpaint kooka korn kpager kpdf kpercentage kpf kpilot kpoker kpovmodeler krec kregexpeditor ksayit ksim ksirc ksirtet ksmiletris ksmserver ksnake ksokoban ksplash ksvg ksysv ktip ktnef kuickshow kverbos kview kviewshell kvoctrain kwifimanager kwin kwin4 kworldclock kxsldbg libakode2 libao2 libarts1-akode libarts1-audiofile libarts1-mpeglib libarts1-xine libavahi-compat-libdnssd1 libavahi-core5 libavc1394-0 libavcodec51 libbluetooth2 libboost-python1.34.1 libcucul0 libcurl3 libcvsservice0 libdatrie0 libdirectfb-1.0-0 libdjvulibre21 libdvdread3 libfaad0 libfreebob0 libgail-common libgd2-noxpm libgraphviz4 libgsmme1c2a libgtkhtml2-0 libicu38 libiec61883-0 libindex0 libiw29 libk3b3 libkcal2b libkcddb1 libkdeedu3 libkdepim1a libkgantt0 libkiten1 libkleopatra1 libkmime2 libkpathsea4 libkpimexchange1 libkpimidentities1 libkscan1 libksieve0 libktnef1 liblockdev1 libltdl3 libmagick10 libmimelib1c2a libmozjs1d libmpcdec3 libneon27 libnm-util0 libopensync0 libpisock9 libpoppler-glib3 libpoppler-qt2 libpoppler3 libraw1394-8 libsmbios2 libssh2-1 libsuitesparse-3.1.0 libtalloc1 libtiff-tools libxalan2-java libxalan2-java-gcj libxcb-xlib0 libxerces2-java libxerces2-java-gcj libxtrap6 mpeglib networkstatus openoffice.org-writer2latex pmount poster psutils quanta quanta-data superkaramba svgalibg1 tex-common texlive-base texlive-base-bin texlive-common texlive-doc-base texlive-fonts-recommended xserver-xorg-video-cyrix xserver-xorg-video-imstt xserver-xorg-video-nsc xserver-xorg-video-v4l xserver-xorg-video-vga xulrunner-1.9

1 November 2008

Biella Coleman: Voter Fraud

My colleague Mark Crispin Miller has written an excellent piece in the Wall Street Journal Will this Election Be Stolen?. The issue he raises–voter fraud of various stripes–is the single most important issue the American electorate faces. He concludes with the following powerful thought:
It is not the failure or success of any candidate or party that most matters but the exercise of voting rights, and, through them, our self-government. If either team prevails despite the disenfranchisement of some Americans, that victory will mean all that much less; and if your favorite wins, and then the U.S. doesn’t do anything to fix its voting system (and otherwise restore this faltering democracy), that victory of his won’t matter much at all, since We the People will have lost control for good.

16 June 2008

Martin F. Krafft: Setting up an encrypted Debian system

Even though the Debian installer can set up encrypted partitions, it is optimised for systems with a single data partition, unless you want to enter multiple passphrases when the system boots. The installer configures a LUKS volume using cryptsetup, but it provides no mechanism for the use of key files, only interactive passphrases. I like partitioning my disks and use different filesystems for /tmp, /home, /var, and /usr/local for a number of reasons. I don t like entering more passphrases than necessary. If you can identify with that, the following is for you.
Several people have pointed out to me that one can simply create a single encrypted physical volume with the Debian installer and place logical volumes for the various filesystems in there. You still need a separate /boot partition, in any case. Kapil Hari Paranjape has described the approach. This method is much cleaner and to be preferred. It s quite likely that it also improves the speed since only a single kcryptd process takes care of all of the decryption and encryption needs. I shall try it on my next installation and report on whether the Debian installer supports it. In the mean time, the following is left as reference the approach is still fully functional.

Installing the system The first step to setting up an encrypted Debian system is to perform a normal Debian system installation. When you are asked to partition your harddrive and create filesystems, set up all partitions as encrypted volumes (I suggest to go with the installer defaults and use dm-crypt and AES with the default settings, simply because I have no reason to doubt the installer dveeloper team s choices). Make sure to erase the disks in the process the installer has an option for that. Set up the swap partition as an encrypted volume too, but don t worry too much about the settings at this point; we will recreate the swap partition later. Unless you want to boot off an external medium, such as a USB stick, you will need to create an unencrypted partition for /boot. I will return to this topic, which has security implications, further down. The installer will ask you for passphrases for each of the volumes you create. I suggest you pick a secure passphrase for the root volume (/), but simple passphrases for all the other ones (such as a ), since we will reconfigure them to use key files instead.

Using key files A key file is like a passphrase stored in a file on disk; as opposed to what you know , it s a what you have security asset. Thus, you need to store the file somewhere. When I boot up my system, I unlock the root partition with a passphrase entered interactively, which makes the root filesystem available. I store key files for all other volumes in /etc/keys. Obviously, I need to tell cryptsetup to use those. The first step is to create a key file for each partition and to add it as a decryption key to the LUKS volume. You can do all of the following without unmounting the filesystems. See the following example for hda6, which will prompt for the simple passphrase we entered above to unlock the key ring when adding the key file:
umask 077
mkdir /etc/keys
dd if=/dev/urandom of=/etc/keys/hda6.luks bs=4k count=1
cryptsetup luksAddKey /dev/hda6 /etc/keys/hda6.luks
cryptsetup luksKillSlot /dev/hda6 0 --key-file /etc/keys/hda6.luks

The last command wipes the simple passphrase from the key ring and thus makes it unusable. Now we need to tell cryptsetup to use the key file by editing /etc/crypttab and ensuring a line such as the following exists:
hda6_crypt /dev/hda6 /etc/keys/hda6.luks luks

This tells cryptsetup to create the cryptographic volume hda6_crypt from the base device /dev/hda6, using the key file we created above, and letting it know that it s dealing with a LUKS volume. Repeat this for every partition except your root and swap partitions.

Encrypting the swap partition If you are using an encrypted Debian system, you likely have some security requirements to meet. If that s the case, you must also use an encrypted swap partition. The swap partition can be encrypted in two ways:
  • it can be recreated on every boot, using a random passphrase, or
  • it can be created like the other encrypted volumes with a persistent passphrase
If you want to use suspend-to-disk, you cannot use the first approach as it would overwrite your memory footprint stored in the swap partition. Furthermore, you cannot use a key file like the other partitions, since the root filesystem is not (and must not) be mounted by the time the resume process starts and needs to read the decrypted swap partition. The way I solved this is by telling cryptsetup to compute the passphrase of the swap partition from the decryption key of the volume holding the root filesystem; the cryptsetup package implements this with /lib/cryptsetup/scripts/decrypt_derived. Thus, to set up the swap partition, I do the following, assuming hda2 is the partition holding the encrypted swap and the root filesystem is in hda5_crypt:
swapoff /dev/mapper/hda2_crypt
cryptsetup luksClose hda2_crypt
dd if=/dev/urandom of=/dev/hda2
/lib/cryptsetup/scripts/decrypt_derived hda5_crypt \
    cryptsetup luksFormat /dev/hda2 --key-file -
/lib/cryptsetup/scripts/decrypt_derived hda5_crypt \
    cryptsetup luksOpen /dev/hda2 hda2_crypt --key-file -
mkswap /dev/mapper/hda2_crypt

To tell the system about this swap partition, we need to add it to /etc/crypttab and /etc/fstab; make sure, those files contain lines like the following:
/etc/crypttab:
  hda2_crypt /dev/hda2 hda5_crypt luks,keyscript=/lib/cryptsetup/scripts/decrypt_derived
/etc/fstab:
  /dev/mapper/hda2_crypt swap swap sw 0 0

With this in place, as soon as you configure the system for suspend-to-disk, the swap partition will be automatically set up alongside the root filesystem very early during the boot sequence. To figure out which swap partition to make available at that point, cryptsetup checks the following:
  • a line like RESUME=/dev/mapper/hda2_crypt in /etc/initramfs-tools/conf.d/resume
  • a resume device setting in /etc/uswsusp.conf (see uswsusp.conf(5))
  • an entry in /etc/suspend.conf
  • a resume=/dev/mapper/hda2_crypt in the kernel command line
You can inspect /usr/share/initramfs-tools/hooks/cryptroot if you want to know more about this.

Using UUIDs Even though the above is all you have to do, you might want to consider replacing the device paths in /etc/crypttab with persistent ones. One motivation might be the ability to boot off your drive via a USB adapter, which might cause it to appear as /dev/sda instead of /dev/hda. udev is installed by default on Debian systems and it makes persistent links to partitions available under /dev/disk/by-uuid, using the UUID of the content structure (e.g. the LUKS header). It uses /lib/udev/vol_id to determine those link names. If you replace the entries in /etc/crypttab, make sure to update the initramfs (update-initramfs -u -k all) and consider to use the same approach for the /boot filesystem in /etc/fstab; you ll note that all other filesystems use persistent device paths thanks to the dm-crypt layer. If you ever end up booting off the disk through a USB adapter, you might face the problem where the usb_storage subsystem takes too long to activate, so that the cryptsetup script does not find the devices in time. You can either solve this by adding the rootdelay=x parameter or break=mount to the kernel command line. The first will cause the scripts to wait x seconds before trying to configure and mount the root filesystem; the second would give you a shell that you can exit as soon as the kernel spouted its device initialisation messages at you. If you got this far, you might even want to take it further and replace hda5_crypt with cr_root or the like to abstract those silly partition numbers away even further. This is easier than it sounds, but does require several steps. Do not do this if you re not comfortable reviving your system in case it fails to come back up!
  1. replace the old names with the new names in /etc/crypttab for all volumes except the root volume. If you are using a derived passphrase for the swap partition, make sure to put the new name for the root volume into the third column of the swap partition s configuration line.
  2. modify /etc/fstab accordingly, leave the root filesystem s device path alone.
  3. call update-initramfs -u
  4. modify your bootloader to ask the kernel to boot off the new root volume.
  5. replace the root volume s old name with the new one in /etc/fstab.
  6. reboot, and add break=mount to the kernel command line.
  7. at the busybox prompt, edit (vi) /conf/conf.d/cryptroot and change the first field of the root volume s line to the new name.
  8. exit the shell and watch the boot process complete.
  9. finally, replace the root volume s old name with the new one in /etc/crypttab.
If the system fails to boot up again, you can use the backup initial ramdisk, which update-initramfs left in /boot.

Security implications Apart from the usual security implications related to cryptosystems, passphrases, mathematics, user stupidity, and so on, the approach I outlined will leave you with a pretty well-secured system. Obviously, you should make sure to lock your screen whenever you leave the system unattended or the entire encryption is basically useless. There are two attack vectors on your system, both involving physical access to the machine: Other than that, you should be careful when travelling to totalitarian countries, like the Excited States of America, China, and probably the UK. First off, encryption arouses suspicion, and second, border agents might ask you to decrypt the partitions for them to copy or scan, and refusal to do so might get your turned away at the border. When travelling to those countries, make sure to hide your data properly.

Speed implications Obviously, having your entire system encrypted (including swap) will slow it down. I don t have any quantitative information on that, but after several years of using full-disk encryption on my laptop (an X40, which isn t very powerful), I can say that it remains usable, if you don t rely on disk-intensive operations, such as compiling kernels and the like.

Alternative approaches Several alternative approaches exist, all involving an additional device:
  • if you don t like to type in the passphrase for the root filesystem, you could store it on a USB key (or the like); cryptsetup provides /lib/cryptsetup/scripts/passdev, which can be used to deal with such a situation. You can find more information in cryptsetup s README.Debian file.
  • if you don t like the unencrypted /boot partition, you could boot the system off a USB key, which you can keep separate from the system except for when booting and upgrading the kernel. All you need to do for that is install the bootloader and kernel onto the device and configure it to use the proper encryption volume on the harddisk as root filesystem.
  • cryptsetup also comes with support for PKCS#15 smartcards (opensc and openct).
I have chosen neither of these approaches, because the extra security does not make up for the inconvenience, and the danger of an unbootable system in case of loss of forgetting of the additional device.

23 May 2008

MJ Ray: Updates: Fishmonger re-opens, Bombing Exeter, Eurovision

Three quick updates to items I've posted in the past:- Justin Rolfe has re-opened his fishmonger's shop on Alexandra Parade Weston-super-Mare less than six weeks after what Avon Fire called a "lucky escape" when his van crashed off Kewstoke toll road. I'm glad I didn't try to go to an event in Exeter yesterday that I was invited to. I don't remember seeing that restaurant, but the BBC writes it only opened last September. A religious bombing in Devon. Whatever next? Eurovision is tomorrow night. I'm not going to write in detail here this year (because I don't want the pain of dealing with the planet-purgers again) but I might try to guest blog somewhere else and mention it at the end of tomorrow's post.

8 February 2008

Gunnar Wolf: cat STDERR rot13

Cannot help but laugh and share.
I've been triaging and trying to reproduce some oldish bugs on pkg-perl's packages. Some bugs are no longer there, some have to be forwarded upstream, and so on. Usual tasks, yes.
Until I stumbled with #406227. I just have to laugh and share! Hope nobody feels ashamed - The bug is the result of different people coding maybe under pressure and with quite different mindsets :)
For some reason I fail to understand, the submitter's test case (rot13 implemented over a HTTP proxy) is invoked in the report as ./rot13 2>/dev/null. Of course, when trying to debug a bug report, the first thing to do is not to ignore STDERR. So, off goes the 2>/dev/null. What happens next? 0 gwolf@mosca[2]/tmp$ perl ./rot13 &
[1] 4394
0 gwolf@mosca[3]/tmp$ GET -p http://localhost:8080/ http://www.debian.org/
Can't locate object method "filter" via package "UGGC::Cebkl::ObqlSvygre::fvzcyr=UNFU(0k604160)" (perhaps you forgot to load "UGGC::Cebkl::ObqlSvygre::fvzcyr=UNFU(0k604160)"?) at /usr/share/perl5/HTTP/Proxy/FilterStack.pm line 126.
500 EOF when chunk header expected
WTF... Well, at least the program name gives me a clue... Lets try to "decrypt" the error message... gwolf@mosca[4]/tmp$ echo 'UGGC::Cebkl::ObqlSvygre::fvzcyr=UNFU(0k604160)' rot13
HTTP::Proxy::BodyFilter::simple=HASH(0x604160)
hrm... How comes the filter is filtering its own code and only then refusing to find itself!? Ok, time to open up the manpage - Remember, I'm only group-maintaining this pacakge, I am not yet at all familiar with it! Ok, so the core of the filter is when the submitter states: my $proxy = new HTTP::Proxy();
$proxy->push_filter(response => new HTTP::Proxy::BodyFilter::simple(sub tr/a-zA-z/n-za-mN-ZA-M/; ));
While the manpage states it should be invoked as: my $filter = HTTP::Proxy::BodyFilter::simple->new( sub $ $_[1] =~ s/foo/bar/g; );
$proxy->push_filter( response => $filter );
Of course, once looking at it, the answer is simple: The submitter left out which element to act on in the anonymous function body - The $ $_[1] =~ part. Adding it makes gur svygre jbex nf rkcrpgrq... Err, sorry - makes the filter work as expected. Now, bonus points: For the non-Perlers out ther: How come we get the namespace translated as well? Oh, that's very simple: In Perl, as in Python (and concievably other languages I'm unaware of), the object is passed to any of its methods as the first argument. Functions in Perl receive their arguments via @_ (read: the default array). And, of course, the tr (regex-based transliteration) takes by default the first thing it sees - the object itself. And what happens when you apply a (string-oriented) regex to an object? Of course, it gets stringified - which, by default, in Perl means converting it to the closest possible description: "a hash reference blessed as an object of the class such-and-such at this memory location". That string gets worked on, and we get UGGC::Cebkl::ObqlSvygre::fvzcyr=UNFU(0k604160). This proxy does not die on very-very-short web pages, where the whole content fits on one iteration of the code (although it does not work correctly - the text remains unaltered, of course, as it was not worked on), but if the request spans several chunks, the second time the filter is called, it will be... just gibberish. Oh, and what about the extra $ (...) around $_[1]? Oh, simple: The string is passed as a scalar reference, so it can be modified in place. Yes, it's the Perl way of pass-by-reference instead of pass-by-value (the default behaviour): Of course the parameter is only passed as a value. Only that the value is incidentally a reference - but who cares? ;-) Anyway... Many oddities. I would implement the module in a completely different way, and it looks quite backwardish in my book. But then again, TIMTOWTDI.

27 July 2007

Evan Prodromou: 7 Thermidor CCXV

We've had a lot of great blog traffic about Vinismo. I thought I'd try to pull together a few of the better ones. All in all it's been pretty good. I hope we'll get some more, though! tags:

DemoCampMontreal3 report So, it's been a couple of days and I should probably get around to posting my own DemoCampMontreal3 report.
  • Niko and I started off with our own demo for Vinismo. It was a lot of fun: we talked about the reasons for starting the site, the technical, information architecture and graphics/UI design challenges, and what our future extensions are going to be. At the end of it, we took some questions, which was fun. The most interesting for me was from Roberto Rocha, whose TechnoCit is one of my favourite tech columns in Montreal. He asked, "Your typical contributor will be much older. What will you do to make your wiki more accessible to them?" It was a good question I don't have an answer to yet, but I want to think about it more.
  • The second demo was by Heri Rakotomalala, who showed off his social-networking GTD tool, WorkCruncher. It's a TODO list with a twist: items that you don't get done age off the list. You have to re-commit to doing a task on an almost daily basis. I think it's a great and refreshing design; my TODO list gets depressing long and filled with unfinishable tasks, and I get too intimidated to work on ones that really matter. I think Heri might have to make some concessions to people's expectations for TODO lists -- maybe a way to automatically archive tasks, rather than deleting them entirely...?
  • The third demo was by the gang from Defensio, who are providing an great anti-spam Web service similar to Akismet. They had a few examples of where they're different, but I'm not well-versed enough in comment spam issues to understand them. My guess is that since they're getting into the market after Akismet, though, they have the opportunity to make a smarter technology. Their one downside? They used slides -- which the rules of DemoCamp. They did demo the service, though.
  • Fourth up was the indefatigable Simon Law. Simon's project? To turn back time. Talk about ambitious! His effort consisted of making a typical kitchen clock turn backwards. He disassembled the clock and explained how it worked to the audience. It was great, except for two things: the clock didn't work at the end of the demo (although he got it working by the end of DemoCamp), and he took a few minutes to draw a diagram of the clock; in my mind, that's just a low-tech PowerPoint slide.
  • Fifth, and quite fascinating, was a tool that Jerome Paradis showed off. It was a Google-Maps mashup that filtered special emails for an informal private jet sharing network. Apparently, companies who charter private jets often have space in the jets, so they'll make that space available. People who need a last-minute charter jet can send email from their Blackberries and such, and if there's availability they get contacted by the charter companies. The interesting part? These people use a highly structured lingo ("O/W" = "one way"), and Jerome's tool scrapes these emails to make the data into a mapping app for his customers. Very interesting!
All in all, it was a good night -- probably made better by the Argentina Cabernet Sauvignon I had. There's a DemoCampMontreal4 scheduled for August 17th, but I won't be in town for it. Too bad for me! tags:

7 April 2007

Russell Coker: Linux Tour Bus

I have seen buses used for tours that contain bunk beds. If one or more such buses were hired then a group of Linux people could go on a moving Linux conference. This would have to take place in an area with many reasonable size cities in a close area and where there is a good number of Linux people in such cities. Probably the EU is the only area.

A bus (or several buses depending on demand) would then take a group of Linux people through the major cities and have a conference in each one.

Currently there are conferences such as the Debian conference DebConf which receive sufficient sponsorship money to pay for many speakers to attend. Having a similar conference traveling around Europe shouldn't cost any more money and will give plenty of time for the people in the bus to do some coding along the way.

We already have the Geek Cruises, my idea is to do a similar thing but on the road. Also it isn't practical to transport an entire conference, so it would probably just be speakers on buses and the audiences would vary from city to city.

30 October 2006

Axel Beckert: BarCamp Zurich -- Resume

The BarCamp Zurich 2006 is over. On the way there I thought about what I would do during time slots with no interesting talks. But when I tried to make up my personal schedule, I noticed that I rather would have the opposite problem: Too many interesting talks at the same time… Well, to many interesting talks at all, although I only went to tech talks and left out the biz talks. I first went to the Podcasting & Co. talk by Timo Hetzel, since I never heard or made a podcast, but was curious about podcasts in general. Besides statistics and rankings he spoke about where people listen to podcast (most listeners seem to do that during commuting), what people like in podcasts, why companies podcast, etc. And that a very big share of all podcast listeners use iTunes as podcast client and except juice (never heard of it before) all other podcast clients seem to be irrelevant. My conclusion: I haven’t missed anything not having listened to or made podcasts neither do I need to listen or make podcasts in the future. They’re irrelevant. To me. :-) Then I had to choose between the talks AJAX@localhost (PDF) by Harry Fuecks and Realtime Collaborative Text Editing and SubEthaEdit by the Coding Monkeys. I heard about realtime collaborative editing once know that it’s a challenging task for the developer. I also know what AJAX is (and that I would only use or recommend it for bells and whistles, but not for content in general), but “AJAX@localhost” sounded like writing normal applications using AJAX. It sounded interesting and evil at the same time. I had to go there! ;-) Others had similar expectations after reading the talk’s title, so I was quite surprised that it was about something completely different, namely about debugging AJAX on the localhost but under conditions usually only appearing if you’re running AJAX application not from localhost but from somewhere on the net: You may have different lags with every request, so some requests may reach the server before others, which may screw up the whole AJAX application, if the developers didn’t think about it and only tested it on localhost. (Hence the talk’s title…) My conlusion: I will use and recommend AJAX even more seldom, since there seem to be even more design misconceptions than I thought before. But I’ll once have a look at the Webtuesday meeting, he mentioned. For the third time-slot, I didn’t need long to decide where to go: I already knew a little bit about Microformats and I wanted to know more. Tag Trade also sounded interesting, but the second part of the talk’s title, Paid Learning sounded like business and so I had no scruples to cold-shoulder that talk. I probably didn’t learn anything really new in the microformats talk, but my knowledge about microformats is now more concrete, and after talking with Cédric Hüsler later during a break, I would even trust myself to start and define a new microformat. Then I went to the HG Caféteria together with Gürkan and two German guys. While waiting in the queue, we were talking about our jobs and our favourite Linux distributions. I got some rhubarb pie and a rum truffles, assuming that the Caféteria uses no alcohol in their products like all other SV restaurant I know. But this one seemed to have quite a lot of alcohol, since it felt like my breath was burning… Well, this resulted in my second SV feedback form submission… Next I went to Alex Schröder’s talk about multilingual websites, Oddmuse and the Emacs Wiki, although also the talk A-Life about simulating evolution sounded promising. Alex asked the listeners about their experiences with multilingual websites and showed what Oddmuse offers as partial solution to the general multilingualism problems. But regarding the comments from the auditorium, there probably won’t be a perfect solution until computers can translate perfectly… The next talk I visited was Gabor’s talk about his master thesis Organizing E-Mail which resulted in a soon to be released Mozilla Thunderbird extension called BuzzTrack. From the other concepts he showed, I found Microsoft’s SNARF (Social Network and Relationship Finder) and IBM’s Thread Arcs most interesting as well as the fact that there is no e-mail client seems to have a majority at all. Directly after Gabor I had my own talk about Understanding Shell Quoting, so I also couldn’t go to Adrian Heydecker’s talk about Learning with Hypertext and Search Engines. I had only about three and a half listeners of whom several to my surprise where here because they didn’t know what “shell quoting” is. I really didn’t expect that. But that seems to be one of the differences between a BarCamp and a Linux Conferences: People come here to see something new, something they haven’t heard about before. On Linux events most people come, because they already heard about some special topic and want to know more or learn something about it. On Linux event my shell talks usually were attracting many visitors while at a BarCamp, talks presenting an idea, a concept or a tool seem to much more interesting for the attendees. So for the next BarCamp I perhaps exhume my Website Meta Language talk which never seemed to hit the nerve of Linux event attendees, since it tried to “sell” a different concept of generating website than most were used to. At least one listener excepted the talk to be named “shell escaping”, but IMHO escaping is only one quoting technic and it’s not only used for quoting. But perhaps I should take the word “escaping” in the title though for the next time. Happily most of the listeners seem to have learned something new from the talk and Silvan Gebhardt was really happy about his new knowledge about ssh ~ escapes, although I mainly talked about how to quote them than how to use them. :-) During the last slot I visited the session about the upcoming BarCamp Alsace 2 and the yet to be planned BarCamp Rhine, a BarCamp to be held on a ship traveling from Basel in Switzerland down the Rhine, stopping in Strasbourg, Karlsruhe, Rhein-Main-Area and perhaps even Cologne and Amsterdam. Contrary to my initial thoughts, the day was over very fast and I had no single boring minute during the BarCamp. Wow! After we’ve been kicked out of the building by ETH janitors, we joined again at the Bar N-68. On the way there I met Urban M ller who attended BarCamp Zurich, too. We talked quite a lot and it was very interesting to see behind the scenes of e.g. map.search.ch. Later I joined the French speaking table, talking with Gregoire Japiot from WineCamp France and Alex Schröder. Around 9pm I left the N-68 as one of the last BarCampers, tired but with new knowledge, new ideas, new acquaintances and a new hobby: BarCamping. What a luck that BarCamps aren’t that often, otherwise I couldn’t afford this new hobby. ;-) As a relaxing end I met with Alex Schröder and Christophe Ducamp on Sunday morning for brunch in the restaurant Gloria in the Industriequartier. When we were leaving the Gloria I noticed their book board with a lots of BookCrossing books and I took “The Da Vinci Code” with me, since I saw the movie and people were telling me that the book is much better. I’ll see…

Next.