Search Results: "ag"

8 March 2014

Benjamin Mako Hill: V-Day

My friend Noah mentioned the game VVVVVV. I was confused because I thought he was talking about the visual programming language vvvv. I went to Wikipedia to clear up my confusion but ended up on the article on VVVVV which is about the Latin phrase vi veri universum vivus vici meaning, by the power of truth, I, while living, have conquered the universe . There is no Wikipedia article on VVVVVVV. That would be ridiculous.

7 March 2014

Steve Kemp: Time to get back to my roots: Perl

Today I wrote a perl Test::RemoteServer module:
#!/usr/bin/perl -w -I.
use strict;
use warnings;
use Test::More tests => 4;
use Test::RemoteServer;
#
#  Ping Tests
#
ping_ok( "192.168.0.1",       "Website host is up: IPv4" );
ping6_ok( "www.steve.org.uk", "Website host is up: IPv6" );
#
#  Socket tests
#
socket_open( "ipv4.steve.org.uk", "2222", "OpenSSH is running" );
socket_closed( "ipv4.steve.org.uk", "22", "OpenSSH is not available on :22" );
I can see a lot of value in defining tests that are carried out against remote hosts - even if they're more basic than the kind of comprehensive testing you'd get via Custodian, Nagios, etc. Being able to run "make test" and remotely probe services is cool. Unfortunately I suspect the new-hotness is to couple the testing with your Chef, Puppet, CFengine, Slaughter, Ansible, etc, policies. That way you have two things: Coming to CPAN in the near future anyway, I can throw it up on Github in advance if there is any interest..

Keith Packard: glamor-hacking

Brief Glamor Hacks Eric Anholt started writing Glamor a few years ago. The goal was to provide credible 2D acceleration based solely on the OpenGL API, in particular, to implement the X drawing primitives, both core and Render extension, without any GPU-specific code. When he started, the thinking was that fixed-function devices were still relevant, so that original code didn t insist upon modern OpenGL features like GLSL shaders. That made the code less efficient and hard to write. Glamor used to be a side-project within the X world; seen as something that really wasn t very useful; something that any credible 2D driver would replace with custom highly-optimized GPU-specific code. Eric and I both hoped that Glamor would turn into something credible and that we d be able to eliminate all of the horror-show GPU-specific code in every driver for drawing X text, rectangles and composited images. That hadn t happened though, until now. Fast forward to the last six months. Eric has spent a bunch of time cleaning up Glamor internals, and in fact he s had it merged into the core X server for version 1.16 which will be coming up this July. Within the Glamor code base, he s been cleaning some internal structures up and making life more tolerable for Glamor developers. Using libepoxy A big part of the cleanup was a transition all of the extension function calls to use his other new project, libepoxy, which provides a sane, consistent and performant API to OpenGL extensions for Linux, Mac OS and Windows. That library is awesome, and you should use it for everything you do with OpenGL because not using it is like pounding nails into your head. Or eating non-tasty pie. Using VBOs in Glamor One thing he recently cleaned up was how to deal with VBOs during X operations. VBOs are absolutely essential to modern OpenGL applications; they re really the only way to efficiently pass vertex data from application to the GPU. And, every other mechanism is deprecated by the ARB as not a part of the blessed core context . Glamor provides a simple way of getting some VBO space, dumping data into it, and then using it through two wrapping calls which you use along with glVertexAttribPointer as follows:
pointer = glamor_get_vbo_space(screen, size, &offset);
glVertexAttribPointer(attribute_location, count, type,
              GL_FALSE, stride, offset);
memcpy(pointer, data, size);
glamor_put_vbo_space(screen);
glamor_get_vbo_space allocates the specified amount of VBO space and returns a pointer to that along with an offset , which is suitable to pass to glVertexAttribPointer. You dump your data into the returned pointer, call glamor_put_vbo_space and you re all done. Actually Drawing Stuff At the same time, Eric has been optimizing some of the existing rendering code. But, all of it is still frankly terrible. Our dream of credible 2D graphics through OpenGL just wasn t being realized at all. On Monday, I decided that I should go play in Glamor for a few days, both to hack up some simple rendering paths and to familiarize myself with the insides of Glamor as I m getting asked to review piles of patches for it, and not understanding a code base is a great way to help introduce serious bugs during review. I started with the core text operations. Not because they re particularly relevant these days as most applications draw text with the Render extension to provide better looking results, but instead because they re often one of the hardest things to do efficiently with a heavy weight GPU interface, and OpenGL can be amazingly heavy weight if you let it. Eric spent a bunch of time optimizing the existing text code to try and make it faster, but at the bottom, it actually draws each lit pixel as a tiny GL_POINT object by sending a separate x/y vertex value to the GPU (using the above VBO interface). This code walks the array of bits in the font and checking each one to see if it is lit, then checking if the lit pixel is within the clip region and only then adding the coordinates of the lit pixel to the VBO. The amazing thing is that even with all of this CPU and GPU work, the venerable 6x13 font is drawn at an astonishing 3.2 million glyphs per second. Of course, pure software draws text at 9.3 million glyphs per second. I suspected that a more efficient implementation might be able to draw text a bit faster, so I decided to just start from scratch with a new GL-based core X text drawing function. The plan was pretty simple:
  1. Dump all glyphs in the font into a texture. Store them in 1bpp format to minimize memory consumption.
  2. Place raw (integer) glyph coordinates into the VBO. Place four coordinates for each and draw a GL_QUAD for each glyph.
  3. Transform the glyph coordinates into the usual GL range (-1..1) in the vertex shader.
  4. Fetch a suitable byte from the glyph texture, extract a single bit and then either draw a solid color or discard the fragment.
This makes the X server code surprisingly simple; it computes integer coordinates for the glyph destination and glyph image source and writes those to the VBO. When all of the glyphs are in the VBO, it just calls glDrawArrays(GL_QUADS, 0, 4 * count). The results were encouraging :
1: fb-text.perf
2: glamor-text.perf
3: keith-text.perf
       1                 2                           3                 Operation
------------   -------------------------   -------------------------   -------------------------
   9300000.0      3160000.0 (     0.340)     18000000.0 (     1.935)   Char in 80-char line (6x13) 
   8700000.0      2850000.0 (     0.328)     16500000.0 (     1.897)   Char in 70-char line (8x13) 
   6560000.0      2380000.0 (     0.363)     11900000.0 (     1.814)   Char in 60-char line (9x15) 
   2150000.0       700000.0 (     0.326)      7710000.0 (     3.586)   Char16 in 40-char line (k14) 
    894000.0       283000.0 (     0.317)      4500000.0 (     5.034)   Char16 in 23-char line (k24) 
   9170000.0      4400000.0 (     0.480)     17300000.0 (     1.887)   Char in 80-char line (TR 10) 
   3080000.0      1090000.0 (     0.354)      7810000.0 (     2.536)   Char in 30-char line (TR 24) 
   6690000.0      2640000.0 (     0.395)      5180000.0 (     0.774)   Char in 20/40/20 line (6x13, TR 10) 
   1160000.0       351000.0 (     0.303)      2080000.0 (     1.793)   Char16 in 7/14/7 line (k14, k24) 
   8310000.0      2880000.0 (     0.347)     15600000.0 (     1.877)   Char in 80-char image line (6x13) 
   7510000.0      2550000.0 (     0.340)     12900000.0 (     1.718)   Char in 70-char image line (8x13) 
   5650000.0      2090000.0 (     0.370)     11400000.0 (     2.018)   Char in 60-char image line (9x15) 
   2000000.0       670000.0 (     0.335)      7780000.0 (     3.890)   Char16 in 40-char image line (k14) 
    823000.0       270000.0 (     0.328)      4470000.0 (     5.431)   Char16 in 23-char image line (k24) 
   8500000.0      3710000.0 (     0.436)      8250000.0 (     0.971)   Char in 80-char image line (TR 10) 
   2620000.0       983000.0 (     0.375)      3650000.0 (     1.393)   Char in 30-char image line (TR 24)
This is our old friend x11perfcomp, but slightly adjusted for a modern reality where you really do end up drawing billions of objects (hence the wider columns). This table lists the performance for drawing a range of different fonts in both poly text and image text variants. The first column is for Xephyr using software (fb) rendering, the second is for the existing Glamor GL_POINT based code and the third is the latest GL_QUAD based code. As you can see, drawing points for every lit pixel in a glyph is surprisingly fast, but only about 1/3 the speed of software for essentially any size glyph. By minimizing the use of the CPU and pushing piles of work into the GPU, we manage to increase the speed of most of the operations, with larger glyphs improving significantly more than smaller glyphs. Now, you ask how much code this involved. And, I can truthfully say that it was a very small amount to write:
 Makefile.am             2 
 glamor.c                5 
 glamor_core.c           8 
 glamor_font.c         181 ++++++++++++++++++++
 glamor_font.h          50 +++++
 glamor_priv.h          26 ++
 glamor_text.c         472 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 glamor_transform.c      2 
 8 files changed, 741 insertions(+), 5 deletions(-)
Let s Start At The Very Beginning The results of optimizing text encouraged me to start at the top of x11perf and see what progress I could make. In particular, looking at the current Glamor code, I noticed that it did all of the vertex transformation with the CPU. That makes no sense at all for any GPU built in the last decade; they ve got massive numbers of transistors dedicated to performing precisely this kind of operation. So, I decided to see what I could do with PolyPoint. PolyPoint is absolutely brutal on any GPU; you have to pass it two coordinates for each pixel, and so the very best you can do is send it 32 bits, or precisely the same amount of data needed to actually draw a pixel on the frame buffer. With this in mind, one expects that about the best you can do compared with software is tie. Of course, the CPU version is actually computing an address and clipping, but those are all easily buried in the cost of actually storing a pixel. In any case, the results of this little exercise are pretty close to a tie the CPU draws 190,000,000 dots per second and the GPU draws 189,000,000 dots per second. Looking at the vertex and fragment shaders generated by the compiler, it s clear that there s room for improvement. The fragment shader is simply pulling the constant pixel color from a uniform and assigning it to the fragment color in this the simplest of all possible shaders:
uniform vec4 color;
void main()
 
       gl_FragColor = color;
 ;
This generates five instructions:
Native code for point fragment shader 7 (SIMD8 dispatch):
   START B0
   FB write target 0
0x00000000: mov(8)          g113<1>F        g2<0,1,0>F                        align1 WE_normal 1Q  ;
0x00000010: mov(8)          g114<1>F        g2.1<0,1,0>F                      align1 WE_normal 1Q  ;
0x00000020: mov(8)          g115<1>F        g2.2<0,1,0>F                      align1 WE_normal 1Q  ;
0x00000030: mov(8)          g116<1>F        g2.3<0,1,0>F                      align1 WE_normal 1Q  ;
0x00000040: sendc(8)        null            g113<8,8,1>F
                render ( RT write, 0, 4, 12) mlen 4 rlen 0        align1 WE_normal 1Q EOT  ;
   END B0
As this pattern is actually pretty common, it turns out there s a single instruction that can replace all four of the moves. That should actually make a significant difference in the run time of this shader, and this shader runs once for every single pixel. The vertex shader has some similar optimization opportunities, but it only runs once for every 8 pixels with the SIMD format flipped around, the vertex shader can compute 8 vertices in parallel, so it ends up executing 8 times less often. It s got some redundant moves, which could be optimized by improving the copy propagation analysis code in the compiler. Of course, improving the compiler to make these cases run faster will probably make a lot of other applications run faster too, so it s probably worth doing at some point. Again, the amount of code necessary to add this path was tiny:
 Makefile.am             1 
 glamor.c                2 
 glamor_polyops.c      116 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 glamor_priv.h           8 +++
 glamor_transform.c    118 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 glamor_transform.h     51 ++++++++++++++++++++++
 6 files changed, 292 insertions(+), 4 deletions(-)
Discussion of Results These two cases, text and points, are probably the hardest operations to accelerate with a GPU and yet a small amount of OpenGL code was able to meet or beat software easily. The advantage of this work over traditional GPU 2D acceleration implementations should be pretty clear this same code should work well on any GPU which offers a reasonable OpenGL implementation. That means everyone shares the benefits of this code, and everyone can contribute to making 2D operations faster. All of these measurements were actually done using Xephyr, which offers a testing environment unlike any I ve ever had build and test hardware acceleration code within a nested X server, debugging it in a windowed environment on a single machine. Here s how I m running it:
$ ./Xephyr  -glamor :1 -schedMax 2000 -screen 1024x768 -retro
The one bit of magic here is the -schedMax 2000 flag, which causes Xephyr to update the screen less often when applications are very busy and serves to reduce the overhead of screen updates while running x11perf. Future Work Having managed to accelerate 17 of the 392 operations in x11perf, it s pretty clear that I could spend a bunch of time just stepping through each of the remaining ones and working on them. Before doing that, we want to try and work out some general principles about how to handle core X fill styles. Moving all of the stipple and tile computation to the GPU will help reduce the amount of code necessary to fill rectangles and spans, along with improving performance, assuming the above exercise generalizes to other primitives. Getting and Testing the Code Most of the changes here are from Eric s glamor-server branch:
git://people.freedesktop.org/~anholt/xserver glamor-server
The two patches shown above, along with a pair of simple clean up patches that I ve written this week are available here:
git://people.freedesktop.org/~keithp/xserver glamor-server
Of course, as this now uses libepoxy, you ll need to fetch, build and install that before trying to compile this X server. Because you can try all of this out in Xephyr, it s easy to download and build this X server and then run it right on top of your current driver stack inside of X. I d really like to hear from people with Radeon or nVidia hardware to know whether the code works, and how it compares with fb on the same machine, which you get when you elide the -glamor argument from the example Xephyr command line above.

Russell Coker: Depression

Andrew Solomon gave an interesting TED talk about depression [1]. I ve had problems with depression at various times through my life, about 18 months ago I recognised it as depression after reading a list of depression symptoms on the Beyond Blue site. I think that they have changed their site recently they now have an interactive checklist for depression on their web site [2] (or maybe I just missed the interactive part last time). There is a strong correlation between Autism and depression, this is noted both in research and described on the web, Elspeth s article on Bluehackers is a good description of this [3]. Her experiences differ from mine in some ways, but it s within the common range of experiences you see described on Autism forums etc. Depression is getting more widely known, organisations such as Beyond Blue and Bluehackers are doing good work in spreading information to people who might be depressed and people who know them. The general advice is to see a psychologist, which is good advice for average people. Alexithymia and Choice of Psychologists One problem with such advice is that it doesn t apply so well to people with alexithymia (read the Wikipedia page) [4], that means most people on the Autism Spectrum. The Wikipedia page says psychosomatic illness or substance abuse is frequently exacerbated should these individuals enter psychotherapy . Based on people I know and accounts I ve read on the Internet I expect that anyone on the Autism Spectrum who sees a psychologist that doesn t specialise in Autism (which means most psychologists) will get a result that s about the opposite of what one might desire. In theory a psychologist could recognise someone as being possibly on the Autism Spectrum and refer them to an expert for assessment, but I haven t heard of that happening to an adult. I think that most people who have some degree of alexithymia will avoid psychologists, without ever seeing one you can just know that it s going to be unpleasant. So while you wouldn t want someone who has alexithymia to visit a random psychologist in practice that shouldn t happen too often as such people will be more likely to reject any advice about seeing a psychologist. My page of Autism self-diagnosis tests has a link to an Alexithymia test [5]. If you get a high score on that test (or if taking the test seems too unpleasant) then it s best to see a psychologist who specialises in Autism. Such psychologists are usually happy to work for people who don t quite meet the Autism diagnostic criteria, but they may strongly recommend an Autism assessment so that they can determine the best strategies for treatment. In terms of addressing such problems it seems that the best thing we can do is try and reduce the stigma associated with Autism. The vast majority of people on the Autism Spectrum have little in common with Rain Man. Many of the so-called Autism advocacy organisations make things worse by implying that everyone who is Autistic is unable to live an independent life which helps them in fundraising but doesn t help us.

Vincent Fourmond: Removing the background of an image

While in the process of polishing up a paper, I had to draw figures using a well-known proprietary program for drawing chemical formula whose name I will not write down, out of decency (but you know which one it is anyway). I'm using the mac version (yes, no version for Linux) which has a neat glitch in that the "Save As" button does not work at all: no way to export your chemical formula as a SVG/PDF/PNG picture or whatever ! So I resorted to the neat Print as PDF feature of macintosh. I had to edit the resulting PDF files by hand as some of the colors were wrong (great !), and when I tried to include the PDF, I realized they had a white background... No way !So I dug in ImageMagick's convert documentation, and came up with the following code that converts the PDF file into a PNG with a transparent background:
convert -density 600 figure.pdf -channel alpha \
  -fx '((r == 1 && g == 1 && b == 1) ? 0 : 1)' figure.png
There are still some few points that look white but are probably not that white (due to antialiased rendering of the PDF file ?). There are many more things that can be done using the -fx operator, this page was helpful to me !Edit: while the alpha channel seems to be on by default for PDF files, it is not necessarily the case for all images. If the above doesn't work, try adding -alpha Set before the -fx bit.

6 March 2014

Steve Kemp: So I bought some new hardware, for audio purposes.

This week I received a logitech squeezebox radio, which is basically an expensive toy that allows you to listen to either "internet radio", or music streamed from your own PC via a portable device that accesses the network wirelessly. The main goal of this purchase was to allow us to listen to media stored on a local computer in the bedroom, or living-room. The hardware scans your network looking for a media server, so the first step is to install that: The media-server has a couple of open ports; one for streaming the media, and one for a user-browsable HTML interface. Interestingly the radio-device shows up in the web-interface, so you can mess around with the currently loaded playlist from your office, while your wife is casually listening to music in the bedroom. (I'm not sure if that's a feature or not yet ;) Although I didn't find any alternative server-implementations I did find a software-client which you can use to play music from the central server - slimp3slave - and again you can push playlists, media, etc, to this. My impressions are pretty positive; the device was too expensive, certainly I wouldn't buy two, but it is functional. The user-interface is decent, and the software being available and open is a big win. Downsides? No remote-control for the player, because paying an additional 70 is never going to happen, but otherwise I can't think of anything. (Shame the squeezebox product line seems to have been cancelled (?)) Procmail Alternatives?
Although I did start hacking a C & Lua alternative, it looks like there are enough implementations out there that I don't feel so strongly any more. I'm working in a different way to most people, rather than sort mails at delivery time I'm going to write a trivial daemon that will just watch ~/Maildir/.Incoming, and move mails out of there. That means that no errors will cause mail to be lost at SMTP/delivery time. I'm going to base my work on Email::Filter since it offers 90% of the primitives I want. The only missing thing is the ability to filter mails via external commands which has now been reported as a bug/omission.

Ian Wienand: Skipping pages with django.core.paginator

Here's a little snippet for compressing the length of long Django pagination by just showing context around the currently selected page. What I wanted was the first few and last few pages always selectable with some context pages around the currently selected page; e.g. Example of skip markers in paginator If that's what you're looking for, some variation on below may be of use. In this approach, you build up a list of pages similar to the paginator object page_range but with only the relevant pages and the skip-markers identified.
from django.core.paginator import Paginator
import unittest
class Pages:
    def __init__(self, objects, count):
        self.pages = Paginator(objects, count)
    def pages_to_show(self, page):
        # pages_wanted stores the pages we want to see, e.g.
        #  - first and second page always
        #  - two pages before selected page
        #  - the selected page
        #  - two pages after selected page
        #  - last two pages always
        #
        # Turning the pages into a set removes duplicates for edge
        # cases where the "context pages" (before and after the
        # selected) overlap with the "always show" pages.
        pages_wanted = set([1,2,
                            page-2, page-1,
                            page,
                            page+1, page+2,
                            self.pages.num_pages-1, self.pages.num_pages])
        # The intersection with the page_range trims off the invalid
        # pages outside the total number of pages we actually have.
        # Note that includes invalid negative and >page_range "context
        # pages" which we added above.
        pages_to_show = set(self.pages.page_range).intersection(pages_wanted)
        pages_to_show = sorted(pages_to_show)
        # skip_pages will keep a list of page numbers from
        # pages_to_show that should have a skip-marker inserted
        # after them.  For flexibility this is done by looking for
        # anywhere in the list that doesn't increment by 1 over the
        # last entry.
        skip_pages = [ x[1] for x in zip(pages_to_show[:-1],
                                         pages_to_show[1:])
                       if (x[1] - x[0] != 1) ]
        # Each page in skip_pages should be follwed by a skip-marker
        # sentinel (e.g. -1).
        for i in skip_pages:
            pages_to_show.insert(pages_to_show.index(i), -1)
        return pages_to_show
class TestPages(unittest.TestCase):
    def runTest(self):
        objects = [x for x in range(0,1000)]
        p = Pages(objects, 10)
        self.assertEqual(p.pages_to_show(0),
                         [1, 2, -1, 99, 100])
        self.assertEqual(p.pages_to_show(1),
                         [1,2,3,-1,99,100])
        self.assertEqual(p.pages_to_show(2),
                         [1,2,3,4,-1,99,100])
        self.assertEqual(p.pages_to_show(3),
                         [1,2,3,4,5,-1,99,100])
        self.assertEqual(p.pages_to_show(4),
                         [1,2,3,4,5,6,-1,99,100])
        self.assertEqual(p.pages_to_show(5),
                         [1,2,3,4,5,6,7,-1,99,100])
        self.assertEqual(p.pages_to_show(6),
                         [1,2,-1,4,5,6,7,8,-1,99,100])
        self.assertEqual(p.pages_to_show(7),
                         [1,2,-1,5,6,7,8,9,-1,99,100])
        self.assertEqual(p.pages_to_show(50),
                         [1,2,-1,48,49,50,51,52,-1,99,100])
        self.assertEqual(p.pages_to_show(93),
                         [1,2,-1,91,92,93,94,95,-1,99,100])
        self.assertEqual(p.pages_to_show(94),
                         [1,2,-1,92,93,94,95,96,-1,99,100])
        self.assertEqual(p.pages_to_show(95),
                         [1,2,-1,93,94,95,96,97,-1,99,100])
        self.assertEqual(p.pages_to_show(96),
                         [1,2,-1,94,95,96,97,98,99,100])
        self.assertEqual(p.pages_to_show(97),
                         [1,2,-1,95,96,97,98,99,100])
        self.assertEqual(p.pages_to_show(98),
                         [1,2,-1,96,97,98,99,100])
        self.assertEqual(p.pages_to_show(99),
                         [1,2,-1,97,98,99,100])
        self.assertEqual(p.pages_to_show(100),
                         [1,2,-1,98,99,100])
if __name__ == '__main__':
    unittest.main()
Then somehow pass through the pages_to_show to your view (below I added it to the paginator object passed) and use a template along the lines of
<ul class="pagination">
 % if pages.has_previous % 
  <li><a href="foo.html?page=  pages.previous_page_number  ">&laquo;</a></li>
 % else % 
  <li class="disabled"><a href="#">&laquo;</a></li>
 % endif % 
 % for page in pages.pages_to_show % 
   % if page == -1 % 
  <li class="disabled"><a href="#">&hellip;</a></li>
   % elif page == pages.number % 
  <li class="active"><a href="#">  page_num  </a></li>
   % else % 
  <li><a href="foo.html?page=  page_num  "> page_num </a>
   % endif % 
 % endfor % 
 % if pages.has_next % 
  <li><a href="foo.html?page=  pages.next_page_number  ">&raquo;</a></li>
 % else % 
  <li class="disabled"><a href="#">&raquo;</a></li>
 % endif % 
</ul>

Andrew Pollock: [life] Day 37, Playgroup, bubble tea, and Android phones

Today was Playgroup. We had a pretty early start to the day, and Zoe wanted porridge for breakfast, so we made that together in the Thermomix, and she got dressed while it cooked. After a bit of TV, we biked over. I got to witness semi-firsthand my first mother-to-mother altercation. I said hi to a mother I'd met a previous week, and asked her how she was, and got "Pissed off" as the response. Apparently a reasonably large boy (I'd put him at about the 4 mark) had been in the face of her daughter, and distressed her daughter a bit, so she'd apparently politely told the boy to back off, and then been revved by the mother of the boy for talking to her son. The boy continued hanging around her daughter, and she left Playgroup not long after telling me all about it. I worked out later on that the mother of the boy was one of the women who helped run the Playgroup. Fun times. He was a bit out of control. He full on screamed at me at the end of Playgroup for no reason. His mother did make him apologise to me for that, without me requesting it. There were some dress ups (mostly skirts) today, so Zoe had a fabulous time cavorting around on the stage in a big pink lacy skirt, doing "ballet". She dragged me up on stage with her, and we danced around for a bit. It was quite fun. She was very helpful at pack up time, and was bringing a lot of equipment back to the storage room at the back of the hall. I was helping load stuff into the large crate in the storage room. After parachute time, I lost sight of her briefly, and after a quick search, deduced that she must have gotten herself locked in the storage room. Sure enough, that's where she was. She was only missing for as long as it took me to notice and search in all the other possible places, so she wasn't particularly distressed. Poor thing. After Playgroup, we biked back home, and I had another photo I wanted to print out, so we drove over to Officeworks and had a sausage sizzle after we got the photo printed out, and then drove back home. Zoe had a pretty good bedtime last night, so I decided to try skipping her nap today. I wanted to get the 5-8 player extension pack for Settlers of Catan, which Mind Games in the Myer Centre had, so we hopped on a bus to the city. When we've been on longer public transport rides, Zoe's asked me if she can play games on my phone. This of course keeps her amused, but leaves me with nothing to amuse myself (I need to remember to bring a book). I have this cupboard full of old Android phones, so I decided to blow the dust off my old Nexus S and make it a dedicated games phone for her when we're on public transport. This worked pretty well today. She also decided to bring her handbag, which was convenient. I think she was pretty excited at the idea of her "own phone", even if it lacked a SIM card. After we'd bought the extension pack, I thought we could do something slightly more fun than just traipse into the city and traipse back out again. I've been meaning to introduce her to bubble tea for a while, so took this opportunity to track some down in the Myer Centre. We settled on some honeydew milk tea. She liked the honeydew, but after a couple of mouth fulls, was over the tapioca balls. She was a good sport and gave them a good college try though. It was fun watching her play around with the phone on the way home, and just fumbling her way around the user interface. She found the alarm setting UI, and was having a great old time creating alarms. (I'm glad I cleared them all out when we got home, there was at least one set for 4am). She had fun exploring all the ring tones. Towards the end of the journey home she figured out the camera, and was having a fabulous time taking snaps out the bus window. She still needs to work on keeping her hands and fingers away from the lens. She watched some TV while I made dinner and then I joined her on the couch for a cuddle while I flicked through my copy of 50 Dangerous Things (you should let your children do) and 365 Science Experiments. My cousin Renata came around for dinner with us tonight. Zoe didn't have a particularly great dinner, but she did go to bed nice and easily. I'm not sure how she'll sleep tonight. I've gone back to having a light-weight doona on her, as it's possible last night's wake up was temperature-related.

5 March 2014

Steve Kemp: Replacing ugly things would save the world many hours

There are some tools that we use daily, whether we realize it or not, that are unduly ugly. Over time you learn to use them and you forget just how hard they are to learn, and you take it for granted. Today I had to guide somebody through using procmail, and I'd forgotten how annoying it is. In brief I use procmail in three ways, each of which I had to document: There are some projects that are too entrenched to ever be replaced ("make", I'm looking at you), but procmail? I reckon there's a chance a replacement would be useful, quickly. Then again, maybe I'm biased.

Joachim Breitner: Creative use of screen-message

I just learnt that the math and computer science student body is using screen-message to power an information screen in their room:
There are five instances of screen-message, configured to use non-standard colors and a fixed-width fonts, and controlled by the rather new remote control feature, where screen-messages repeatedly keeps reading text from standard input until a form feed character (ASCII 0x0C) comes, and displays that. The window manager ratpoison takes care of tiling the instances. It looks quite a bit like a poor man s version of dividuum s great info-beamer. I was told that they also tried out info-beamer, but the Raspberry Pi was too slow and weak for that, while screen-message and ratpoison run happily on such small hardware. Correction: Info-beamer was not tried, but rather ruled out as overkill for this task.

H ctor Or n Mart nez: Debian build system

There are many ways to build Debian distributions from source. None of them are trivial enough for beginners and all of them require complex setups. For instance, the Debian official setup uses the following software components: Around the core components, there are other software components needed to run the Debian distribution: Thanks to Debian ftp-master and buildd team, all software is built for several architectures and several ports. Most Debian infrastructure is managed and maintained by Debian System Administration team.

Simplified Debian build infrastructure

Debian Wiki has been growing different random pages trying to ease the setup and configuration problems concerning to Debian build system infrastructure. The above infrastructure barely documents what it is involved to create and generate Debian unstable ( sid ) distribution suite. In order to produce Debian stable distribution suite, there are software transitions to happen and yet another Debian team gets involved in the process, our beloved Debian release team. Once distribution reaches its maturity level and gets released as in its stable version, it gets updates also lead by release team and security related updates, which yet another team is responsible for them, the Debian security team. and there are lots of Debian teams doing many other things you might enjoy, have a byte

Andrew Pollock: [life] Day 36, Kindergarten, cleaning and pee pee accidents in hardware stores

Today was my "clean the house" day. I made a start, then went for my chiropractic assessment and adjustment. I now only need to see him fortnightly, which is nice. I knocked out most of the cleaning, and then went for a massage and had some lunch. The next thing I knew it was about time to pick up Zoe. I biked to Kindergarten, and she was asleep again. She woke up reasonably easily, but was grumpy not long after that. She was very uncooperative, and didn't want to go home on the bike, and wouldn't let me put sunscreen on her. After a bit of work, I managed to extract from her that she didn't like my "sweaty hands" from riding the bike there, so I washed my hands and then she let me sunscreen her up. I'll try doing pick ups in the car next week I think. We got home, and I needed to get some more kitty litter, so we went out to Pet Barn. Zoe had a great time checking out all the fish, and also playing with aquarium decor. It got me thinking that maybe an aquarium might be a good birthday present for her, if not this year, maybe next year. It turns out that fish aren't particularly expensive, so they'd make for easy presents for people. After that, we popped across to Bunnings, because I needed a couple of things. While we were there, Zoe announced she wanted to go to the toilet, so we started trekking off to where I thought the toilets were. Turned out I was wrong, and they were at the opposite end of the building, so we had to trek all the way back again. Zoe got all the way into the ladies bathroom and failed to make it onto the toilet in time, so that concluded our visit to Bunnings. We got home and she got changed, and I started making dinner. While dinner was cooking, Zoe watched a bit of TV, and then I decided to try making some lunch box pies, essentially a piece of buttered bread jammed into a muffin tray with some creamed corn and bacon in the bottom and beaten egg on top, thrown in the oven. Zoe helped sample the bacon and cracked the eggs for me, while I buttered bread and cut the crusts off and jammed slices into muffin trays. Bedtime was remarkably uneventful for a change.

4 March 2014

H ctor Or n Mart nez: Debian build system

There are many ways to build Debian distributions from source. None of them are trivial enough for beginners and all of them require complex setups. For instance, the Debian official setup uses the following software components: Around the core components, there are other software components needed to run the Debian distribution: Thanks to Debian ftp-master and buildd team, all software is built for several architectures and several ports. Most Debian infrastructure is managed and maintained by Debian System Administration team.

Simplified Debian build infrastructure

Debian Wiki has been growing different random pages trying to ease the setup and configuration problems concerning to Debian build system infrastructure. The above infrastructure barely documents what it is involved to create and generate Debian unstable ( sid ) distribution suite. In order to produce Debian stable distribution suite, there are software transitions to happen and yet another Debian team gets involved in the process, our beloved Debian release team. Once distribution reaches its maturity level and gets released as in its stable version, it gets updates also lead by release team and security related updates, which yet another team is responsible for them, the Debian security team. and there are lots of Debian teams doing many other things you might enjoy, have a byte

Mario Lang: Personal mail server

I used to use Fetchmail and Gnus with its mail splitting capabilites and nnml backend to handle my private mail setup for many years. This worked pretty fine since I am used to accessing my home server through SSH. I have Gnus running inside of a GNU screen session. So I can check mail from remote by using a SSH terminal. This works fine as long as I have a good SSH terminal on a desktop or laptop computer. However, it does not work very well on tablets or smaller mobile devices. Additionally, mail splitting has become a performance burden over the years. I do not want to wait for Gnus to sort incoming mail into different folders while checking for new mail. That is something which should have been done in the background already, and thats what we are going to cover with the setup described below. So I had to change my simple setup to accomodate for the new trends in mobile computing. The obvious core of such a setup is an IMAP server which receives and stores your mail such that different clients can access it. So the time of my Gnus nnml storage are definitely over. Mail is no longer stored in and by my mail client.
Dovecot While there are several IMAP server solutions out there, I find Dovecot fits my needs quite nicely. I've decided to store my mail in Maildir format in ~/Maildir. I prefer storing data like that in the home directory to avoid having to backup separate files from /var/. Maildir also features some index files which should help performance in the long run. Incoming mail will solely be delivered by fetchmail and should be checked for spam. While I can probably configure Exim to run SpamAssassin on mails before delivering them to Dovecot, there is a much more elegant solution: the dovecot local delivery agent (LDA). /usr/lib/dovecot/deliver takes mail from standard input and performs Sieve filtering and updates the mail indexes. We will call this executable more or less directly from fetchmail. Incoming mail from mailing lists will be sorted into different folders using Sieve. Dovecot needs to be told to enable the sieve plugin and to create new folders on demand. /etc/dovecot/local.conf:
disable_plaintext_auth = yes
mail_location = maildir:~/Maildir
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
protocol lda  
  mail_plugins = sieve
 
Sieve scripts are actually quite intuitive once you have a template to start from. ~/.dovecot.sieve:
require "fileinto";
if exists "X-Spam-Flag"  
  # Store spam tagged by SpamAssassin into dedicated Spam folder
  if header :contains "X-Spam-Flag" "YES"  
    fileinto "Spam";
   
  elsif exists "X-Cron-Env"  
  # Store mails from Cron daemon in dedicated folder
  fileinto "cron";
  elsif exists "List-Id"  
  # File list-mail into dedicated folders, matching on List-Id
  if header :contains "List-Id" "boost-users.lists.boost.org"  
    fileinto "boost-users";
    elsif header :contains "List-Id" "brltty.mielke.cc"  
    fileinto "brltty";
    elsif header :contains "List-Id" "debian-accessibility.lists.debian.org"  
    fileinto "debian-accessibility";
    elsif header :contains "List-Id" "debian-devel-announce.lists.debian.org"  
    fileinto "debian-devel-announce";
    elsif header :contains "List-Id" "debian-devel.lists.debian.org"  
    fileinto "debian-devel";
    elsif header :contains "List-Id" "spirit-general.lists.sourceforge.net"  
    fileinto "spirit-general";
   
  # ...
 
SpamAssassin Since I want automatic classification of spam messages, I use SpamAssassin. Just install spamassasin and enable spamd in /etc/default/spamassassin:
ENABLE=1
We will use spamc in the Fetchmail configuration.
Fetchmail My ~/.fetchmailrc is a straightforward list of some mailboxes to fetch mail from. I use the mda directive to skip the MTA and send mail through SpamAssassin and deliver it to Dovecot via its LDA mechanism. ~/.fetchmailrc:
set daemon 1200 # Poll at 10 minute intervals
poll blind.guru protocol IMAP: ssl;
# ... add more sources here ...
mda "/usr/bin/spamc -u %T -e /usr/lib/dovecot/deliver -d %T"
To avoid spreading access information in too many configuration files I am using the ability of Fetchmail to use netrc to retrieve account passwords. ~/.netrc:
machine blind.guru login mlang password <hidden>
Gnus I am using Gnus to read mail, newsgroups and RSS feeds since many years now. It would be quite a mouthful to explain all the customizations I am using by now. But there is one very important bit in the context of this article: How to access the IMAP server? In my setup, Emacs and therefore Gnus is running on the same machine as the IMAP server. So I can avoid authentication at all. This configuration will avoid unnecessary password prompts or caching. In ~/.emacs or ~/.gnus:
(setq gnus-secondary-select-methods '((nnimap "localhost"
                                       (nnimap-stream shell)))
      nnimap-shell-program "/usr/lib/dovecot/imap")
With this you should be able to subscribe to your IMAP folders from within Gnus with ease. Sorting incoming mails into folders is now performed by the IMAP server through Sieve scripts. Instead of changing Gnus' configuration I now edit ~/.dovecot.sieve when I subscribe to a new mailing list. If you add a new Sieve rule for a mailing list and the associated folder does not exist yet, Dovecot will autocreate it, very convenient.
Mobile devices Now all that is left is a way for your mobile devices to read and eventually send mail. This is very much dependant on your network setup, so I am not going to go into any detail here. If you are accessing your mail setup from a tablet in your local network you might get away without tinkering with your router configuration. If you want to read/send mail on the go you need some way to get to your external IP. Either it is stable enough or you need some dynamic DNS service. You will definitely want to forward IMAP and maybe SMTP ports from your router to your home server. If you don't have an existing SMTP server for your mobile device that accepts your outgoing mails you can also set one up yourself and deliver outgoing mails from your mobile device to the world with Exim or qmail. I am personally using Exim since I am going with the default MTA for Debian. Configuring Exim to take mail from iOS devices was as simple as enabling an appropriate authentication method and adding an account to /etc/exim4/passwd. I have to admit though that I don't particularily like Exim's configuration files. That is why I ended up using dovecot's LDA in the first place.

Daniel Pocock: Ganglia-Nagios-Bridge and JMXetric new releases

ganglia-nagios-bridge v1.1.0 A new release of ganglia-nagios-bridge is now available. The package has also been uploaded to Debian for those who use that and it will appear in backports soon. The main improvement in this version is slightly better support for generating an UNKNOWN alert in Nagios when fresh metrics fail to appear in Ganglia. There is also a new wiki comparing the various Ganglia and Nagios integration solutions. JMXetric v1.0.6 and gmetric4j v1.0.6 The latest JMXetric and gmetric4j releases improve support for JBoss, specifically, working around some issues that occur when a profiling agent accesses the logger or MBean platform during JVM startup. The page about JBoss, Wildfly and Tomcat integration details for Ganglia/JMXetric performance monitoring has also been updated to cover these improvements. The process of having JMXetric and gmetric4j packaged is now under way and it has been verified that they just work with the tomcat7 packages on Debian and Ubuntu (config sample included).

NOKUBI Takatsugu: Add grammar.xml entries in LanguageTool

At last weekend, I attended to Kanto LibreOffice Hackathon (Japanese). I could made two achieviements. Any event is a good trigger for me.

Andrew Pollock: [life] Day 35, Kindergarten, taxes, makeup BJJ

I foolishly had a late night last night, for no particularly good reason, so I slept in quite late this morning. I didn't start getting going until about 8am. How decadent. I'd forgotten to send Zoe to Kindergarten with her water bottle yesterday, and forgotten to give it to Sarah when she picked her up yesterday, so I dropped it and a fresh set of spare clothes out to Kindergarten after I had breakfast and a shower. I managed to sneak in and out without Zoe seeing me. I did some more work on my taxes before popping out for lunch to discuss my main start up idea. I also renamed my old shelf company today. I biked over to Kindergarten for pick up, and sure enough, Zoe was fast asleep. It took quite a bit of work to rouse her. I was pretty sure she was just faking it though. She moved pretty fast when I said I had a surprise at home for her. We biked home, and I gave her the surprise (some Pop Rocks I'd been saving for a special occasion) and headed out by car to a makeup Brazilian Jiu-Jitsu class to make up for the one she missed on Friday. It turned out that trying to do an afternoon class after Kindergarten and being woken up from a nap wasn't the best idea. Zoe tired out pretty quickly, during the warm up exercises, and was moderately uncooperative for the remainder of the class (she kept going on strike). I've never driven to or from BJJ classes until today, and I wasn't actually sure how best to drive home, since we've always taken the Norman Park Greenway. Zoe was asking for a "different way" anyway, so that worked out well. We winged it through Coorparoo and the back of Norman Park and tacked onto the end of how we'd normally cycle home. Zoe had enough time to watch a little bit of TV before Sarah arrived to pick her up.

3 March 2014

Richard Hartmann: Facial recognition

Dear crazyweb, while there seem to be good GUI tools to enable facial recognition with FLOSS, they all fall short of my requirements. And while there seem to be a lot of research projects with open code, they seem to be lacking in the "usable in real life" department. It seems as if there should be something to scratch that itch, but I couldn't find it... Thus, my wishlist for facial recognition software I will gladly follow up with a workflow blog post assuming I end up with useful feedback.

Gunnar Wolf: Pushing keyring updates. Let us bury your old 1024D key!

I have just pushed our pseudo-monthly batch of keyring updates to Debian. I am happy to inform you that, while the situation described in Clint Adams' interesting assessment of the state of the Debian keyring (and the quite constructive conversation that followed) still holds, and we still have way too many weak (1024D) keys in the Debian keyring, we got a noticeable effect as a result of said thread: 20 key upgrade requests in somewhat over a one week period! (mostly from DDs, with two from DMs IIRC). So, for any DD or DM reading this and not following the debian-project list where this thread took place: As keyring maintainers, we no longer consider 1024D keys to be trustable. We are not yet mass-removing them, because we don't want to hamper the project's work, but we definitively will start being more aggressively deprecating their use. 1024D keys should be seen as brute-force vulnerable nowadays. Please do migrate away from them into stronger keys (4096R recommended) as soon as possible. If you have a key with not-so-many active DD signatures (with not-so-many 2) waiting to get it more signed, stop waiting and request the key replacement. If you do not yet have a 4096R key, create a new one as soon as possible and get some signatures on it. Once 2 DDs have signed it, please request us to replace your old key. If you cannot get to meet two DDs in person, please talk to us and we will find out what to do.

DebConf team: DebConf11 Sponsored registration date has been extended (Posted by nattie)

The deadline for sponsored registration for DebConf11 in Banja Luka, originally May 8th, has officially been extended to May 19th. The reasons for the extension are as follows: Thanks to a significant increase in sponsorship from one of our main sponsors, we have been permitted and encouraged to welcome more people to the conference. The extension is a one-off occurrence in response to this new development, and to allow any potential attendees time to access more information before registering. Registration is required for everybody who plans to attend any part of Debconf. Access to the conference venue, food and accommodation is controlled through the registration badge. The only exception to the registration requirement is for those who plan to only attend Debian Day. It is possible to request sponsorship for food, accommodation, and travel. Those who have been involved or contributed to Debian or to other open source projects are eligible to apply for sponsorship, which will be decided on a case-by-case basis. The professional and corporate categories are for those who are sent and sponsored by their companies. Professional registration covers the actual costs of conference attendance, while corporate registration is just over double that price, and is aimed at those who, besides paying the actual costs, also want to contribute money and help Debian. the DebConf team

Next.

Previous.