Search Results: "ed"

13 March 2014

Mario Lang: Living in RAM

I switched to SSD on my home server a few years ago, at a time when SSH was still new and sort of hyped. However, I immediately knew that this type of technology is for me: it is fast and rather silent. SSD storage has this air of being fragile though. You are supposed to not wear it out too much. It's not like in the good old days where you would write to your hard disk without any extra thought.

tmpfs Before I switched to SSD I was already using tmpfs to store /tmp in RAM, for performance reasons mostly. /etc/default/tmpfs:

RAMTMP=yes

Since RAM is relatively cheap these days I have lots of it in my home server.

/var/cache/apt/archives When I switched to SSD, I extended this concept by mounting /var/cache/apt/archives via tmpfs. This is to save disk I/O for package updates. And for a machine running sid which is more or less regularily updated, this saves quite some useless writes to disk. Roughly 100MB per update.

/tmp/src Nowadays, I have almost all of my source code checkouts and build-trees in /tmp/src. myrepos makes this very easy to handle. Simply clone all repositories you are interested in, and remember their details with mr register. After a reboot, all you need to do is:

$ mkdir /tmp/src
$ cd !$
$ mr up

Your repository locations have (likely) been recorded in ~/.mrconfig. Of course there is the danger of accidentally loosing data due to a power outage. However, I feel this leads to a rather clean workflow. Changes need to be pushed anyway. So since you can not trust your local repository to really persist, you are forced to push your changes regularily, which is a good thing! Besides, I have a UPS at home which gives me roughly one hour of backup power. This was enough to sustain every power outage I have witnessed in the past years. Typically, outages are rather short lived in my area, somewhere between a few seconds and 5 minutes. It is rather rare that electricity goes missing for more than half an hour. The only downside of this approach is that from time to time, you will use more bandwidth by re-cloning repositories after a reboot. But really, who needs reboots on Linux? Last time I had half a year uptime, and only rebooted because I wanted to jump from linux 3.8 to 3.13. This approach only works on long-running machines though. It is probably not very useful for a laptop.

12 March 2014

Matthew Garrett: Dealing with Apple ACPI issues

I wrote about Thunderbolt on Apple hardware a while ago. Since then Andreas Noever has somehow managed to write a working Thunderbolt stack, which awesome! But there was still the problem I mentioned of the device not appearing unless you passed acpi_osi="Darwin" on the kernel command line, and a further problem that if you suspended and then resumed it vanished again.

The ACPI _OSI interface is a mechanism for the firmware to determine the OS that the system is running. It turns out that this works fine for operating systems that export fairly static interfaces (Windows, which adds a new _OSI per release) and poorly for operating systems that don't even guarantee any kind of interface stability in security releases (Linux, which claimed to be "Linux" regardless of version until we turned that off). OS X claims to be Darwin and nothing else. As I mentioned before, claiming to be Darwin in addition to Windows was enough to get the Thunderbolt hardware to stay alive after boot, but it wasn't enough to get it powered up again after suspend.

It turns out that there's two sections of ACPI code where this Mac checks _OSI. The first is something like:

if (_OSI("Darwin")) Store 0x2710 OSYS; else if(_OSI("Windows 2009") Store 0x7D9 OSYS; else

ie, if the OS claims to be Darwin, all other strings are ignored. This is called from \_SB._INI(), which is the first ACPI method the kernel executes. The check for whether to power down the Thunderbolt controller occurs after this and then works correctly.

The second version is less helpful. It's more like:

if (_OSI("Darwin")) Store 0x2710 OSYS; if (_OSI("Windows 2009")) Store 0x7D9 OSYS; if

ie, if the OS claims to be both Darwin and Windows 2009 (which Linux will if you pass acpi_osi="Darwin"), the OSYS variable gets set to the Windows 2009 value. This version gets called during PCI initialisation, and once it's run all the other Thunderbolt ACPI calls stop doing anything and the controller gets powered down after suspend/resume. That can be fixed easily enough by special casing Darwin. If the platform requests Darwin before anything else, we'll just stop claiming to be Windows.

Phew. Working Thunderbolt! (Well, almost - _OSC fails and so we disable PCIe hotplug, but that's easy to work around). But boo, no working battery. Apple do something very strange with their ACPI battery interface. If you're running anything that doesn't claim to be Darwin, Apple expose an ACPI Control Method battery. Control Method interfaces abstract the complexity away into either ACPI bytecode or system management traps - the OS simply calls an ACPI method, magic happens and it gets an answer back. If you claim to be Darwin, Apple remove that interface and instead expose the raw ACPI Smart Battery System interface. This provides an i2c bus over which the OS must then speak the Smart Battery System protocol, allowing it to directly communicate with the battery.

Linux has support for this, but it seems that this wasn't working so well and hadn't been for years. Loading the driver resulted in modprobe hanging until a timeout occurred, and most accesses to the battery would (a) take forever and (b) probably fail. It also had the nasty habit of breaking suspend and resume, which was unfortunate since getting Thunderbolt working over suspend and resume was the whole point of this exercise.

So. I modified the sbs driver to dump every command it sent over the i2c bus and every response it got. Pretty quickly I found that the failing operation was a write - specifically, a write used to select which battery should be connected to the bus. Interestingly, Apple implemented their Control Method interface by just using ACPI bytecode to speak the SBS protocol. Looking at the code in question showed that they never issued any writes, and the battery worked fine anyway. So why were we writing? SBS provides a command to tell you the current state of the battery subsystem, including which battery (out of a maximum of 4) is currently selected. Unsurprisingly, calling this showed that the battery we wanted to talk to was already selected. We then asked the SBS manager to select it anyway, and the manager promptly fell off the bus and stopped talking to us. In keeping with the maxim of "If hardware complains when we do something, and if we don't really need to do that, don't do that", this makes it work.

Working Thunderbolt and working battery. We're even getting close to getting switchable GPU support working reasonably, which is probably just going to involve rewriting the entirety of fbcon or something similarly amusing.

comments

Francesca Ciceri: All hail the Spam Reviewers!

With the help of Enrico and the Debian Listmaster Team -and in particular Alexander Wirt: thank you very much! - the spam reviewers of the Debian mailing lists have been added to the list of Debian Contributors.
While some statistics about the reviewing job already existed, adding these data to the Debian Contributors list is another little step to map all kind of Debian contributions. Wondering what exactly is the job of the spam reviewers? It consists in checking all the messages of the Debian mailing lists reported as spam: if a message reported as spam gets three reviews as spam (and none as ham) it is removed from the archive.
Only DDs can do this: if you are one check here how to do it.
But there's one - and probably more important job - that everyone can do: report spam. When a message is reported as spam by 5 or more persons, it will make to the reviewers' queue. There are some organized spam cleaning efforts you can join: here there's a list of them, while on this page you can find more information on the whole process. Wondering what exactly is the Debian Contributors list? It's a brilliant effort started by Enrico to map all kind of contributions in Debian.
You can read more about it here and here.

Vasudev Kamath: Working around type system of Go with unsafe

Go language has a strong type system unlike C, and some times this will be head ache when we want to interact with C data types with Cgo or just to convert a Go type to lets say byte slice. I recently faced the same problem and after poking around things, I learned the Go provides unsafe package which can be used to work around the Go's type system.

Problem Recently I started using Cgo to use some C library I had to write some tools for development and testing. The reason I chose Go for this was prototyping and writing some quick tools is much easier in Go than done in C. The C library had some function which takes pointer to array types and fills it with some values. The problem I was facing here was how to create a C array in Go. Go does have array but its largely used as internal representation for much efficient type called slice and I can't directly cast a byte slice into an C array. Second problem I had was I had to store arbitrary Go types like float (float32,float64) and int (int32, int64) etc. into C array. So in brief, the problems that needed to solve are

Find a way to convert byte slice from Go into C array and vice versa.
Find a way to convert and store Go types into a C array.

Solution Basically C array are sequence of memory location which can be statically or dynamically allocated. In Cgo its possible to access C standard library functions for memory allocation, so why not use it. The memory allocation function then returns the pointer to starting of allocated memory, we can use this pointer to write Go's bytes into the memory location and bytes from memory location into Go's slice. The pointer returned by C allocation functions are not directly usable for memory dereferencing in Go, here is where unsafe package kicks in. We will cast the return of C allocation function as unsafe.Pointer type and from the documentation of unsafe package,

A pointer value of any type can be converted to a Pointer.

A Pointer can be converted to a pointer value of any type.

A uintptr can be converted to a Pointer.

A Pointer can be converted to a uintptr.

So we can then cast unsafe.Pointer to uintptr which is the Go type which is large enough to hold any memory address in Go and can be used for pointer arithmetic just like we do in C (of course with some more castings). Below I'm pasting a simplified code in C which I wrote for this post.

#ifndef __BYTETEST_H__
#define __BYTETEST_H__
typedef unsigned char UBYTE;
extern void ArrayReadFunc(UBYTE *arrayout);
extern void ArrayWriteFunc(UBYTE *arrayin);
#endif

#include "bytetest.h"
#include <stdio.h>
#include <string.h>
void ArrayReadFunc(UBYTE *arrayout)
 
     UBYTE array[20] =  1, 2, 3, 4,5, 6, 7, 8, 9, 10,
                        11, 12, 13, 14, 15, 16, 17,
                        18, 19, 20 ;
     memcpy(arrayout, array, 20);
 
void ArrayWriteFunc(UBYTE *arrayin)
 
     UBYTE array[20];
     memcpy(array, arrayin, 20);
     printf("Byte slice array received from Go:\n");
     for(int i = 0; i < 20; i ++) 
             printf("%d ", array[i]);
      
     printf("\n");

Functions are written just for this post and they don't really do anything. As you can see ArrayReadFunc takes a pointer to array and fills it with content of another array using memcpy. Function ArrayWriteFunc on other hand takes pointer to array and copies its content to internal array. I've added print logic to ArrayWriteFunc just to show that values passed from Go are making it here. Below is the Go code which uses the above C files passes byte slice to get value out of C code and array made of byte slice to C function to send values in.

package main
/*
#cgo CFLAGS: -std=c99
#include "bytetest.h"
#include <stdlib.h>
*/
import "C"
import (
     "fmt"
     "unsafe"
)
func ReadArray() unsafe.Pointer  
       var outArray = unsafe.Pointer (C.calloc(20,1))
       C.ArrayReadFunc((*C.UBYTE)(outArray))
       return outArray
 
func WriteArray(inArray unsafe.Pointer)  
       C.ArrayWriteFunc((*C.UBYTE)(inArray))
 
func CArrayToByteSlice(array unsafe.Pointer, size int) []byte  
       var arrayptr = uintptr(array)
       var byteSlice = make([]byte, size)
       for i := 0; i < len(byteSlice); i ++  
               byteSlice[i] = byte(*(*C.UBYTE)(unsafe.Pointer(arrayptr)))
               arrayptr ++
        
       return byteSlice
 
func ByteSliceToCArray (byteSlice []byte) unsafe.Pointer  
       var array = unsafe.Pointer(C.calloc(C.size_t(len(byteSlice)), 1))
       var arrayptr = uintptr(array)
       for i := 0; i < len(byteSlice); i ++  
              *(*C.UBYTE)(unsafe.Pointer(arrayptr)) = C.UBYTE(byteSlice[i])
              arrayptr ++
        
       return array
 
func main() 
        carray := ReadArray()
        defer C.free(carray)
        carraybytes := CArrayToByteSlice(carray, 20)
        fmt.Println("C array converted to byte slice:")
        for i := 0; i < len(carraybytes); i ++  
                fmt.Printf("%d ", carraybytes[i])
         
        fmt.Println()
        gobytes := []byte 21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
                31, 32, 33, 34, 35, 36, 37, 38, 39, 40 
        gobytesarray := ByteSliceToCArray(gobytes)
        defer C.free(gobytesarray)
        WriteArray(gobytesarray)

Functions ReadArray and WriteArray are just wrapper to the calls to C counter parts ArrayReadFunc and ArrayWriteFunc. ReadArray returns unsafe.Pointer which is allocated C array and should be freed by caller. WriteArray takes unsafe.Pointer which is pointing to memory location containing C array. Now the functions of interest are CArrayToByteSlice and ByteSliceToCArray. It should be pretty clear from the above code to understand what is happening in these functions. Still I will just put explain them briefly. ByteSliceToCArray allocates a C array using calloc from C standard library. It then creates a uintptr, a pointer type in Go which is used to dereference the each memory location and store bytes from the input byte slice in them. CArrayToByteSlice on other hands creates a uintptr type by casting input unsafe.Pointer type and then uses this pointer type to dereference values from memoy and store it in byte slice with suitable casting. So lets build the code and run it and see the output:

C array converted to byte slice:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Byte slice array received from Go:
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

So yes it actually works and values are moving across C and Go. This solves first problem in hand next is converting arbitrary Go types into byte slices. There are many cases where we would like to convert an arbitrary Go types like (int, float) into bytes. One such use case I found was when writing a TCP client for communicating with a Server written using C speaking custom protocol. Here I'm just going to show how to convert types like float, int to byte slice, I've not tried converting structures but it is certainly possible. Below is the function which can convert int32,float32 into byte slice it can also be extended for other types.

func CopyValueToByte(value interface ) []byte  
     var valptr uintptr
     var slice []byte
     switch t := value.(type)  
     case int32:
             i := value.(int32)
             valptr = uintptr(unsafe.Pointer(&i))
             slice = make([]byte, unsafe.Sizeof(i))
     case float32:
             f := value.(float32)
             valptr = uintptr(unsafe.Pointer(&f))
             slice = make([]byte, unsafe.Sizeof(f))
     default:
             fmt.Fprintf(os.Stderr,"Unsupported type: %T\n", t)
             os.Exit(1)
      
     for i := 0; i < len(slice); i++  
             slice[i] = byte(*(*byte)(unsafe.Pointer(valptr)))
             valptr++
      
     return slice

This function is generic which can take various types of value. First we will use Go's type assertion to determine the type and creates a uintptr pointer for the value and allocates byte slice depending on the size of the value as calculated using unsafe.Sizeof. Later it uses the pointer to dereference value from memory location and copies each byte into byte slice. The idea used here is every type is represented as certain number of bytes in the memory. Below is the entire program.

package main
import (
     "fmt"
     "unsafe"
     "os"
)
func CopyValueToByte(value interface ) []byte  
     var valptr uintptr
     var slice []byte
     switch t := value.(type)  
     case int32:
             i := value.(int32)
             valptr = uintptr(unsafe.Pointer(&i))
             slice = make([]byte, unsafe.Sizeof(i))
     case float32:
             f := value.(float32)
             valptr = uintptr(unsafe.Pointer(&f))
             slice = make([]byte, unsafe.Sizeof(f))
     default:
             fmt.Fprintf(os.Stderr,"Unsupported type: %T\n", t)
             os.Exit(1)
      
     for i := 0; i < len(slice); i++  
             slice[i] = byte(*(*byte)(unsafe.Pointer(valptr)))
             valptr++
      
     return slice
 
func main()  
     a := float32(-10.3)
     floatbytes := CopyValueToByte(a)
     fmt.Println("Float value as byte slice:")
     for i := 0; i < len(floatbytes); i++  
             fmt.Printf("%x ", floatbytes[i])
      
     fmt.Println()
     b := new(float32)
     bptr := uintptr(unsafe.Pointer(b))
     for i := 0; i < len(floatbytes); i++  
             *(*byte)(unsafe.Pointer(bptr)) = floatbytes[i]
             bptr++
      
     fmt.Printf("Byte value copied to float var: %f\n", *b)

The above conversion can also be achieved using encoding/binary package provided by Go. But its been told to me that it makes things pretty slow.

Conclusion So goes unsafe.Pointer is really powerful thing which allows us to work around the Go's type system but as package documentation says it should be used with care

PS: I'm not really sure if its recommended to use allocation functions from C standard library, I will wait for expert gophers to comment on that.

John Goerzen: Agile Is Dead (Long Live Agility)

In an intriguing post, PragDave laments how empty the word agile has become. To paraphrase, I might say he s put words to a nagging feeling I ve had: that there are entire books about agile, conferences about agile, hallway conversations I ve heard about whether somebody is doing this-or-that agile practice correctly. Which, when it comes down to it, means that they re not being agile. If process and tools, even if they re labeled as agile processes and tools, are king, then we ve simply replaced one productivity-impairing dictator with another. And he makes this bold statement:

Here is how to do something in an agile fashion: What to do:

Find out where you are

Take a small step towards your goal

Adjust your understanding based on what you learned

Repeat

How to do it: When faced with two or more alternatives that deliver roughly the same value, take the path that makes future change easier. Those four lines and one practice encompass everything there is to know about effective software development.

He goes on to dive into that a bit, of course, but I think this man has a rare gift of expressing something complicated so succinctly. I am inclined to believe he is right.

Neil McGovern: Entering the fray 2014 DPL election

After a number of years in Debian, in various roles, I decided to put my name forward for election to Debian Project Leader. My platform has now been published, and campaigning is now under way!

Featured image is CC-BY by State Library of Victoria Collections

Petter Reinholdtsen: How to add extra storage servers in Debian Edu / Skolelinux

On larger sites, it is useful to use a dedicated storage server for storing user home directories and data. The design for handling this in Debian Edu / Skolelinux, is to update the automount rules in LDAP and let the automount daemon on the clients take care of the rest. I was reminded about the need to document this better when one of the customers of Skolelinux Drift AS, where I am on the board of directors, asked about how to do this. The steps to get this working are the following:

Add new storage server in DNS. I use nas-server.intern as the example host here.
Add automoun LDAP information about this server in LDAP, to allow all clients to automatically mount it on reqeust.
Add the relevant entries in tjener.intern:/etc/fstab, because tjener.intern do not use automount to avoid mounting loops.

DNS entries are added in GOsa , and not described here. Follow the instructions in the manual (Machine Management with GOsa in section Getting started). Ensure that the NFS export points on the server are exported to the relevant subnets or machines:

root@tjener:~# showmount -e nas-server
Export list for nas-server:
/storage         10.0.0.0/8
root@tjener:~#

Here everything on the backbone network is granted access to the /storage export. With NFSv3 it is slightly better to limit it to netgroup membership or single IP addresses to have some limits on the NFS access. The next step is to update LDAP. This can not be done using GOsa , because it lack a module for automount. Instead, use ldapvi and add the required LDAP objects using an editor.

ldapvi --ldap-conf -ZD '(cn=admin)' -b ou=automount,dc=skole,dc=skolelinux,dc=no

When the editor show up, add the following LDAP objects at the bottom of the document. The "/&" part in the last LDAP object is a wild card matching everything the nas-server exports, removing the need to list individual mount points in LDAP.

add cn=nas-server,ou=auto.skole,ou=automount,dc=skole,dc=skolelinux,dc=no
objectClass: automount
cn: nas-server
automountInformation: -fstype=autofs --timeout=60 ldap:ou=auto.nas-server,ou=automount,dc=skole,dc=skolelinux,dc=no
add ou=auto.nas-server,ou=automount,dc=skole,dc=skolelinux,dc=no
objectClass: top
objectClass: automountMap
ou: auto.nas-server
add cn=/,ou=auto.nas-server,ou=automount,dc=skole,dc=skolelinux,dc=no
objectClass: automount
cn: /
automountInformation: -fstype=nfs,tcp,rsize=32768,wsize=32768,rw,intr,hard,nodev,nosuid,noatime nas-server.intern:/&

The last step to remember is to mount the relevant mount points in tjener.intern by adding them to /etc/fstab, creating the mount directories using mkdir and running "mount -a" to mount them. When this is done, your users should be able to access the files on the storage server directly by just visiting the /tjener/nas-server/storage/ directory using any application on any workstation, LTSP client or LTSP server.

Andrew Pollock: [life] Day 43, a lovely lazy day, until we got on the bike

Zoe slept through the night again last night. Two nights of uninterrupted sleep has been totally amazing! We had a slower start to the day because we didn't have to be anywhere. I've been meaning to do a massive cull of Zoe's wardrobe for a while, and this morning seemed like as good an opportunity as any, so after TV, I managed to get Zoe to cooperate with trying on clothes for about an hour. She really enjoyed pulling everything out of her wardrobe and putting it all back again neatly. After that, she asked me if I could make her a bus out of the last remaining moving box, so I managed to make something approximating a bus. The port-a-robe boxes are really rather handy for this sort of thing. The balcony is starting to look like a cardboard box shanty town, with a cubby house, a rocket and now a bus. I think we need to host a few more play dates so they all get a good work out. It was time for lunch after that. I'd had a bucket of caramel popcorn, which I used for the journey to Coochiemudlo the other week. Zoe had asked me how popcorn was made, so I had resolved to make popcorn with her sometime soon. Yesterday at the health food shop, I opportunistically noticed some popping corn, so I grabbed a small bag of it. It turned out that buying peanut oil was good for more than just making peanut butter, as it came in handy today for popping corn. I've never popped corn before, aside from occasionally getting a bag of the microwave stuff, so it was a fun experience to do it "from scratch". It turned out pretty well, with only a few burned ones and a few unpopped ones. It'd be fun to try and dry out a corn cob and totally do it from scratch. After that I tried to get Zoe to take a nap, but it didn't work out so well, so after her quiet time we cleaned up her bedroom and generally fooled around for a bit. Throughout the day, Zoe did lots of little bouts of self-directed play. She dredged up this little wooden xylophone that we'd made in the US from a Kiwi Crate and had a great time banging away on it until she stood on it and broke it. The doctor's kit got a good workout. It was just a really nice, relaxing day at home. Zoe's Kindergarten was having a play date meet-and-greet in the park, for all the parents to meet each other. It was a fair way from home, but there was a nice, albeit circuitous, bike path to get there, and I decided to try and bike there, without really properly estimating the distance or length of time it would take. The play date was due to start at 3:30pm, and I was planning on leaving at 3pm. We were about 5 minutes late getting out of the door. I think it ended up taking us about 50 minutes to get there. It was a beautiful ride though, once we got onto the Bulimba Creek bikeway, through the Minippi Parklands. It would have been the longest ride Zoe's had on the bike, and she wasn't fazed at all. The play date itself was nice. For some reason, I'm a bit shy about going and introducing myself to the mothers (I kept wanting to write "other mothers"), but I sucked it up and made a few introductions. I'm terrible about remembering names though. At least I've got a few faces I can say hi to at drop off and pick up now. Sarah met us at the park and hung out for a bit as well, and then Zoe and her left together and I biked back home. I was feeling pretty good, and thought I'd try and bang out a run when I got home, but I didn't end up having the fuel in the tank, and blew up after about 1.5km.

Dirk Eddelbuettel: QuantLib 1.4 packages also available for Ubuntu and Windoze

QuantLib release 1.4 came out a couple of days ago. Sticking with standard practice, I immediately updated the Debian packages I have been maintaining for over a dozen years now, and corresponding binaries are available for Debian flavors unstable ("sid") and testing ("jessie"). I also updated my RQuantLib package accordingly (though it needed no changes -- yay to stable APIs). However, I also run Ubuntu on a few machines at work and home, and proceeded to create local packages for both 32-bit ("i386") and 64-bit ("amd64") variants of the current release 13.10 ("saucy"). As these may be of use to others, I am now making these available here from my box so that folks can run a current QuantLib version on those machines. Lastly, Brian Ripley was kind enough to also fire up his cross-compilation setup on his Linux box to create similar Windows binaries, These are e.g. used on CRAN and by the win-builder service to compile the RQuantLib CRAN Windows builds of RQuantLib. That corresponding archive file can be obtained as Quantlib14.zip see this short page for details. My thanks to Brian for helping with this. I hope this may be useful to the few people wishing to compile and/or extend RQuantLib and QuantLib.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

11 March 2014

John Sullivan: Drupal UI win

Wouter Verhelst: The apt-get install game

"Given a Debian system with just the set of essential packages installed, find a single package name that, together with its dependencies, will increase the used diskspace of the system by as much as possible." My entry: openarena. Off to bed, now.

Steve Kemp: Discovering back-end servers automatically?

Recently I've been pondering how to do service discovery. Pretend you have a load balancer which accepts traffic and routes incoming requests to different back-ends. The loadbalancer might be pound, varnish, haproxy, nginx, or similar. The back-ends might be node applications, apache, or similar. The typical configuration of the load-balancer will read:

# forward
# backends
backend web1    .host = "10.0.0.4";  
backend web2    .host = "10.0.0.6";  
backend web3    .host = "10.0.0.5";  
#  afterword

I've seen this same setup in many situations, and while it can easily be imagined that there might be "random HTTP servers" on your (V)LAN which shouldn't receive connections it seems like a pain to keep updating the backends. Using UDP/multicast broadcasts it is trivial to announce "Hey I'm a HTTP-server with the name 'foo'", and it seems to me that this should allow seamless HTTP load-balancing. To be more explicit - this is normal:

The load-balancer listens for HTTP requests, and forwards them to back-ends.
When back-ends go away they stop receiving traffic.

What I'd like to propose is another step:

When a new back-end advertises itself with the tag "foo" it should be automatically added and start to receive traffic.

i.e. This allows backends to be removed from service when they go offline but also to be added when they come online. Without the load-balancer needing its configuration to be updated. This means you'd not give a static list of back-ends to your load-balancer, instead you'd say "Route traffic to any service that adfvertises itself with the tag 'foo'.". VLANS, firewalls, multicast, udp, all come into play, but in theory this strikes me as being useful, obvious, and simple. (Failure cases? Well if the "announcer" dies then the backend won't get traffic routed to it. Just like if the backend were offline. And clearly if a backend is announced, but not receiving HTTP-requests it would be dropped as normal.) If I get the time this evening I'll sit down and look at some load-balancer source code to see if any are written in such a way that I could add this "broadcast discovery" as a plugin/minor change.

Chris Lamb: Two aesthetically pleasing Python snippets

The following Python tree structure recently resurfaced. I find it rare to see functional or recursive techniques expressed so succinctly in Python:

def tree():
    return collections.defaultdict(tree)

Another construction I am quite fond of is:

for _, dirnames, filenames in os.walk(path):
    break

...which leaves dirnames and filenames in the current scope, containing the immediate subdirectories and files of the specified path. Perhaps a little too cute, but appealing in its own way.

Steve Langasek: My CuBox-i has arrived

A couple of weeks ago, Gunnar Wolf mentioned on IRC that his CuBox-i4 had arrived. This resulted in various jealous noises from me; having heard about this device making the rounds at the Kernel Summit, I ordered one for myself back in December, as part of the long-delayed HDification of our home entertainment system and coinciding with the purchase of a new Samsung SmartTV. We've been running an Intel Coppermine Celeron for a decade as a MythTV frontend and encoder (hardware-assisted with a PVR-250), which is fine for SD video, but really doesn't cut it for anything HD. So after finally getting a TV that would showcase HD in all its glory, I figured it was time to upgrade from an S-Video-out, barely-limping-along tower machine to something more modern with HDMI out, eSATA, hardware video decoding, and whose biggest problem is it's so small that it threatens to get lost in the wiring! Since placing the order, I've been bemused to find that the SmartTV is so smart that it has had a dramatic impact on how we consume media; between that and our decision to not be a boiled frog in the face of DISH Network's annual price increase, the MythTV frontend has become a much less important part of our entertainment center, well before I ever got a chance to lay hands on the intended replacement hardware. But that's a topic for another day. Anyway, the CuBox-i4 finally arrived in the mail on Friday, so of course I immediately needed to start hacking on it! Like Gunnar, who wrote last week about his own experience getting a "proper" Debian install on the box, I'm not content with running a binary distribution image prepared by some third party; I expect my hardware toys to run official distro packages assembled using official distro tools and, if at all possible, distributed on official distro images for a minimum of hassle. Whereas Gunnar was willing to settle for using third-party binaries for the bootloader and kernel, however, I'm not inclined to do any such thing. And between my stint at Linaro a few years ago and the recent work on Ubuntu for phones, I do have a little knowledge of Linux on ARM (probably just enough to be dangerous), so I set to work trying to get the CuBox-i4 bootable with stock Debian unstable. Being such a cutting-edge piece of hardware, that does pose some challenges. Support for the i.MX6 chip is in the process of being upstreamed to U-Boot, but the support for the CuBox-i devices isn't there yet, nor is the support for SPL on i.MX6 (which allows booting the variants of the CuBox-i with a single U-Boot build, instead of requiring a different bootloader build for each flavor). The CuBox-i U-Boot that SolidRun makes available (with source at github) is based on U-Boot 2013.10-rc4, so more than a full release behind Debian unstable, and the patches there don't apply to U-Boot 2014.01 without a bit of effort. But if it's worth doing, it's worth doing right, so I've taken the time to rebase the CuBox-i patches on top of 2014.01, publishing the results of the rebase to my own github repository and submitting a bug to the Debian U-Boot maintainers requesting its inclusion. The next step is to get a Debian kernel that not only works, but fully supports the hardware out of the box (a 3.13 generic arm kernel will boot on the machine, but little things like ethernet and hdmi don't work yet). I've created a page in the Debian wiki for tracking the status of this work.

Andrew Pollock: [life] Day 42, a good night's sleep, Kindergarten, make up swim class and dinner with Nana

Zoe slept through the night for the first time in recent memory, and was up at around 6am. It was lovely. I'm thinking the slightly lower temperature helped. I'm definitely going to explore air conditioning for the bedrooms before next Summer. I'd picked up some egg cups on the weekend, so we did soft boiled eggs in the Thermomix for breakfast. They turned out lovely this time, and Zoe seemed to enjoy eating hers. She dressed herself while they were cooking. Her Kindergarten t-shirts arrived yesterday, so now she looks like all the other kids. It's very cute. It rained on and off in the morning, and I was in two minds about whether to chance it on the bike or not. As it happened we ended up messing around a bit before leaving, so we drove, and it was a good call, because it started raining again as we pulled out of the driveway. Drop off was super smooth. Megan had been away yesterday, but was back today, so Zoe was very happy to see her. My financial advisor came out for a chat about investments mid-morning, and then I got stuck into my US taxes. I will not miss having to do these tax returns. The 2013 one is going to be particularly complicated by the fact we sold our home and were only in the country for like 80 days. I biked over to Kindergarten to pick up Zoe, as the weather had improved. Surprisingly, she was awake, but her teacher said she'd just woken up from a nap. We managed to clear out pretty quickly. I stopped at the health food shop on the way home to get some peanut oil for the dinner I was making. Due to a miscommunication, Zoe had missed her Saturday swim class. I explained to the swim school the error, and they were kind enough to allow Zoe to have a make up class anyway, so we went to that after Kindergarten. Zoe wanted to do some more practice riding her bike. She's got a little 10 inch bike with no training wheels and a gyrobike front wheel. We haven't had a lot of success learning to ride to date, due to a combination of her height and just time, so it was interesting that she decided today she wanted to do some more practice. She wanted to "ride" her bike from home to the park on the way to swim class, but I managed to convince her to walk the bike to the park. It turns out even walking the bike is a bit of a challenge for her, so I ended up carrying it most of the way. It's heavy. We got to the park, and had a bit of a run with the bike. I'm glad I decided to take her knee pads and elbow pads, because she did stack it fairly decently, but managed to survive mostly unscathed. To her credit, she even got back on the bike again. She's a tough kid. Once we made it to the playground, she lost all interest in the bike, and we had about 15 minutes to kill before we needed to continue to the pool. We walked/carried the bike from the park to the pool, and the make up class Zoe ended up in was Megan's regular class, so the girls had a great time together. After swim class, we walked back home. Zoe's Nana was coming over for dinner, and I wanted to make a satay chicken dish. I'd bought some peanuts from the Peanut Wagon on Sunday, when I was in Nanango for my uncle's 90th birthday party, and I wanted to try making peanut butter from scratch in the Thermomix, and use that to make the satay sauce. Long story short, dinner wasn't super great. I'd made a satay chicken in the Thermomix previously, but I couldn't find the recipe I used last time. I used this recipe tonight, but didn't really follow it properly because I was trying to cut corners due to the time. I tried cooking wild rice in the microwave and it was a total flop. The satay chicken itself turned out okay, but it was a shame about the rice. It was cool making my own peanut butter. Zoe had a great time playing with her Nana, and after we dropped her off at the train station went to bed without a peep.

Ian Wienand: Tenvis IP391W meta-page

Recently I purchased a Tenvis IP391W-HD camera. I would be unlikely to recommend it. The price is certainly right and the picture quality is quite good. The Android and iPhone apps do work to watch the stream live. However, the interface is terrible and almost useless without Internet Explorer. There is a RTSP stream (rtsp://admin:password@ip) which VLC can seem to handle, but not mplayer. The recording format (.h264) is not viewable by VLC or mplayer and all I could find is a Windows .exe to convert them to an .avi. The motion detection gets troubled by the dark. It would really only be useful for something permanently well-lit. It did send me emails via gmail. I have got it recording to a NFS server, but I don't have a lot of confidence in the reliability of it. I think I have it configured to record in 3600-second blocks (given the interface, it's hard to tell if I've set it up to the network, or to the internal flash, etc), but it seems to intersperse 60 minute recordings with random small recordings. Given the whole idea of a security camera is to record the unexpected, you want a lot of confidence you're actually recording, which you don't get with this. You can see below it recorded 3 hour blocks, then started going a little crazy...

-rw-r--r-- 1 nobody nogroup  69M Mar 11 01:25 0-003035.v264
-rw-r--r-- 1 nobody nogroup  69M Mar 11 02:25 0-013049.v264
-rw-r--r-- 1 nobody nogroup  69M Mar 11 03:26 0-023103.v264
-rw-r--r-- 1 nobody nogroup 5.9M Mar 11 03:31 0-033117.v264
-rw-r--r-- 1 nobody nogroup 1.5M Mar 11 03:40 0-034350.v264
-rw-r--r-- 1 nobody nogroup  17M Mar 11 04:02 0-035259.v264
-rw-r--r-- 1 nobody nogroup 306K Mar 11 04:10 0-041548.v264
-rw-r--r-- 1 nobody nogroup 4.9M Mar 11 04:23 0-042457.v264

There is a support forum, where I found the following files scattered in various posts. From what I can tell, they are the latest as of this writing. I can confirm they work with my IP391W-HD, which the system tells me is GM8126 hardware and came with firmware 1.2.8.3.

1.3.3.3.pk2 - firmware (b56f211a569fb03a37d13b706c660dcb)

web.pk2 - a UI update that includes dropbox support. This is really for the model that has pan and tilt, so those buttons don't work. (0e42e42bd6f8034e87dcd443dcc3594d)

V264ToAVIen.exe - converts the output to an AVI file that mplayer will play (with some complaints) (9c5a858aa454fed4a0186cf244c0d234)

www.modern.ie offers free limited-time Windows VM's which will work to upload this firmware. Just make sure you use a bridged network in the VM; I'm guessing the firmware ActiveX control tells the camera to TFTP the data from it, which doesn't work via NAT. Somewhat worryingly, you can telnet to it and get a login prompt (TASTECH login). So it has a built-in backdoor you can't disable. There have been some efforts to hack the device. leecher@dose.0wnz.at did an excellent job reverse engineering the .pk2 format and writing tenvis_pack.c (no license, I'm generously assuming public domain). I used this to recreate the firmware above with a telnet daemon listening with a shell on port 2525 (no password, just telnet to it)

1.3.3.3-telnet.pk2 (19cf63ca8f0928af8ef5a9eb8326aae6)

It's interesting to poke around, but it seems like the whole thing is really driven by a binary called ipc8126

/ # ipc8126 --help
*** TAS-Tech IPCAM/DVS
*** Version: 1.3.3.3
*** Release date: 2013-08-05 15:48:32

In general, I'd say hackability is quite low. Warning : any of the above might turn your camera into a paperweight. It worked for me, but that's all I can say...

10 March 2014

Simon Kainz: DUCK for packagers

I finally managed to package duck 0.4 for Debian/unstable. Additionally to duck.debian.net you can now run all (in fact even more than on the website) checks on your own. Intended as as helper tool for packaging work, it inspects debian/control files as well as upstream metadata files. By trying to use the approriate tool for each URL, it tries to find out (eg. using git ls-remote for GIT repos) if a given repository is availabe. Email addresses in Maintainer and Uploaders fields as well as mailto: URLs are checked by trying to find MX, A or AAAA records for the email domain. Usage Without any additional options, duck looks for the following files:

debian/control
debian/upstream (if it's not a directory :-) )
debian/upstream-metadata.yaml
debian/upstream/metadata

Just hop into your extracted package source tree (the one containing the debian subdir), and run:

$ duck

If everything is ok, you won't get any output. To see what's going on, run:

$ duck -v
debian/control: Maintainer: Simon Kainz simon@familiekainz.at: OK
debian/control: Vcs-Git: git://anonscm.debian.org/collab-maint/duck.git: OK
debian/control: Vcs-Browser: http://anonscm.debian.org/gitweb/?p=collab-maint/duck.git: OK
debian/control: Homepage: http://duck.debian.net: OK

Errors show up like this:

$ duck -v
debian/control: Maintainer: Simon Kainz simon@domain.invalid: ERROR
Simon Kainz simon@domain.invalid: No MX entry found.
Simon Kainz simon@domain.invalid: No A entry found.
Simon Kainz simon@domain.invalid: No AAAA entry found.
debian/control: Vcs-Git: git://anonscm.insanetypo..debian.org/collab-maint/duck.git: ERROR
fatal: unable to connect to anonscm.insanetypo..debian.org:
anonscm.insanetypo..debian.org: Name or service not known
debian/control: Vcs-Browser: http://anonscm.debian.org/gitweb/?p=collab-maint/duck.git: OK
debian/control: Homepage: http://duck.debian.net: OK
skainz@zidpc9027:~/debian/sid/home/skainz/packages/duck_2014/duck$

Missing features Currently the following VCS URLs are not supported:

Vcs-Arch
Vcs-Cvs
Vcs-Mtn

If someone knows how to get the state of one of the repos based on the VCSs above, preferably without needing to check out the complete source code, please contact me. And remember: "Always duck before dput" :-)

Elena 'valhalla' Grandi: Baking swirls...

I've spent a good part of the weekend baking and taking pictures of cookies, because of reasons.

And now I'm forced to eat them, I can't let them go stale O:- )

(Actually, I plan to share at least some of them at @LIFO this night.)

Jaldhar Vyas: 7DRL 2014/Procedural Death Jam

Each year the 7DRL competion gives me an opportunity to abandon developing a roguelike game part way through. This year I am going to strive mightily to actually complete a playable game though owing to family commitments I have already gotten off to a rocky start which does not bode well. But enough negativity... So this year 7DRL coincides with the first ever Procedural Death Jam. This is a similar type of thing except the genre is Procedural Death Labyrinth. Procedural Death Labyrinths are a subset of Roguelikes (or is it the other way round?) If you have taken advantage of the wonderful offer Valve has given to Debian developers, you can find a number of examples of Procedural Death Labyrinths on Steam. The most important part of any software development project is of course choosing a cool name. My game is called "The Girl Who Played With The Dragons Nest" (Can you guess what I've been reading?) It is written in C++ with the Curses toolkit. It will of course be open source. I intend to blog my progress here if you care to follow along.

Andrew Pollock: [life] Day 41, Running, Kindergarten, baking

I got up not as early as I'd have liked, and did the first 10km run I've done for a while, with my new shoes and the orthotics in them. It was a bit cooler, and despite not being particularly well hydrated, I lasted the distance, and managed to complete it in under an hour. I'm a long way off my best time at the moment, but I'm hoping as it cools down my running fitness will improve. I ended up doing a bunch of startup-related stuff today, instead of Debian stuff. It felt good to make some vaguely forward progress, even if it was just business operations planning stuff. I picked up Zoe from Kindergarten by bike. We'd both confirmed with her several times since Friday as to which mode of transport she wanted to use to come home, and she'd indicated the bike. Zoe was asleep as usual, but woke up pretty easily. The Kindergarten has indicated that they want everyone out of the centre by 2:30pm, which makes it a bit of a challenge to rouse a deeply sleeping child and exit the centre in 10 minutes. I managed to sunscreen her while she was still horizontal, so that sped things up, and it helped greatly that she didn't have a post-nap meltdown. One of her class mates and her mother were hanging around waiting for her big brother to finish school, and the bike trailer tends to be a good conversation starter, so we chatted for a little bit before leaving. It was nice to properly meet another parent, and I'm glad to see Zoe's started learning some of her class mates names. I had a kale, carrot and apple juice for lunch today, and as I hate wasting the pulp out of the juicer, I'd previously found a nice muffin recipe that I could use the kale and carrot pulp in. I'd previously baked a batch of mini muffins for Zoe's lunches, and was pleased that she liked them. We were getting a bit low, so I thought we'd bake another batch this afternoon, so that's what we did after we got home. Once they were in the oven, Zoe went off to watch TV and I made a start on dinner. Bedtime went nice and smoothly tonight.

Next.

Previous.