Search Results: "ed"

9 July 2012

John Goerzen: Decisions on Listening to Music

A couple of days ago, I commented about my thoughts on finding a better way to listen to music, and asked for suggestions. I ve checked out the avenues suggested, and here are my thoughts so far. Streaming service: Spotify Of the streaming service offerings, Spotify seems the most compelling. It has both Linux and Android clients. The Linux client is unofficial and a bit buggy, but serviceable. The service is fairly nice, and the serendipity of listening to radio of tracks like any track or artist I may pick is an incredible feature. It will mean a different mindset; rather than carefully choosing what I buy, I have a vast catalog at my fingertips. I d have to give up some control, but might be better off for it. Spotify can play local files, but sadly lacks support for FLAC that most of my collection is in. A little work with the shell, and I had a quick multi-core FLAC to MP3 conversion done. It can also identify local files and match them up with its catalog, but it does a somewhat poor job of it, despite the fact that my files are tagged by Musicbrainz. There are also a fair number of albums in my collection (from Magnatune and from local artists) that aren t in Spotify s catalog and probably never will be. I may wind up using git-annex or something like it to sync them between PCs. Surprisingly, it doesn t have any kind of web-based player available. It has a lot of features, but many of them are under-documented. Subsonic / Supersonic The idea of being able to stream music anywhere certainly has appeal, and the idea of streaming that music from my own machine my own collection is quite the appealing one. Subsonic seems to be the standard that people recommend for this. It has a web-based player that works well, as well as mobile clients for various platforms. The Android mobile client is particularly well-regarded and is probably the best one here, aside from Spotify. The playlists work well, as expected, and generally it has the appearance of a well-rounded system. There is a lot of flexibility in bitrates; maximum bitrates can be set per-user and per-client. It, of course, transcodes on the fly. My chief gripe about Subsonic is that it has no true album/artist/genre browser. Its main browser is mostly a filesystem browser. I have used clients that show album/artist/genre panes or sortable lists, based on tags, for so long that my filesystem organization is rather different centered around bitrate of rip, origin, etc. It is not very easy to navigate my collection in that way. The search feature can search artist and album, and this can partially make up for it. But there is simply no way to see a list of all artists in the collection. The server is written in Java, and takes up 362MB at startup. That s, I guess, decent for a Java server, but doesn t make me pleased as I d be running it on lower-end hardware. Supersonic is a git-maintained actively synced subsonic fork which removes the license cost (the source is GPL3 anyhow so that wasn t a big deal). Ampache Often mentioned in the same breath as Subsonic, this is yet another frequent suggestion for streaming your own collection. Its primary player is web-based, like Subsonic. It has full artist/genre browsers, and adds tag cloud and various other options as well by default, it loaded the tag cloud with my genre tags, which was a nice touch. The search feature is a little less robust, as it simply searches anywhere and returns a list of all matching tracks, rather than listing matching albums and artists before going into the list of all tracks. But that is somewhat of a minor nit. Ampache s web-based player, liks Subsonic s, uses Flash, but the implementation is really quite awkward. You can t simply click a play button and have playback start. Rather, you must add things to the playlist (queue), a frame on the web interface. Then, you must click play in THAT frame, which launches a new window for the player and transfers its contents to it. You thus have two separate running playlists. And if you think you can just add to the playlist of the player window, think again; you can only replace it. It is simply not possible to add a track to the end of that list without losing your existing place. This makes the usefulness of the web-based player rather, well, low. Playlist functionality in Ampache is rather odd. You can have playlists, but if you want to add a track to a saved playlist, first you have to clear you current playlist, then add the track to it, then save it to the existing playlist. Frontends for Ampache exist in banshee and in amarok. Banshee s is broken, and simply silently fails to do anything. Unless you look in the console, in which case you ll see a .NET backtrace. Banshee has had these mysterious errors for me for years. The Amarok plugin is so basic that you can browse by artist but not by album, and is almost unusable. There is also a rhythmbox plugin in Debian, but it is apparently for an old version of Rhythmbox and no longer works. The best frontend is a dedicated Ampache frontend called Viridian. It has a nice interface, provides features such as taskbar integration, and generally works pretty well. The only thing it s bad at is playlists, and here it s even worse than the web interface. It doesn t support modifying the server-side playlists at all. And accessing one is an annoying load playlist operation, that replaces your current play queue (also called the playlist ) with its contents by default. Several Android clients exist for Ampache, though it seems only one has seen any updates since 2010. mpd This is an interesting program, as it is designed more to provide multiple points of control for a single player than it is to stream a single collection to multiple devices. Nevertheless, I tested its HTTP and icecast stream modules. (It can also stream with PulseAudio, but PulseAudio causes devious breakage on my workstation, so I refuse to bother.) This could be made to work reasonably well on my LAN using the gmpc client. There are several Android clients as well, though none of them integrate as well as Subsonic or Spotify. Most seem focused on controlling a remote player. The ones that do stream do so with some hiccups, and don t integrate controls with the lock screen. gmpc is a reasonable control program and, combined with mplayer, makes a decent player. It can browse the database every way I d want to. It doesn t let me right-click on tracks to add them to playlists, but I can use copy/paste to do so, or add them to the play queue and from there to a playlist. It s annoying, but not completely unworkable. But here s the real concern. mpd has no provision whatsoever for any sort of encryption such as SSL. If I want to be listening to my music collection from on the road, I don t really wish to open up a streaming music service for the entire Internet. Nor do I really want to mess with SSH tunnels and the like, which are cumbersome on Android (and can be in general). This is pretty much a fatal flaw to me. Google Music An interesting service, but ultimately one I don t see the point for in my situation where I already have my music on a PC that is always on. It would probably accommodate my entire collection free, but how many weeks would it take me to get it there? And how would it stay in sync? It simply doesn t offer me any compelling advantage over streaming direct from my PC, so I don t see much need to bother with it. Others I briefly considered Streeme (poor mobile support), Audiogalaxy (no Linux support), Jinzora (appears dead), and Sockso (no mobile support). I also briefly considered other streaming services such as Pandora, Slacker, rdio, etc., but based on reviews didn t think that they were worth bothering with. Conclusions I m left mainly considering Spotify and Subsonic. For right now, I m going to try Spotify and see where that leads me, but may wind up getting back to Subsonic if I can t find tracks I want with Spotify.

Christian Perrier: DebConf running: take 6 (South-West suburbs of Managua)

I've been lazy today: I just went to the same place than yesterday..:-) I actually waslate and missed the meeting with No l who indeed went running in about the same neighbourhood, as well as Martin Bagge who went running on the road to the "Intermezzo" restaurant I described earlier..:-) So, no real fancy description today. And, as usual, the full GPS trace is here. The interesting challenge is to see if I can manage to achieve a run every day..:-). We'll see!

8 July 2012

Jonathan Wiltshire: Point Release Security, Reloaded

When I first undertook the tracking of minor security fixes in point releases, I quickly out-scaled flat text files and a good memory. A Python library and sqlite database helped automate sending notifications and keeping tabs, but the manual work associated with tracking incoming bugs from the security team, applications to and responses from the release team, and the action or inaction of maintainers was still too time-consuming to be useful. This weekend I deployed pyprsc2, with a public view at http://prsc.debian.net/tracker/<bug>. I had planned to do this at Debconf12, but given the circumstances still, it needed doing anyway and what better time? Result: my work now involves adding tracks where required; keeping an eye on the notified list for manual prods; and after a point release, archiving the included bugs and updating the suite version numbers. Bliss. Features: Todo: Technical: prsc.debian.net leverages large parts of the Django MVC framework in fact, this was really a learning exercise in disguise since I want to use Django on some more complex projects later. BTS synchronisation is handled by python-debianbts, and synchronisation with proposed-updates is through XML and lxml/objectify (thanks to the release team s awesome XML queue viewer and Adam adding bug numbers to it). Since this was a learning exercise, some of the Python is probably questionable at best and downright wrong at worst, so it probably needs some work still.
Point Release Security, Reloaded is a post from: jwiltshire.org.uk Flattr

Petter Reinholdtsen: Debian Edu interview: George Bredberg

The Debian Edu / Skolelinux project have users all over the globe, but until recently we have not known about any users in Norway's neighbour country Sweden. This changed when George Bredberg showed up in March this year on the mailing list, asking interesting questions about how to adjust and scale the just released Debian Edu Wheezy setup to his liking. He granted me an interview, and I am happy to share his answers with you here. Who are you, and how do you spend your days? I'm a 44 year old country guy that have been working 12 years at the same school as 50% IT-manager and 50% Teacher. My educational background is fil.kand in history and religious beliefs, an exam as a "folkhighschool" teacher, that is, for teaching grownups. In Norwegian I believe it's called "Vuxenupplaring". I also have a master in "Technology and social change". So I'm not really a tech guy, I just like to study how humans and technology interact and that is my perspective when working with IT. How did you get in contact with the Skolelinux/Debian Edu project? I have followed the Skolelinux project for quite some time by now. Earlier I tested out the K12-LTSP project, which we used for some time, but I really like the idea of having a distribution aimed to be a complete solution for schools with necessary tools integrated. When K12-LTSP abandoned that idea some years ago, I started to look more seriously into Skolelinux instead. What do you see as the advantages of Skolelinux/Debian Edu? The big point of Skolelinux to me is that it is a complete distribution, ready to install. It has LDAP-support, MS Windows integration tools and so forth already configured, saving an administrator a lot of time and headache. We were using another Linux based thin-client system called Thinlinc, that has served us very well. But that Skolelinux is based on VNC and LTSP, to me, is better when it comes to the kind of multimedia used in schools. That is showing videos from Youtube or educational TV. It is also easier to mix thin clients with workstations, since the user settings will be the same. In our VNC-based solution you had to "beat around the bush" by setting up a second, hidden, home-directory for user settings for the workstations, because they will be different from the ones used on the thin clients. Skolelinux support for diskless workstations are very convenient since a school today often need to use a class room projector showing videos in full screen. That is easily done with a small integrated media computer running as a diskless workstation. You have only two installs to update and configure. One for the thin clients and one for the workstations. Also saving a lot of time. Our old system was also based on Redhat and CentOS. They are both very nice distributions, but they are sometimes painfully slow when it comes to updating multimedia support and multimedia programs (even such as Gimp), leaving us with a bit "oldish" applications. Debian is quicker to update. What do you see as the disadvantages of Skolelinux/Debian Edu? Debian is a bit too quick when it comes to updating. As an example we use old HP terminals as thinclients, and two times already this year (2012) the updates you get from the repositories has stopped sound from working with them. It's a kernel/ALSA issue. So you have to be more careful properly testing the updates before you run them in a production environment. This has never happened with CentOS. I also would like to be able to set my own domain-settings at install time. In Skolelinux they are kind of hard coded into the distribution, when it comes to LDAP and at least samba integration. That is more a cosmetic/translation issue, and not a real problem. Running MS Windows applications within the Skolelinux environment needs to be better supported. That is, running them seamlessly via RDP, and support for single-sign on. That will make the transition to free software easier, because you can keep the applications you really need. No support will make it impossible if you work in a school where some applications can't be open source. As for us we really need to run Adobe InDesign in our journalist classes. We run a journalist education, and is one of the very few non university ones that is ok:d by Svenska journalistf rbundet (Swedish journalist association). Our education gives the pupils the right of membership there, once they are done. This is important if you want to get a job. Adobe InDesign is the program most commonly used in newspapers and magazines. We used Quark Express before, but they seem to loose there market to Adobe. The only "equivalent" to InDesign in the opensource world is Scribus, and its not advanced enough. At least not according to the teacher. I think it would be possible to use it, because they are not supposed to learn a program, they are supposed to learn how to edit and compile a newspaper. But politically at our school we are not there yet. And Scribus lacks a lot of things you find i InDesign. We used even a windows program for sound editing when it comes to the radio-journalist part. The year to come we are going to try Audacity. That software has the same kind of limitations compared to Adobe Audition, but that teacher is a bit more open minded. We have tried Ardour also, but that instead is more like a music studio program, not intended for the kind of editing taking place in a radio studio. Its way to complex and the GUI is to scattered when you only want to cut, make pass-overs, add extra channels and normalise. Those things you can do in Audacity, but its not as easy as in Audition. You have to do more things manually with envelopes, and that is a bit old fashion and timewasting. Its also harder to cut and move sound from one channel to another, which is a thing that you do frequently because you often find yourself needing to rearrange parts of the sound file. So, I am not sure we will succeed in replacing even Audition, but we will try. The problem is the students have certain expectations when they start an education towards a profession. So the programs has to look and feel professional. Good thing with radio, there are many programs out there, that radio studios use, so its not as standardised as Newspaper editing. That means, it does not really matter what program they learn, because once they start working they still have to learn the program the studio uses, so instead focus has to be to learn the editing part without to much focus on a specific software. Which free software do you use daily? Myself I'm running Linux Mint, or Ubuntu these days. I use almost only open source software, and preferably Linux based. When it comes to most used applications its OpenOffice, and Firefox (of course ;) ) Which strategy do you believe is the right one to use to get schools to use free software? To get schools to use free software there has to be good open source software that are windows based, to ease the transition. But it's also very important that the multimedia support is working flawlessly. The problems with Youtube, Twitter, Facebook and whatever will create problems when it comes to both teachers and students. Economy are also important for schools, so using thin clients, as long as they have good multimedia support, is a very good idea. It's also important that the open source software works even for the administration. It's hard to convince the teachers to stick with open source, if the principal has to run Windows. It also creates a problem if some classes has to use Windows for there tasks, since that will create a difference in "status" between classes, so a good support for running windows applications via the thin client (Linux) desktop is essential. At least at our school, where we have mixed level of educations, from high-school to journalist-school. Update 2012-07-09 08:30: Paul Wise tipped me on IRC about three useful sources related to Free Software for radio stations: the LWN article Radio station management with Airtime, Airtime which claim to be a Free open source radio automation software and Rivendell which claim to be complete radio broadcast automation solution. All of them seem useful to the aspiring radio producer.

Raphael Geissert: Updates to http.debian.net

It's been a week with quite some changes to http.debian.net, the Debian mirrors redirector, and it keeps coping very well with the continuously increasing traffic: over 1.5 million requests from APT clients in the last seven days! half-million more from the week before.

The good news is, those rare Hash Sum mismatch errors should be mostly gone. Ditto for some other sort of errors. There is now a second server that takes care of monitoring the mirrors and is ready to handle some of the traffic if there's any need. With this new server, http.debian.net will soon be available over IPv6 too.

Those who are at Debconf12 and followed my advice to use http.debian.net will have noticed that it is redirecting users to the local mirror. So, once again, forget about switching mirrors!

Thanks are also due to J rg Jaspert and Philipp Kern, for the new server and for the work needed to allow http.debian.net access to Debconf12's local mirror, respectively.

Many thanks again to those who keep providing feedback and have helped the project along the way.

What's next? even more improvements and fixing some issues, some of which involve further collaboration and cooperation with the mirror administrators.

Some parts of the Debian mirrors network are fragile and may bite every once in a while, but those are being worked on. If you administer a mirror, please use ftpsync and submit your Debian mirror.
You might be happy to know that I'm working on restricting mirrors to specific Autonomous System, or countries. More on this later.

Iustin Pop: Thumbnails and other EXIF metadata fun

So, I was trying to write a script for myself to sanitise the EXIF metadata in images produced by my camera. I had a text file for a while with some tags that I should remove (like the camera serial number, etc.), but this became cumbersome and I wanted a fully scripted workflow. I thought initially it's a simple exercise: look at the current tags, decide either which one I wanted to remove (black-list) or alternatively keep (white-list; preferred solution) and then call exiftool with the appropriate options. Boy oh boy EXIF is just part of the equation. There are of course also the MakerNotes, which are manufacturer-specific and most of the time editable only as a block. Furthermore, there's no way to easily just duplicate each individual entry in this group to a standard EXIF or even XMP section, so basically you either keep to much or just give up a lot. Some of them can be copied to XMP, for example lens information, flash, etc. but white-balance (with the exiftool version that I have) is not, for example. Ideally I could just copy all maker notes to XMP, and then drop that entire group. For the record, the way to drop all and just copy some is to run:
exiftool -all= -tagsfromfile @ -Orientation -AllDates  other tags  file.jpg
But do not add "-makernotes", because that will bring the entire block back. The other fun thing is the number of thumbnails/preview images embedded. Looking at a random jpeg file, I found: Now, I doubt even the use of one thumbnail, but 3? And one being more than half a megabyte? Someone's going overboard at Nikon On the plus side, removing all thumbnails and sanitising the meta-data makes the jpeg files quite a bit smaller, which also means easier to share, at least; not sure if I want to actually clean the master copies (even though I keep the NEFs).

Dirk Eddelbuettel: New package RcppCNPy with release 0.1.0 (and 0.0.1 earlier last week)

A few days ago I had blogged about getting NumPy data in R by using a simple converter script. That works fine, but it is a little annoying to have to write an entire file only to read from it again. So I kept looking around for a better solution---and soon found the cnpy library by Carl Rogers which provides simple C++ functions to read and write NumPy files. Bringing such a C++ library to R is done very easily via Rcpp modules. The resulting package contains a single R file with a single line: loadModule("cnpy", TRUE). And it relies on the following module declarations in the a C++ file:
RCPP_MODULE(cnpy) 
    using namespace Rcpp;
    function("npyLoad",                         // name of the identifier at the R level
             &npyLoad,                          // function pointer to helper function defined above
             List::create( Named("filename"),   // function arguments including default value
                           Named("type") = "numeric"),
             "read an npy file into a numeric or integer vector or matrix");
    function("npySave",                         // name of the identifier at the R level
             &npySave,                          // function pointer to helper function defined above
             List::create( Named("filename"),   // function arguments including default value
                           Named("object"), 
                           Named("mode") = "w"),
             "save an R object (vector or matrix of type integer or numeric) to an npy file");
 
which give us at the R prompt
R> library(RcppCNPy)
Loading required package: Rcpp
R> npyLoad
internal C++ function <0x243af70>
    docstring : read an npy file into a numeric or integer vector or matrix
    signature : Rcpp::RObject npyLoad(std::string, std::string)
R> npySave
internal C++ function <0x23033e0>
    docstring : save an R object (vector or matrix of type integer or numeric) to an npy file
    signature : void npySave(std::string, Rcpp::RObject, std::string)
R> 
these two functions (and their docstrings) defined above. That's all! Well there are about one hundred more lines dealing with whether we have integer or numeric data, and whether we use a vector or a matrix. But all in all pretty simple... So version 0.1.0 of this new package RcppCNPy completes the initial release 0.0.1 from earlier in the week by adding The NEWS entry for this release (as well as the initial one) follow:
News for Package RcppCNPy Changes in version 0.1.0 (2012-07-07)
  • Added automatic use of transpose to automagically account for Fortran-vs-C major storage defaults between Python and R.
  • Support for integer types in dependent on the int64_t type which is available only when the -std=c++0x switch is used at build-time (and CRAN still discourages use of it)
  • Added support for reading gzip'ed files ending in ".npy.gz"
  • Added regression tests in directory tests/
  • Added a vignette describing the package
  • Added a timing benchmark in demo/timings.R
Changes in version 0.0.1 (2012-07-04)
  • Initial version, as a straightforward Rcpp modules wrap around the cpny library by Carl Rogers (on github under a MIT license).
  • At present, npy files can be read and written for vectors and matrices of either numeric or integer type. Note however that matrices are currently transposed because of the default Fortran ordering done by numpy.
I will follow up with a little usage example later. CRANberries also provides a diffstat report for 0.1.0 relative to 0.0.1. As always, feedback is welcome and the rcpp-devel mailing list off the R-Forge page for Rcpp is the best place to start a discussion.

Lars Wirzenius: Obnam 1.1 (backup software)

I've released Obnam version 1.1, released 2012-06-30, but only announced now since I had trouble building the packages for code.liw.fi.

Steve Kemp: Misc update.

I got a few emails about the status panel I'd both toyed with and posted. The end result is that the live load graphs now have documentation, look prettier, and contain a link to the source code. Apart from that this week has mostly involved photographing cute cats, hairy dogs, and women in corsets. In Debian-related news njam: Insecure usage of environmental variable was closed after about 7 months, and I reported a failure of omega-rpg to drop group(games) privileges prior to saving game-state. That leads to things like this:
skx@precious:~$ ls -l   grep games
-rw-r--r--   1 skx games   14506 Jul  8 15:20 Omega1000
Not the end of the world, but it does mean you can write to directories owned by root.games, and potentially over-write level/high-score files in other packages leading to compromises. ObQuote: "Your suffering will be legendary, even in hell! " - Hellraiser II (Did you know there were eight HellRaiser sequels?)

Petter Reinholdtsen: Why do schools waste money on IT?

In the Debian Edu / Skolelinux project, we have realised that one of the major blockers for the project success is the purchasing skills in schools and municipalities. We provide what the happy users of Debian Edu / Skolelinux say they need and to a lower cost than the alternatives, and yet so few schools decide to use our solution. I was pleased to discover the same observation done by mySociety and Tom Steinberg in his blog post "Can you recognize the million pound chair?". Read it and weep for the spending of your tax money. Of course there are other factors involved as well, like our projects bad marketing skills and the Linux community fragmentation causing worry with the people on the outside, so we as a project need to keep working hard to gain users, but it is a up-hill battle when public decision makers are unable to understand computer system purchases.

7 July 2012

Tiago Bortoletto Vaz: Some pictures of DC11 Cheese and Wine party

In DebConf9 i worked on some photo editing and ended with a sort of over-saturated album which is still available online. For the C&W party in DebConf11 i tried to delivery a black and white set of pictures by just doing some adjustments using ufraw. Pictures were taken with a Nikon D50 camera + Nikkor 50mm 1/1.4 in raw mode. Here are some examples of the result:
debconf11debconf11debconf11
debconf11debconf11debconf11
debconf11debconf11debconf11
You can enjoy these and other DebConf11 shots by following the links in DC11 wikipage. (update: sorry for this old post being re-published in the planet, i've migrated my blog and this post got a new timestamp)

Tiago Bortoletto Vaz: Recycling old Debian t-shirts

debian-bed ...and having beautiful dreams :)

Tiago Bortoletto Vaz: VoIP in Neo Freerunner with Qtmoko and Linphone

I keep using a Neo Freerunner with Qtmoko in a Debian Squeeze (+ some Wheezy) chroot as my daily phone. It also works well for SMS, GPS, MUA, IM, IRC and some Web browsing with Aurora (using both Wifi and GPRS). Last weekend I managed to play a little with VoIP possibilities in Qtmoko and ended with a functional SIP client in my hands. I want to share some points which may be useful for other brave freerunners: UPDATE: I didn't make clear that I get a very nice audio quality using internal mic/speaker with the above voip-handset state file. The issue is only when using the headset, which demands understanding and setting up a proper state file. Enjoy some shots:

Christian Perrier: DebConf running: take 5 (South-West suburbs of Managua)

The challenge is now more and more to find good places to run without too much cars and noises....and not too far away. Today, I think I made it well..:-) After a look on the map, and some wild guess from what I now know of the way the city is organized, I decided to go south-west of Managa, from the hotel Seminole. This is again a hilly run because, anyway, wherever you go south in Managua, you're going up. Ralf, who arrived just yesterday, joined me and, I guess, enjoyed the run. After going through an obviously quite rich neighbourhood with nice villas, gardeners, huge US truck-style cars, we went close to the Mormonschurch...and a mosque, then headed westward close to "Colegio Americano", with fences, walls, guards in the corner, etc. Seems that any tiny bit of USA in the world needs an army to protect it. Crossing la Pista Suburbana, we then went on a very quiet roads towards the hill, in a very green and peaceful area. House there were really not as fancy as in other places, even seeming quite "poor" in some way. However, we never felt any problem and people we meet on the way are always friendly and smiling : "Hola", "Buenos Dias", etc. At the end of the road, after about half an hour, we decided to head back. However, as going back the same way is boring, I suggested we head up west as my phone's map (it is very helpful to have a smartphone with GPS and Google Maps for wandering through unknwon places) was mentioning another road going north-west a little bit westward. So we entered a small trail between house farms...with dogs! These were a bit "scary" as they were barking at us and of course running entirely freely. On the other hand, none was really aggressive and we had no problem. I however saw a few people really staring at us as I guess there are not so many runners in this place..:-). But still, we never felt unsafe: just in a quite uncommon place. After abotu 500m we found the "road" that was on my map: indeed a path going down slowly along the hill. And there was the marvel. An incredible panoramic view going going the volcanoes that are East and North of Leon : San Cristobal, Telica Rota, Pilas elHoj and last but definitely not least: EL MOMOTOMBO. A nearly perfect pyramid-style volcano that lies about 50km north-west of Managua, on the eastern shore of Managua Lake. The view there is...just fantastic with also a 180 panorama to West and North-West of the lake and the volcanoes area. After this, all we had to do was heading back to the hotel and share this with you...:-) As usual, the full GPS trace is here.

Bartosz Fe&#324;ski: God bless LVM

Correct me if I m wrong (and I really want to be wrong).
Is that whole LVM crap expects me to be forecaster/fortune-teller? I wanted to upgrade some squeeze based virtual machine to wheezy. Sounds nice, yeah?
So before the upgrade you should ensure that you have backup, right?
Assuming that whole system is LVM based all you have to do before upgrade is LVM snapshot.
I did it and I was quite surprised that I had to declare its size.
I did it basing on assumption that upgrade shouldn t take more than let say 5GB. Tada!
mastersrv:~# lvs
  LV       VG      Attr   LSize   Origin  Snap%  Move Log Copy%  Convert
  backup   virtual -wi-ao  50,00g                                       
  mainsrv  virtual -wi-ao 300,00g                                       
  redminbu virtual swi-a-   5,00g redmine   0,00                        
  redmine  virtual owi-a-  19,53g
Now I did upgrade redmine virtual machine and this is what I got:
mastersrv:~# lvs
  LV       VG      Attr   LSize   Origin  Snap%  Move Log Copy%  Convert
  backup   virtual -wi-ao  50,00g                                       
  mainsrv  virtual -wi-ao 300,00g                                       
  redminbu virtual Swi-I-   5,00g redmine 100.00                        
  redmine  virtual owi-ao  19,53g
These attr flags mean Snapshot invalid and Invalid snapshot .
End of the story this is last time I actually tried that method of backup.
Happily I have also some traditional backup (mysqldump+rsync) but I m really surprised that LVM works this way.
There was plenty of free space to enlarge that volume and this is what you get in your logs:
[13402279.777238] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Unable to allocate exception? Did it really try to do it?
mastersrv:~# pvs
  PV         VG      Fmt  Attr PSize   PFree  
  /dev/md0   virtual lvm2 a-   884,01g 514,46g
I don t think so.

Petter Reinholdtsen: Free Timetabling Software - nice free software

Included in Debian Edu / Skolelinux is a large collection of end user and school specific software. It is one of the packages not installed by default but provided in the Debian archive for schools to install if they want to, is a system to automatically plan the school time table using information about available teachers, classes and rooms, combined with the list of required courses and how many hours each topic should receive. The software is named FET, and it provide a graphical user interface to input the required information, save the result in a fairly simple XML format, and generate time tables for both teachers and students. It is available both for Linux, MacOSX and Windows. This is the feature list, liftet from the project web site: I have not used it myself, as I am not involved in time table planning at a school, but it seem to work fine when I test it. If you need to set up your schools time table, and is tired of doing it manually, check it out. A quick summary on how to use it can be found in a blog post from MarvelSoft. If you find FET useful, please provide a recipe for the Debian Edu project in the Debian Edu HowTo section.

Keith Packard: FEC

Forward Error Correction with the TI CC1120 Sub-GHz transceiver We re building a new, fancier flight computer and decided to give the new, fancier TI CC1120 chip a try. It has significantly more transmit power (+16dBm) and receive sensitivity (-109dBm) than the transciever built in to the TI CC1111 RF SoC that we ve built our other boards with. Given that we d already decided to use a fancier STM32L processor, using the nicest stand-alone radio chip we could find seemed to make good sense. One of the nice features found in the CC1111 is support for FEC using a 1/2 rate constraint length 4 convolutional code. Both the encoder and the soft-decision Viterbi decoder are built right into the chip; all we had to do was flip a couple of bits and we got error correction, interleaving, data whitening and CRC checking all for free. We assumed that, as the CC1120 was advertised as a better replacement for our CC1111 radio, that it would also include all of these fine features. After we got the chip all soldered down to our first prototype boards, we read through the reference manual looking for the right register values to flip for forward error correction. We found support for CRC computation and data whitening, but no mention of interleaving or forward error correction at all. Thanks, TI. Of course, without the FEC pieces, the other packet handling hardware is worthless you can t exactly do a CRC on the encoded data and expect it to be useful on the other end of the link, and data whitening happens after the CRC. Basic CC1120 driver I assumed I d eventually figure out something to do about the FEC pieces and started writing a basic CC1120 driver AltOS, the operating system used in all of our projects. I started with the transmit side, as that seemed easier as I could use the existing packet hardware by uploading the desired bit sequence using whatever fancy encoding desired and the hardware would happily transmit it. I first tested that using low data-rate bits (2kbps), transmitting an alternating sequence of zeros and ones to generate a 1kHz tone on a regular 70cm receiver. That seemed to work just fine, demonstrating that I had some grasp of the basic operation of the chip. Convolutional Encoding in Software Not to be daunted by a small challenge in software, I set out to replicate the encoding scheme used in the CC1111 chips for use in our CC1120-based design. We obviously want our CC1120 boards to interoperate with the CC1111 boards. Fortunately, TI fully documents the CRC, data whitening, convolutional encoding and interleaving done in the hardware. I created my own implementation of the encoder (licensed, as usual, under the GPLv2) which you can see in the file ao_fec_tx.c It s not quite as flexible as the hardware, it always whitens the data and always appends the CRC bytes to the end of the packet, along with the necessary trellis termination bytes. The core of that code is surprisingly simple:
static const uint8_t ao_fec_encode_table[16] =  
/* next 0  1      state */
    0, 3,   /* 000 */
    1, 2,   /* 001 */
    3, 0,   /* 010 */
    2, 1,   /* 011 */
    3, 0,   /* 100 */
    2, 1,   /* 101 */
    0, 3,   /* 110 */
    1, 2    /* 111 */
 ;
uint8_t
ao_fec_encode(const uint8_t *in, uint8_t len, uint8_t *out)
 
    uint8_t     extra[AO_FEC_PREPARE_EXTRA];
    uint8_t     extra_len;
    uint32_t    encode, interleave;
    uint8_t     pair, byte, bit;
    uint16_t    fec = 0;
    const uint8_t   *whiten = ao_fec_whiten_table;
    extra_len = ao_fec_prepare(in, len, extra);
    for (pair = 0; pair < len + extra_len; pair += 2)  
        encode = 0;
        for (byte = 0; byte < 2; byte++)  
            if (pair + byte == len)
                in = extra;
            fec  = *in++ ^ *whiten++;
            for (bit = 0; bit < 8; bit++)  
                encode = encode << 2   ao_fec_encode_table[fec >> 7];
                fec = (fec << 1) & 0x7ff;
             
         
        interleave = 0;
        for (bit = 0; bit < 4 * 4; bit++)  
            uint8_t byte_shift = (bit & 0x3) << 3;
            uint8_t bit_shift = (bit & 0xc) >> 1;
            interleave = (interleave << 2)   ((encode >> (byte_shift + bit_shift)) & 0x3);
         
        *out++ = interleave >> 24;
        *out++ = interleave >> 16;
        *out++ = interleave >> 8;
        *out++ = interleave >> 0;
     
    return (len + extra_len) * 2;
 
ao_fec_encode_table takes care of the convolutional piece, ao_fec_whiten_table (not shown) contains the data whitening codes while the little loop at the bottom does the interleaving. I didn t spend a lot of time optimizing this as it seemed to run plenty fast on the 32MHz processor. With the bits all nicely encoded, I passed them to my simple CC1120 driver and let it send them along to our CC1111 receiver. Much to my delight, with only a few bugs fixed, it worked just fine! I d managed to get one direction working, validating the hardware and making it clear that we d at least be able to use the board in flight for telemetry. Getting Soft-decision data out of the CC1120 In packet mode, the CC1120 receiver converts the incoming GFSK stream directly into bits, making its best guess as to whether the detected value was a one or zero. Of course, like any FM modulation, the receiver will be producing intermediate values, especially when the signal is weak or masked by interference. A Viterbi decoder can make good use of this information;these noisy intermediate values are less influential in the final resulting path cost which allows the strong 0/1 values to pull the solution to the right one. Soft decision decoding yields a theoretical 2dB gain over hard decision decoders, and given our desire for maximum packet reception over long distances, it seemed important to make this work. The CC1120 has a way to recover the soft decision values, but only one 8-bit value at a time. It does, however, helpfully provide a GPIO wire which can be programmed to interrupt the CPU when the soft decision data has arrived. So, I hooked up a GPIO line on the processor to the GPIO line on the CC1120 and let it interrupt once per bit time, or 38400 times per second, to read this valuable soft decision data out of the radio part. According to TI support, this is the only way to make this work, and that they generally recommend that developers use the packet mode stuff and throw away this valueable source of data. I was happily surprised to learn that the STM32L can in fact survive this interrupt rate, and by setting interrupt priorities appropriately, I manage to reliably capture an entire packets worth of data without dropping a single bit. I set up a simple packet dumping function to run on the prototype board and captured raw encoded bytes from our CC1111-based flight computer so that I d have some real data to work from. Soft-decision Viterbi Decoding The sample encoding implementation from TI made that piece fairly straightforward. Lacking a similar sample for the decoding piece meant that I d be starting from scratch. And, implementing something that seemed like magic to me when I started. I started by reading a few articles on Viterbi decoding: I found Chip Fleming s tutorial very useful as it included a bunch of worked examples using a convolutional encoders with constraint length (k) of 3 instead of 4; this makes all of the trellis diagrams half the size. I also sent my friend, Phil Karn, a query about this process, and he responded with a bunch of helpful suggestions, while also recommending that I just steal his DSP and FEC library implementation. Of course, I don t like using code that I can t understand, and his code only implements much longer codes than I needed, so I d have to understand it anyways to make it work for me. Phil s code is a marvel, but it s not exactly comprehensible to someone who doesn t even know what a Viterbi decoder is supposed to do. So, with Chip s tutorial on my screen, I wrote a primitive Viterbi decoder. This did only the Viterbi step, and split that into two phases. The first computed the cost for each state in the system for every received pair of symbols received. It also tracked the chain of states through the entire decode process:
    for (state = 0; state < 8; state++)  
        struct ao_soft_sym zero = ao_soft_sym(ao_fec_encode_table[state * 2 + 0]);
        struct ao_soft_sym one = ao_soft_sym(ao_fec_encode_table[state * 2 + 1]);
        uint8_t zero_state = ao_next_state(state, 0);
        uint8_t one_state = ao_next_state(state, 1);
        int zero_cost = ao_cost(s, zero);
        int one_cost = ao_cost(s, one);
        zero_cost += cost[b][state];
        one_cost += cost[b][state];
        if (zero_cost < cost[b+1][zero_state])  
            prev[b+1][zero_state] = state;
            cost[b+1][zero_state] = zero_cost;
         
        if (one_cost < cost[b+1][one_state])  
            prev[b+1][one_state] = state;
            cost[b+1][one_state] = one_cost;
         
     
At the end of the packet, it found the least costly path and walked that from back to front, generating output bits (yes, one bit per element of the bits array):
for (state = 1; state < 8; state++)  
    if (cost[b][state] < c)  
        c = cost[b][state];
        min_state = state;
     
 
for (b = len/2; b > 0; b--)  
    bits[b-1] = min_state & 1;
    min_state = prev[b][min_state];
 
Completely separate from this code were the interleaving and data whitening stages. Keeping those separate allowed them to be tested independently, which made debugging them far easier. None of this code ever ran on the STM32L processor, it was instead run in a test framework on my laptop. I first got this working with data generated by the software convolutional encoder, then I managed to get it to decode the raw packets that I had recorded directly from the receiver it self. This marked a significant milestone in my mind I d managed to replace, in software, the CC1111 FEC encoder and decoder. However, the decoder performance was not fast enough to manage back-to-back packets yet. Having something in hand that worked meant that further development would be directly comparable to something that did work, making optimization much easier. Yet another lesson in making it work, and then making it work fast. Optimizing the Viterbi Decoder The first step was to reduce memory usage within the decoder. While it s best to defer computing the path until the very last bit has been received, it s also true that the chances of a bit affecting old data reduces as the distance between the new and old data increases. With the rule of thumb being a distance of k 6 or k 7, I saved 32 output bits for each state and wrote the oldest of 8 each time the buffer filled up, making a distance of 24 bits between the newest saved data and the current decoding position. That meant a fixed amount of storage was needed for an arbitrarily long packet. I tested a variety of different history sizes from 8 to 56 bits (cooresponding to 16, 32 and 64 bits of saved data for each state). With 100000 random packets generated that had gaussian noise added to each bit, here are the error rates for the different history lengths:
History LengthTotal PacketsCorrectIncorrect
8 bits10000090520 (90.52%)9480 (9.48%)
24 bits10000093614 (93.61%)6386 (6.39%)
56 bits10000093620 (93.62%)6380 (6.38%)
The performance of the 8-bit history and 16-bit history versions turned out to be essentially identical (given the 32-bit nature of the STM32L, that shouldn t be too surprising). The performance of the 56-bit version, which manipulated uint64_t data types was sigificantly slower, and given the very modest benefit, deemed not worth it. If we move to a larger constraint length (k > 4), or started using a punctured convolutional encoding, we would want to re-measure this. The next step was to combine the de-interleaving, decoding, de-whitening and CRC computation into a single loop. This would eliminate a pile of data copying and extra memory usage. Nothing fancy here, but it made enough of a difference that I could decode most of the incoming packets now, dropping only about 25% of them. To get to being able to decode every packet, I needed to start decoding the packet while it was being received. Because of interleaving, I had to have 32 bits of data to be able to decode anything, so it was a simple matter of having my per-soft-value interrupt handler start the decoder going after each interleave block of data was received. With this in place, I was able to decode a 76-byte payload (160 transmitted bits) in 14.7 ms, or just about 50% of the time it took to transmit the packet. Unrolling one of the inner decoding loops eliminated a bunch of computation and sped this up even more, decoding packets in 9.5ms. Success at last, a solid 100% of packets received were being decoded, as long as the CPU was otherwise idle. Finally, I went and re-read Phil Karn s code and pondered over how it works. It was opaque until I wrote a message back to Phil asking how it worked and pointing out sections that were confusing. Fortunately, simply phrasing my confusion in words made me understand how the code works. Here s my original code for computing the per-state cost metric:
/* Metric for a zero bit */
uint32_t    bitcost = ((uint32_t) (s0 ^ ao_fec_decode_table[(state<<1)]) +
               (uint32_t) (s1 ^ ao_fec_decode_table[(state<<1)+1]));
 
    /* Total path metric and state for a zero bit */
    uint32_t    cost0 = cost[p][state] + bitcost;
    uint8_t     state0 = ao_next_state(state, 0);
    /* Compare the total path metric against all existing
     * metrics heading into the same new state, choosing
     * the least expensive and killing the others.  Record
     * the new lowest cost and output bit.
     */
    if (cost0 < cost[n][state0])  
        cost[n][state0] = cost0;
        bits[n][state0] = (bits[p][state] << 1)   (state & 1);
     
 
 
    /* Total path metric and state for a one bit.  Because
     * encoding a one is always exactly the opposite of
     * encoding a zero from any state, this cost is
     * 255*2 - bitcost
     */
    uint32_t    cost1 = cost[p][state] + 510 - bitcost;
    uint8_t     state1 = ao_next_state(state, 1);
    if (cost1 < cost[n][state1])  
        cost[n][state1] = cost1;
        bits[n][state1] = (bits[p][state] << 1)   (state & 1);
     
 
Before this loop is run, I have to initialize cost[n] to a large value so that the comparisons will be true for the first case to reach each state. Phil s code, in contrast, doesn t have to initialize cost[n] as we know which two states can reach any possible new state. His code looks like this:
/* Metric for a zero bit */
bitcost = ((uint32_t) (s0 ^ ao_fec_decode_table[(state<<1)]) +
       (uint32_t) (s1 ^ ao_fec_decode_table[(state<<1) 1]));
/* Only state and state+4 reach state<<1 with a zero bit */
/* Cost from state to state<<1 for a zero bit*/
m0 = cost[p][state] + bitcost;
/* Cost from state+4 to state<<1 for a zero bit */
m1 = cost[p][state+4] + (510 - bitcost);
/* Which source state is cheaper? */
bit = m0 > m1;
/* Record the cheaper cost and the add to the path of bits */
cost[n][state<<1] = bit ? m1 : m0;
bits[n][state<<1] = (bits[p][state + (bit<<2)] << 1)   (state&1);
/* State and state+4 reach (state<<1)+1 with a one bit */
/* Cost from state to (state<<1)+1 for a one bit */
m0 -= (bitcost+bitcost-510);
/* Cost from state+4 to (state<<1)+1 for a one bit */
m1 += (bitcost+bitcost-510);
/* Which source state is cheaper? */
bit = m0 > m1;
/* Record cheaper cost and add to the path of bits */
cost[n][(state<<1)+1] = bit ? m1 : m0;
bits[n][(state<<1)+1] = (bits[p][state + (bit<<2)] << 1)   (state&1);
This code does two states at a time, and so while it s slightly longer than the above, it only needs to run half as many times. The resulting code now decodes packets in 5.7ms. The final version of the code can be see in ao_fec_rx.c The whole AltOS operating system, with many drivers and support for CC1111, AVR and STM32L processirs is available, under the GPLv2 from git://git.gag.com/scm/git/fw/altos

Robert Collins: Reprap driver pinouts

This is largely a memo-to-my-future self, but it may save some time for someone else facing what I was last weekend. I ve been putting together a Reprap recently, seeded by the purchase of a partially assembled one from someone local who was leaving town and didn t want to take it with them. One of the issues it had was that 2 of the stepstick driver boards it uses were burnt out, and in NZ there are no local suppliers that I could find. There is however a supplier of Easydriver driver boards, which are apparently compatible. (The Reprap electronics is a sanguinololu, which has a fitted strip that exactly matches stepstick (or pololu) driver boards. The Easydrivers are not physically compatible, but they should be pin compatible.. no? I mapped across all the pins carefully, and the only issues were: there are three GND s on the Easydriver vs 2 on the stepstick, and the PFD pin isn t exposed on the stepstick board so it can t be mapped across. I ended up with this mapping (I m not sure where pin 1 is *meant* to be on the stepstick, so I m starting with VMOT, the anti-clockwise corner pin on the same side as the 2B/2A/1A/1B pins, when looking down on an installed board pin 1, and going clockwise from there). Stepstick Easydriver VMOT M+
GND GND
2B B2
2A A2
1A A1
1B B1
VDD +5V
GND GND
Dir Dir
Step Step
Slp Slp
Rst Rst
Ms3 Nothing
Ms2 Ms2
Ms1 Ms1
En Enable But, when I tried to use this, the motor just jammed up solid. A bit of debugging and trial and error later and I figured it out. The right mapping for the motor pins: 2B B2
2A B1
1A A1
1B A2 Thats right, the two boards have chosen opposed elements for labelling of motors coils pins on the step stick 1/2 refers to the coil and A/B the two ends that need to have voltage put across them, on the easydriver A/B refer to the coil and 1/2 the two ends Super confusing, especially as I haven t been doing much electronics for oh, a decade or so. I m reminded very strongly of Rusty s scale of interface usability here.

6 July 2012

Gregor Herrmann: DebCamp impressions

I'm here in managua since sunday evening, & by now DebCamp is slowly getting to its end. So here are some of my impressions so far: thanks to all organizers & sponsors who make this experience happen!

Christian Perrier: DebConf running: take 4 (Masaya volcano)

Yet another Grand Plan today: run in Masaya Volcan National Park. Masaya Volcano is one of the most active volcanoes in Nicaragua. It is located only 20km south-east of Managua, so going there is as easy as renting a taxi. The original plan was to drive to the entrance of the National park (on the road to Masaya city), then run up to the volcan, so from 250m altitude up to about 600m. I therefore rented a taxi with the kind help of Norman, and we agreed to meet up at 08:30 at the hotel, along with No l and Daniel (who was planning to walk up, not running). After discussing here or there, Kurt and Gaudenz joined, both with the intent of running up. Others would also have liked to join, but I wasn't in the mood of organizing a full bus of Debconfers..:-). My plan was to enjoy pyself the hike...not to manage several taxis, etc. We thus packed in Denis blue taxi...a fancy and very visible customized blue car that looked like it had been the topic of on episode of the "Pimp My Ride" TV show..:-) Sitting 4 people at the back of a regular car without hitting some blue neon lights, or 7" TV screens is kinda tricky, but we made it. When arriving at the park's entrance, we however learned that it is forbidden to walk or hike up the road by foot, because of potential high concentrations of Carbon Dioxide. Sadly, we then had to pack again in the car and go up with it. After reaching the parking close to the volcano's active crater (that is said to have a small lava lake at its bottom, one of the very few in the world), we decided to head up for going round the Nindira crater, an inactive crater located slightly above. Unfortunately, again, the path around the Masaya crater is forbidden to walk on, because of landslides. We indeed still had great fun by running around this crater (which is about a 2.5 kilometers round trip, very very hilly and sometimes hard to run on such as a mountain path. After about 50 minutes and two laps (after all, we had a giant stadium!), we headed back to the parking, packed 5 sweating geeks in the car and went back to the hotel. Full GPS trace is here. I also put a few pictures on Facebook, supposedly visible by anybody : Masaya crater and Nindiri crater.

Next.

Previous.