Search Results: "zed"

11 March 2015

Joey Hess: 7drl 2015 day 5 type directed spell system development

I want my 7drl game Scroll to have lots of interesting spells. So, as I'm designing its spell system, I've been looking at the types, and considering the whole universe of possible spells that fit within the constraints of the types. My first throught was that a spell would be a function from World -> World. That allows any kind of spell that manipulates the game map. Like, for instance a "whiteout" that projects a stream of whitespace from the player's mouth. Since Scroll has a state monad, I quickly generalized that; making spell actions a state monad M (), which lets spells reuse other monadic actions, and affect the whole game state, including the player. Now I could write a spell like "teleport", or "grow". But it quickly became apparent this was too limiting: While spells could change the World map, the player, and even change the list of supported spells, they had no way to prompting for input. I tried a few types of the Event -> M () variety, but they were all too limiting. Finally, I settled on this type for spell actions: M NextStep -> M NextStep. And then I spent 3 hours exploring the universe of spells that type allows! To understand them, it helps to see what a NextStep is:
type Step = Event -> M NextStep
data NextStep = NextStep View (Maybe Step)
Since NextStep is a continuation, spells take the original continuation, and can not only modify the game state, but can return an altered continuation. Such as one that prompts for input before performing the spell, and then calls the original continuation to get on with the game. That let me write "new", a most interesting spell, that lets the player add a new way to cast an existing spell. Spells are cast using ingredients, and so this prompts for a new ingredient to cast a spell. (I hope that "new farming" will be one mode of play to possibly win Scroll.) And, it lets me write spells that fail in game-ending ways. (Ie, "genocide @"). A spell can cause the game to end by returning a continuation that has Nothing as its next step. Even better, I could write spells that return a continuation that contains a forked background task, using the 66 line contiuation based threading system I built in day 3. This allows writing lots of fun spells that have an effect that lasts for a while. Things like letting the player quickly digest letters they eat, or slow down the speed of events. And then I thought of "dream". This spell stores the input continuation and game state, and returns a modified continuation that lets the game continue until it ends, and then restores from the point it saved. So, the player dreams they're playing, and wakes back up where they cast the spell. A wild spell, which can have different variants, like precognitive dreams where the same random numbers are used as will be used upon awaking, or dreams where knowledge carries back over to the real world in different ways. (Supports Inception too..) Look how easy it was to implement dreaming, in this game that didn't have any notion of "save" or "restore"!
runDream :: M NextStep -> M NextStep -> (S -> S) -> M NextStep
runDream sleepcont wakecont wakeupstate = go =<< sleepcont
   where
         go (NextStep v ms) = return $ NextStep v $ Just $
        maybe wake (go <=<) ms
         wake _evt = do
                 modify wakeupstate
                 wakecont
I imagine that, if I were not using Haskell, I'd have just made the spell be an action, that can do IO in arbitrary ways. Such a spell system can of course do everything I described above and more. But, I think that using a general IO action is so broad that it hides the interesting possibilities like "dream". By starting with a limited type for spells, and exploring toward more featureful types, I was able to think about the range of possibilities of spells that each type allowed, be inspired with interesting ideas, and implement them quickly. Just what I need when writing a roguelike in just 7 days!
Fun comment thread on reddit

Tiago Bortoletto Vaz: Freeing myself from flickr

A flickr standard account gives you a free as in facebook service (I really wanted to reuse it!!!). I don't know about the pro account, but I don't believe it will give you much respect. Anyway, I realized that my photo albums in flickr are still online. And I'm currently unable to access my photos locally. I needed to download all them, then I decided to give flickrbackup a try. I started coding it a few years ago because at that time there was no free tools available for that. And then I abandoned it, too bad I feel. But for my surprise it worked without issues! And that's all I needed in my Debian box:
$ apt-get install flickrbackup $ mkdir myflickr $ flickrbackup -o myflickr/
(this will open a default browser for authentication and will automatically get the API key, then I just need an ENTER to start getting all my albums) I'm not sure whether there're other free tools (as in freedom) for that, but before paying for a license or trusting an online service for downloading your sets please give flickrbackup a chance :) I'll probably set a piwigo instance in a vps. But I fear php. So, suggestions on web galleries are very welcome.

9 March 2015

Ulrike Uhlig: Final report from the Outreach Program round 9

Between december 9th 2014 and march 9th 2015 I ve been working as an intern for Debian on Improving AppArmor support in Debian. Progress Before the internship started, I had no clue about AppArmor at all. Reading the Wikipedia page and the AppArmor wiki did barely enlighten me. At least, until I got my hands dirty and installed and activated it. I was able to do this only because there was already some basic documentation on the Debian wiki on how to get AppArmor running on a Debian system. When the internship started I felt a little bit lost. I started to read the documentation and to play around with some AppArmor profiles. But the error messages I saw and the profiles I opened for reading overwhelmed me. What did all these lines mean? Then one mentor briefed me about the current status of AppArmor in Debian. I took many notes and tried to bring some order into the information I was being given. These notes became the first page of the documentation and my first blog post.
Surprise: suddenly I was in the middle of knowing what we were talking about and how the next few weeks would look like. To start with, I have set up a test environment and continued to test some of the profiles which came with the packages I had installed, namely apparmor-profiles-extra which provided me with a profile for Pidgin. I was very much interested in securing this several thousand lines long C-code application on my machine! As a Pidgin user, I noticed immediately that one of my favourite features pidgin blinklight threw an error in the logs. I tried to fix it on my test VM and eventually made it work. Now knowing that such fixes should first be done upstream, i cloned the corresponding upstream repository and sent my first patch. It was not easy at first to switch to another VCS than Git, although I had been using Bazaar and Subversion years ago. Shortly afterwards, I saw my patch showing up on the upstream mailing list, and some days later it got merged. That was a huge encouragement. I was now able to write more documentation on how to contribute upstream. In order to organize the documentation, we ve set up some User Stories. I did not know about this tool before and had quite a hard time to grasp all the details but it proved to be very useful as some pages started to become long and confusing. Writing the documentation took up a lot of time, but before being able to write anything down, I needed to work everything out! This way I incidentally learned how to interact with many parts of Debian s infrastructure: While playing around with the tools provided to inspect AppArmor status on the system, I even accidentally found a little bug in one of the upstream tools. It got fixed very quickly after I mentioned it on the AppArmor team s mailing list. Organization Although I was uncomfortable with this in the beginning, we organized public meetings every two weeks on IRC. These meetings were also attended by MeetBot who took care of taking notes and leaving a trace of our discussions.
Based on the meetings, I maintained a progress page and todo list on the Debian wiki, which helped us to know which tasks were planned for the next two weeks. Before each meeting I sent a status update to the mailing list, so the mentors could catch up with my work. During the meetings I was not only asked to provide feedback on progress but also on mentoring itself. It proved to be very useful to be able to say where I was stuck and if the mentoring process worked out well enough (it did!).
Furthermore, I got regular, detailed and pedantic (thanks for trying to push me to perfection!) feedback on the team s mailing list. The mentors had introduced me on the upstream IRC channel in the beginning, so most of the people who are active there knew about my existence, something which also proved to be handy! Future The internship time is over, but some work remains to be done. As a new member of the AppArmor Packaging team, I am committed to make it happen: Thanks I feel much more confident after these three months, technically but also personally. Thank you: Holger Levsen and intrigeri for mentoring and encouraging me on this journey and to upstream contributors Christian Boltz, Steve Beattie (and everybody else I might forget here) for providing help and feedback. The Debian community (OPW organizers, sponsors, Alioth maintainers, planet.debian.org maintainers although my posts don t show up anymore :( -and others) has been very friendly and welcoming and I am very happy to be part of it.

6 March 2015

Eddy Petri&#537;or: Occasional Rsnapshot v1.3.1

I was writing in the previous post about Occasional Rsnapshot and how I ended up writing it.

Just before releasing v1.2.1 I realized it would make sense Semantic Versioning which, in just a few words means:
Given a version number MAJOR.MINOR.PATCH, increment the:
MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards-compatible manner, and
PATCH version when you make backwards-compatible bug fixes.
I just released Occasional Rsnapshot v1.3.1 which mainly fixed issue 4:
When deciding to do a backup for interval INT, there should be a check that the oldest snapshot in INT-1 interval is older than the threshold for the INT interval. Otherwise the INT interval will be populated with backups more frequent than desired and it is possible older backups in INT interval to completely lost.
The condition should be:
ts(oldest(INT-1)) - ts(newest(INT)) >= threshold(INT)
For example:
  • if weekly.0 is from 15th of February and daily.6 is from 17th of February, weekly should not be triggered, but
  • if weekly.0 is from 15th of February and daily.6 is from 23rd of February, weekly shall be triggered
This extra check should probably added in can_backup_interval.
It was a small bug, but it might have lead to losing important older backups because newer and more frequent backups would be pushed from hourly, interval by interval up to the yearly interval, in spite of the fact that the distance between backups wouldn't have been respecting the upper interval minimum distance in time.

There was also a small syntax bugfix, but functionally nothing has changed because bash was doing the right thing even with that small error.

If you have started using Occasional Rsnapshot, you definitely want now Occasional Rsnapshot v1.3.1. If you haven't started and you don't have backups, please start doing backups, and while you're at it, you might want to try Occasional Rsnapshot (with Rsnapshot).

5 March 2015

Ulrike Uhlig: So, how usable is AppArmor in Debian?

The short answer is: using AppArmor in Debian is fairly straight forward. Installing and activating it though is not yet a very user friendly experience. One needs to install the apparmor package, then activate it in the kernel by editing a line in the GRUB bootloader and then reboot. The procedure is explained in detail here.
We are working on fixing Debian Bug #702030 which aims at making the installation process easier for normal users by activating the module automatically as soon as one installs the apparmor package. Once you have set this up though, you re good to go. Profiles for confining processes and programs should then be activated automatically. One can verify this through the sudo aa-status command which should output something like this:
apparmor module is loaded.
30 profiles are loaded.
30 profiles are in enforce mode.
 /usr/bin/evince
 /usr/bin/evince-previewer
 /usr/bin/evince-previewer//sanitized_helper
 /usr/bin/evince-thumbnailer
 /usr/bin/evince-thumbnailer//sanitized_helper
 /usr/bin/evince//sanitized_helper
 /usr/bin/irssi
 /usr/bin/pidgin
 /usr/bin/pidgin//launchpad_integration
 /usr/bin/pidgin//sanitized_helper
 /usr/bin/tlsdate
 /usr/bin/tlsdate-helper
 /usr/bin/torbrowser-launcher
 /usr/bin/totem
 /usr/bin/totem-audio-preview
 /usr/bin/totem-video-thumbnailer
 /usr/lib/cups/backend/cups-pdf
 /usr/lib/libvirt/virt-aa-helper
 /usr/sbin/apt-cacher-ng
 /usr/sbin/cups-browsed
 /usr/sbin/cupsd
 /usr/sbin/cupsd//third_party
 /usr/sbin/libvirtd
 /usr/sbin/ntpd
 /usr/sbin/tcpdump
 /usr/sbin/tlsdated
 gst_plugin_scanner
 system_tor
0 profiles are in complain mode.
8 processes have profiles defined.
8 processes are in enforce mode.
 /usr/bin/pidgin
 /usr/bin/torbrowser-launcher
 /usr/sbin/cups-browsed 
 /usr/sbin/cupsd
 /usr/sbin/libvirtd
 system_tor
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
As of today there are only a dozen packages which ship their own profile in Debian: bind9, clamav, cups-browsed & cups-daemon, libvirt-daemon-system, mysql-5.5, lightdm, obfsproxy, sssd, tlsdate, tor, torbrowser-launcher, vidalia. In order to confine other applications, like evince, irssi, ntpd, pidgin, totem or tcpdump, one can install the apparmor-profiles-extra package. All those profiles work very well in a Debian Wheezy environment or higher, from my own experience. (Only the torbrowser-launcher profile needs to be fixed in wheezy-backports, but works well in Jessie or higher I m working on it.) If you want to confine other applications and found a profile which you want to use on your system, you can copy that profile into /etc/apparmor.d/ and then run
aa-enforce /etc/apparmor.d/myprofile
And that s pretty much it!

4 March 2015

Simon Josefsson: EdDSA and Ed25519 goes to IETF

After meeting Niels M ller at FOSDEM and learning about his Ed25519 implementation in GNU Nettle, I started working on a simple-to-implement description of Ed25519. The goal is to help implementers of various IETF (and non-IETF) protocols add support for Ed25519. As many are aware, OpenSSH and GnuPG has support for Ed25519 in recent versions, and OpenBSD since the v5.5 May 2014 release are signed with Ed25519. The paper describing EdDSA and Ed25519 is not aimed towards implementers, and does not include test vectors. I felt there were room for improvement to get wider and more accepted adoption. Our work is published in the IETF as draft-josefsson-eddsa-ed25519 and we are soliciting feedback from implementers and others. Please help us iron out the mistakes in the document, and point out what is missing. For example, what could be done to help implementers avoid side-channel leakage? I don t think the draft is the place for optimized and side-channel free implementations, and it is also not the place for a comprehensive tutorial on side-channel free programming. But maybe there is a middle ground where we can say something more than what we can do today. Ideas welcome!

2 March 2015

Wouter Verhelst: NBD 3.9

I just released NBD 3.9 When generating the changelog, I noticed that 3.8 happened two weeks shy of a year ago, which is far too long. As a result, the new release has many new features: Get it at the usual place.

Gergely Nagy: Distributed Version Control from another angle

I've been using distributed version control systems for a good while, started with TLA back in 2001, when it was still a bunch of shell scripts and was simply called "Arch". I jumped the Git bandwagon quite early too, sometime around May 2005, I believe, though it was only something to test and play with on the side: I didn't migrate my projects over yet. There are many reasons why I preferred these over the systems I used in the past (RCS and CVS; never considered Subversion an option), most of those reasons have been discussed at length over the years, and there's no reason to do so again.The reason for this post, is because I read a blog post about why Git and the other DVCSes fall short. There are a number of things in that article that I find... well, lets just say misinformed. I'll try to explain why.But here's the rub
I have wanted to have the full history while offline a grand total of maybe about six times. And this is merely going down over time as bandwidth gets ever more readily available. If you work as a field tech, or on a space station, or in a submarine or something, then okay, sure, this is a huge feature for you. But for the rest of us, I am going to assert that this is not the use case you need to optimize for when choosing a VCS.
I'm neither a field tech, nor do I work on a space station or a submarine. Yet, having the full history available locally has been crucial for my work for at least the past six years. Initially, I worked offline primarily, because my internet connection was incredibly limited (slow, expensive and unreliable), yet, I had a habit of committing small changes often. I needed a full clone to work effectively, and conveniently. Throw-away branches and all that - I needed them local, I didn't have the resources to push them online every time I wanted a merge.But now that I have fast and reliable internet, why do I need the full history on my own computer? I do, because I do much more research than I do development. The vast majority of my work between 2010 and 2012 involved reading histories, cherry picking and so on. About six hours a day, I was reading the past. Having the full history locally is a terrific aid when your job involves digging through it.Between 2013 and 2014, the amount of history reading dropped significantly, but I still worked with long-lived branches a lot, branches where history matters, where having easy and fast access to them is indispensable. I still had to find changes based on content (not commit message, branch, date or anything - content) on a daily basis. Hitting the network for each and every search would have wasted not only tremendous bandwidth and processing power on the server, but also a significant amount of time.If you do more maintenance and research, or follow a good number of branches (or forks), with the intent to merge and cherry-pick actively from them, having all the histories available locally is a huge improvement. I'm happy to pay the price of "wasting" disk space to have this available. I couldn't work otherwise. Without full, local history, the things I did for syslog-ng would not have been possible. No maintenance, none of the features I implemented, none of the fixes I found. Why? Because I rely on the project's history to aid me in learning the whys, not just the hows.Thank you, but - perhaps unlike others - I like to learn from the past, and prefer doing it efficiently.Say goodbye to blobs (or your sanity (and sometimes both))
You know what kind of external binary asset management system works really well? Subversion. Turns out it works tolerably for source control, too. Maybe you should take a look at it.
Sure, some systems are better at managing binary assets. Git - and usually DVCSes - are terrible at it. There's nothing stopping anyone from using two systems: one for source, another for data. Mind you, I am extremely happy with git-annex. I disagree with Subversion being tolerable for source control. I'd turn this assertion around instead: You know what kind of system works really well for source code? Git. Turns out it works tolerably as an external binary assest management system, too. Maybe you should take a look at it.But then, I rarely need to work with blobs, so I'm likely not the best person to disagree here.Say goodbye to sane version synchronizationOkay, so this part of the article is ill-titled, I believe. Large repositories and version synchronisation are two different, not necessarily related topic. Granted DVCSes may not be the best fit for large repositories, but neither are centralised version control systems, based on my past experience. I had the misfortune of working with a roughly 3-4Gb (checked out) project, with extensive history, managed with a centralised version control system, without local history. It was terrible (and blame was so slow and expensive that we were told never to use it, because it crashed the server). While the system allowed us to have a reasonable overview over the whole project, but it utterly failed at allowing us to work at the component level. How do I see which version of component X we had in project release A? Or the version of component Y we had two weeks ago? Or what if we have component M, made up from components J and K, and belonging to component Z? Good luck managing that in a single repository.With TLA, we had configs, which were a bit cumbersome to use, but captured most of the essential parts of managing component bundling and hierarchical versioning far better than any other solution I've seen so far. I've yet to find an acceptable solution, but I'm convinced it will not have anything to do with centralised systems. You can't capture the delicate relations there.Say what again, I dare you, I double dare you
All of the above might be almost tolerable if DVCSes were easier to use than traditional SCMs, but they aren t.
I'm sorry, but have you tried managing a decently sized project, with a large history, built from multiple components with a traditional SCM? Because that is a colossal pain in the backside.Does Git's UI suck? Yes. But so did CVS's and Subversion's. If you rely on the command-line tools, you're going to feel the pain. That is why you use the IDE integration you surely have, which makes the most common tasks a breeze. That may sound like a git-apology, but remember, we did the same with every other version control system, because smart people prefer integration, and want the computer to work for them.
And yet, when someone dares to say that Git is harder than other SCMs, they inevitably get yelled at, in what I can only assume is a combination of Stockholm syndrome and groupthink run amok by overdosing on five-hour energy buckets.
If you feel the need to understand the innards of Git - then yes, it is harder than traditional SCMs. It is distributed, and different than what you're used to. If you do not have prior knowledge though, the difference in learning the internal working of Git and - say - Subversion all that different. And if you do not care how it works down below (and I dare say, a lot of developers should not, and do not care), then there is nothing to learn, apart from your IDE integration, which ain't different from a traditional SCM's.Mind you, I found it both interesting, and beneficial to learn Git's inner workings. Not only because it allowed me to make better use of my VCS, but because it taught me a bunch of things I could make use of elsewhere. I learned nothing from traditional SCMs.Do you hear the people sing, singing the song of angry men
There is one last major argument that I hear all the time about why DVCSes are superior, which is that they somehow enable more democratic code development.
The reason Git and other tools make code contribution easier is not GitHub pull requests (although, for drive-by contributions those are pretty useful too). The reason they make contribution easier is because you have your own copy of the full history. You can create your own branches easily, you can work with it as if you were the owner. For one-off contributions, that is no real advantage, I'll give you that. It takes more effort to fork & do a PR on GitHub, than to send a patch, I'll also agree with that.But once you start working with a project for more than a few patches, when you want to work with its history too, once you have long-lived branches, the local copy quickly becomes a great asset. Once you want to submit more than a few changes, you won't have to repeat a all steps. And not only that: since you have the full history, merging upstream changes (or just cherry picking from them), rebasing your work, and many other tasks become simpler.That is what it buys you, that is why it makes contribution easier. GitHub pull requests are an icing over the cake at best (and there are a good number of cases where the icing is completely unnecessary, because it doesn't go well with the cake).In closingIn closing, I'd like to highlight that different projects have different requirements. For example, I understand why Facebook, Google and a number of others use a single, huge repository. I happen to hate that setup, and would find working in such an environment horrible. But luckily, I don't work for those companies, and they don't listen to me, either, so everyone's happy. That does not mean I would go out and tell them they're wrong to use such a centralised, single-repository setup. For their use, in their situation, for their requirements, that may be the better option. For mine, not so much.In the end, what we need to realise is that there is no single tool that will fit all needs. There are tasks where Git is a terrible choice, but that does not make it terrible for all cases. Similarly, there are cases where Subversion is superior (I assume - I didn't have the misfortune of meeting with such a case).Instead of bashing the DVCS world, and assuming the Big Company use cases are all that matter, and that everyone uses the tools the same way, accept that some use it to achieve things you didn't think of, jobs you didn't consider.

28 February 2015

Zlatan Todori : Interviews with FLOSS developers: Joey Hess

Edit: Now translated to Chinese. Thanks zhang wei! There is really hardly a better way to open a series of interviewing with developers behind Free Libre Open Source Software project, then with incredible mind such as Joey Hess. To write his contributions to Free software ecosystem, especially in Debian, would be a book by itself. His impact exceeds even his projects - people literally follow his blog posts to see what he is doing and how is he living. A hacker from cabin. If you really need to have a picture of true hacker, then Joey is the one. As this isn't a book I will just mention few projects that he has been behind - git-annex, ikiwiki, etckeeper, debian installer, parts of dpkg, debhelper, devscripts, taskel. So without further waiting here it is. Picture of Joey Hess me: Who are you? joeyh: I'm Joey -- https://joeyh.name/ me: How did you start programming? joeyh: Atari 130XE which came with BASIC and a boring word processor and not much else. No other friends had one, so the only way to get software was to type in demo programs from manual and then begin to change and write my own. So, the easy way to learn. Also some Logo in school. me: How would you now advise others to start programming? joeyh: Difficult question, it seems much harder to get an intimate understanding of things than when I started, and much harder to be motivated to program when there's so much stuff easily available. Maybe simple bare-metal systems like Arduino coupled with real-world interaction are the answer. I've recently been mentoring my nephew who is learning python and Python the Hard Way has gotten him far impressively fast. me: Setup of your development machine? joeyh: Lenovo laptop de-spywared with Debian unstable, xmonad, xfce, vim. me: Your thoughts on Purism (the open hardware laptop initiative that got recently funded on CrowdSupply)? joeyh: I don't know much about that one, but it seems that consumer level hardware has gotten so low quality, and so closed and untrustworthy that it makes sense to either build alternatives that are open, or pick out, as a community, the stuff we can adapt to our needs and concentrate on it. Several projects are trying, I hope they succeed. me: How do you see future of Debian development? joeyh: Well, I've mostly stopped worrying about it. If you look back at my presentations at the past 2 or 3 DebConfs, you'll find my best thoughts on the matter. me: You retired as Debian developer - do you intend sometime soon to come back and/or do you plan to join some other communities? joeyh: It would be glorious to come back, wouldn't it? But I don't think I will. Can't step in the same river twice, and all. Instead, Debian will probably have to put up with me as an annoying upstream author who doesn't ship tarballs, but does ship debian/ directories, and as a bug reporter who enjoys reporting amusing bugs like -0 NaN. I seem to have more time to spend in other online communities since I left Debian, but in a more diffuse way. Maybe that's just what it's like, to be involved in Free Software but not in the embrace of a big project like Debian. me: Some memorable moments from Debian conferences? joeyh: There are so many! Picnicing on berries and tamales at the Portland farmer's market right outside the venue; rainbows and bonfire in Switzerland after crazy busy days; impromptu pipe organ repair in a weird night venue in Edinburgh; walking through Porto Alegre at night with Ian Murdock and how humble he was about what he'd started; hacking all night in Spain; failing to sleep through midnight sun and incessent partying Finland; hanging out in the hotel lobby in Atlanta where we designed Build-Depends. me: Are you a gamer? Valve Steam games are offered for free to Debian Developers - do you use steam and play Valve games? joeyh: I've played through Half Life and Portal, but nethack has claimed more of my time. I mostly enjoy short, indie games, or games that tell us something new about the medium of games, A recent favorite was A Dark Room. But really, I have more pure fun playing real world Tabletop games with friends, like Carcassanne Discovery and Hive. In March, I am going to try to write a roguelike game in one week, in Haskell, for the Seven Day Roguelike Challenge and I'll be blogging about my progress daily. me: You are nowdays a Haskell hacker (git-annex) - what would you like to say about this language and how does it compare to Python, C, JavaScript, Ruby and Perl? joeyh: Not just git-annex; all my current projects are written in Haskell. I think it's amazing how much we expect programmers to keep in their heads while writing code. Is that buffer going to overflow? Is changing the value of that global variable going to break some other part of the code? Is that input sanitized yet? Did that interface change? Haskell solves some of these outright, but more, it makes you start noticing this kind of pervasive issue, and it provides ways to completely eliminate a class of problems from your code. For example http://joeyh.name/blog/entry/making_propellor_safer_with_GADTs_and_type_families/. The class of bugs I avoided there had never affected my code even once, but it was still worth preventing that whole class of bugs, so I don't have to worry about them ever again. me: Would you suggest Haskell as first language to learn especially for those that have an itch for mathematics? joeyh: I think that can work well. Or it can go other the way -- I had an affinity to mathematics when I was young, but it got knocked out of me in the way that happens to many people, and languages like perl and C don't do much to make you want to learn more about higher-order math. I've been picking up a bit more here and there via Haskell. me: How do you compare your productivity in Haskell compared to your Perl days? joeyh: It's very different; I'm a very different programmer now. I probably would bang out quick hacks more quickly when I was writing Perl. But, they tended to stay quick hacks. Now, I might take a little longer to get there, but the code seems a lot more solid, while also being more malleable to turn into larger or different programs. I'm also a lot more drawn toward writing software libraries. me: Can you describe your philosophy of life (you live in cabin, in forest, using a lot of solar power - many people are intrigued (including myself) what drives you towards that kind of life and how does it impact your overall quality of life and happiness. Looking the todays modern predator capitalistic society, in which you could easily earn more then $10.000 a month, you seem to be an anarchist and very humble human)? joeyh: I want to build worthwhile things that might last. Which is super hard in the world of software, both because it's hard to think far ahead at all, and because most jobs don't emphasize that kind of real value. I've been lucky and bootstrapped up to a point where I've been able to work full time on free software for years, and I'm willing to forgo a lot to continue that. Living in the woods without modern conveniences is great, because it's quiet and you can think as much as you like; the internet is just as close as it is anywhere else (maybe a bit slower); and when you've spent too much time quietly thinking you'll need to go chop wood, or haul water, or jump in the river to cool off, depending on the season. (Humble? Like most programmers, I am internally a flaming tower of ego...)
Vote on Hacker News

27 February 2015

Dirk Eddelbuettel: RcppArmadillo 0.4.650.1.1 (and also 0.4.650.2.0)

A new Armadillo release 4.650.1 was released by Conrad a few days ago. Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. It turned out that this release had one shortcoming with respect to the C++11 RNG initializations in the R use case (where we need to protect the users from the C++98 RNG deemed unsuitable by the CRAN gatekeepers). And this lead to upstream release 4.650.1 which we wrapped into RcppArmadillo 0.4.650.1.1. As before this, was tested against all 107 reverse dependencies of RcppArmadillo on the CRAN repo. This version is now on CRAN, and was just uploaded to Debian. Its changes are summarized below based on the NEWS.Rd file.
Changes in RcppArmadillo version 0.4.650.1.1 (2015-02-25)
  • Upgraded to Armadillo release Version 4.650.1 ("Intravenous Caffeine Injector")
    • added randg() for generating random values from gamma distributions (C++11 only)
    • added .head_rows() and .tail_rows() to submatrix views
    • added .head_cols() and .tail_cols() to submatrix views
    • expanded eigs_sym() to optionally calculate eigenvalues with smallest/largest algebraic values fixes for handling of sparse matrices
  • Applied small correction to main header file to set up C++11 RNG whether or not the alternate RNG (based on R, our default) is used
Now, it turns out that another small fix was needed for the corner case of a submatrix within a submatrix, ie V.subvec(1,10).tail(5). I decided not to re-release this to CRAN given the CRAN Repository Policy preference for releases no more than every 1 2 months . But fear not, for we now have drat. I created a drat package repository in the RcppCore account (to not put a larger package into my main drat repository often used via a fork to initialize a drat). So now with these two simple commands
## if needed, first install 'drat' via:   install.packages("drat")
drat:::add("RcppCore")
update.packages()
you will get the newest RcppArmadillo via this drat package repository. And course install.packages("RcppArmadillo") would also work, but takes longer to type :) Lastly, courtesy of CRANberries, there is also a diffstat report for the most recent CRAN release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

26 February 2015

Wouter Verhelst: Dear non-Belgian web developer,

Localization in the web context is hard, I know. To make things easier, it may seem like a good idea to use GeoIP to detect what country an IP is coming from and default your localization based on that. While I disagree with that premise, this blog post isn't about that. Instead, it's about the fact that most of you get something wrong about this little country. I know, I know. If you're not from here, it's difficult to understand. But please get this through your head: Belgium is not a French-speaking country. That is, not entirely. Yes, there is a large group of French-speaking people who live here. Mostly in the south. But if you check the numbers, you'll find that there are, in fact, more people in Belgium who speak Dutch rather than French. Not by a very wide margin, mind you, but still by a wide enough margin to be significant. Wikipedia claims the split is 59%/41% Dutch/French; I don't know how accurate those numbers are, but they don't seem too wrong. So please, pretty please, with sugar on top: next time you're going to do a localized website, don't assume my French is better than my English. And if you (incorrectly) do, then at the very least make it painfully obvious to me where the "switch the interface to a different language" option in your website is. Because while it's annoying to be greeted in a language that I'm not very good at, it's even more annoying to not be able to find out how to get the correctly-localized version. Thanks.

25 February 2015

Eddy Petri&#537;or: Occasional Rsnapshot v1.3.0

It is almost exactly 1 year and a half since I came up with the idea of having a way of making backups using Rsnapshot automatically triggered by my laptop when I have the backup media connected to my laptop. This could mean connecting a USB drive directly to the laptop or mounting a NFS/sshfs share in my home network. Today I tagged Occasional Rsnapshot the v1.3.0 version, the first released version that makes sure even when you connect your backup media occasionally, your Rsnapshot backups are done if and when it makes sense to do it, according to the rsnapshot.conf file and the status of the existing backups on the backup media.

Quoting from the README, here is what Occasional Rsnapshot does:

This is a tool that allows automatic backups using rsnapshot when the external backup drive or remote backup media is connected.

Although the ideal setup would be to have periodic backups on a system that is always online, this is not always possible. But when the connection is done, the backup should start fairly quickly and should respect the daily/weekly/... schedules of rsnapshot so that it accurately represents history.

In other words, if you backup to an external drive or to some network/internet connected storage that you don't expect to have always connected (which is is case with laptops) you can use occasional_rsnapshot to make sure your data is backed up when the backup storage is connected.

occasional_rsnapshot is appropriate for:
  • laptops backing up on:
    • a NAS on the home LAN or
    • a remote or an internet hosted storage location
  • systems making backups online (storage mounted locally somehow)
  • systems doing backups on an external drive that is not always connected to the system
The only caveat is that all of these must be mounted in the local file system tree somehow by any arbitrary tool, occasional_rsnapshot or rsnapshot do not care, as long as the files are mounted.

So if you find yourself in a simillar situation, this script might help you to easily do backups in spite of the occasional availability of the backup media, instead of no backups at all. You can even trigger backups semi-automatically when you remember to or decide is time to backup, by simply pulging in your USB backup HDD.

But how did I end up here, you might ask?

In December 2012 I was asking about suggestions for backup solutions that would work for my very modest setup with Linux and Windows so I can backup my and my wife's system without worrying about loss of data.

One month later I was explaining my concept of a backup solution that would not trust the backup server, and leave to the clients as much as possible the decision to start the backup at their desired time. I was also pondering on the problems I might encounter.

From a security PoV, what I wanted was that:
  1. clients would be isolated from each other
  2. even in the case of a server compromise:
      • the data would not be accessible since they would be already encrypted before leaving the client
      • the clients could not be compromised


    The general concept was sane and supplemental security measures such as port knocking and initiation of backups only during specific time frames could be added.

    The problem I ran to was that when I set up this in my home network a sigle backup cycle would take more than a day, due to the fact that I wanted to do backup of all of my data and my server was a humble Linksys NSLU2 with a 3TB storage attached on USB.

    Even when the initial copy was done by attaching the USB media directly to the laptop, so the backup would only copy changed data, the backup with the HDD attached to the NSLU2 was not finished even after more than 6 hours.

    The bottleneck was the CPU speed and the USB speed. I tried even mounting the storage media over sshfs so the tiny xscale processor in the NSLU2 would not be bothered by any of the rsync computation. This proved to an exercise in futility, any attempt to put the NSLU2 anywhere in the loop resulted in an unacceptable and impractically long backup time.

    All these attempts, of course, took time, but that meant that I was aware I still didn't have appropriate backups and I wasn't getting closer to the desired result.

    So this brings us August 2013, when I realized I was trying to manually trigger Rsnapshot backups from time to time, but having to do all sorts of mental gymnastics and manual listing to evaluate if I needed to do monthly, weekly and daily backups or if weekly and daily was due.

    This had to stop.
    Triggering a backup should happen automatically as soon as the backup media is available, without any intervention from the user.
    I said.

    Then I came up with the basic concept for Occasional Rsnapshot: a very silent script that would be called from cron every 5 minutes, would check if the backup media is mounted, if is not, exit silently to not generate all sorts of noise in cron emails, but if mounted, compute which backup intervals should be triggered, and trigger them, if the appropriate amount of time passed since the most recent backup in that backup interval.

    Occasional Rsnapshot version v1.3.0 is the 8th and most recent release of the script. Even if I used Occasional Rsnapshot since the day 1, v1.3.0 is the first one I can recommend to others, without fearing they might lose data due to it.

    The backup media can be anything, starting from your regular USB mounted HDD, your sshfs mounted backup partition on the home NAS server to even a remote storage such as Amazon S3 online storage, and there are even brief instructions on how to do encrypted backups for the cases where you don't trust the remote storage.

    So if you think you might find anything that I described remotely interesting, I recommend downloading the latest release of Occasional Rsnapshot, go through the README and trying it out.

    Feedback and bug reports are welcome.
    Patches are welcomed with a 'thank you'.
    Pull requests are eagerly waited for :) .

    21 February 2015

    Dominique Dumont: Performance improvement for cme check dpkg

    Hello Thanks to Devel::NYTProf, I ve realized that Module::CoreList was used in a not optimal way (to say the least) in Config::Model::Dpkg::Dependency when checking the dependency between Perl packages. (Note that only Perl packages with many dependencies were affected by this lack of performance) After a rework, the performance are much better. Here s an example comparing check time before and after the modification of libconfig-model-dpkg-perl. With libconfig-model-dpkg-perl 2.059:
    $ time cme check dpkg
    Using Dpkg
    loading data
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    checking data
    check done
    real 0m10.235s
    user 0m10.136s
    sys 0m0.088s With libconfig-model-dpkg-perl 2.060:
    $ time cme check dpkg
    Using Dpkg
    loading data
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    checking data
    check done
    real 0m1.565s
    user 0m1.468s
    sys 0m0.092s All in all, a 8x performance improvement on the dependency check. Note that, due to the freeze, the new version of libconfig-model-dpkg-perl is available only in experimental. All the best
    Tagged: Config::Model, debian, dpkg, package

    16 February 2015

    Julien Danjou: Hacking Python AST: checking methods declaration

    A few months ago, I wrote the definitive guide about Python method declaration, which had quite a good success. I still fight every day in OpenStack to have the developers declare their methods correctly in the patches they submit. Automation plan The thing is, I really dislike doing the same things over and over again. Furthermore, I'm not perfect either, and I miss a lot of these kind of problems in the reviews I made. So I decided to replace me by a program a more scalable and less error-prone version of my brain. In OpenStack, we rely on flake8 to do static analysis of our Python code in order to spot common programming mistakes. But we are really pedantic, so we wrote some extra hacking rules that we enforce on our code. To that end, we wrote a flake8 extension called hacking. I really like these rules, I even recommend to apply them in your own project. Though I might be biased or victim of Stockholm syndrome. Your call. Anyway, it's pretty clear that I need to add a check for method declaration in hacking. Let's write a flake8 extension! Typical error The typical error I spot is the following:
    class Foo(object):
    # self is not used, the method does not need
    # to be bound, it should be declared static
    def bar(self, a, b, c):
    return a + b - c

    That would be the correct version:
    class Foo(object):
    @staticmethod
    def bar(a, b, c):
    return a + b - c

    This kind of mistake is not a show-stopper. It's just not optimized. Why you have to manually declare static or class methods might be a language issue, but I don't want to debate about Python misfeatures or design flaws. Strategy We could probably use some big magical regular expression to catch this problem. flake8 is based on the pep8 tool, which can do a line by line analysis of the code. But this method would make it very hard and error prone to detect this pattern. Though it's also possible to do an AST based analysis on on a per-file basis with pep8. So that's the method I pick as it's the most solid. AST analysis I won't dive deeply into Python AST and how it works. You can find plenty of sources on the Internet, and I even talk about it a bit in my book The Hacker's Guide to Python. To check correctly if all the methods in a Python file are correctly declared, we need to do the following: Flake8 plugin In order to register a new plugin in flake8 via hacking, we just need to add an entry in setup.cfg:
    [entry_points]
    flake8.extension =
    [ ]
    H904 = hacking.checks.other:StaticmethodChecker
    H905 = hacking.checks.other:StaticmethodChecker

    We register 2 hacking codes here. As you will notice later, we are actually going to add an extra check in our code for the same price. Stay tuned. The next step is to write the actual plugin. Since we are using an AST based check, the plugin needs to be a class following a certain signature:
    @core.flake8ext
    class StaticmethodChecker(object):
    def __init__(self, tree, filename):
    self.tree = tree

    def run(self):
    pass

    So far, so good and pretty easy. We store the tree locally, then we just need to use it in run() and yield the problem we discover following pep8 expected signature, which is a tuple of (lineno, col_offset, error_string, code). This AST is made for walking The ast module provides the walk function, that allow to iterate easily on a tree. We'll use that to run through the AST. First, let's write a loop that ignores the statement that are not class definition.
    @core.flake8ext
    class StaticmethodChecker(object):
    def __init__(self, tree, filename):
    self.tree = tree

    def run(self):
    for stmt in ast.walk(self.tree):
    # Ignore non-class
    if not isinstance(stmt, ast.ClassDef):
    continue

    We still don't check for anything, but we know how to ignore statement that are not class definitions. The next step need to be to ignore what is not function definition. We just iterate over the attributes of the class definition.
    for stmt in ast.walk(self.tree):
    # Ignore non-class
    if not isinstance(stmt, ast.ClassDef):
    continue
    # If it's a class, iterate over its body member to find methods
    for body_item in stmt.body:
    # Not a method, skip
    if not isinstance(body_item, ast.FunctionDef):
    continue

    We're all set for checking the method, which is body_item. First, we need to check if it's already declared as static. If so, we don't have to do any further check and we can bail out.
    for stmt in ast.walk(self.tree):
    # Ignore non-class
    if not isinstance(stmt, ast.ClassDef):
    continue
    # If it's a class, iterate over its body member to find methods
    for body_item in stmt.body:
    # Not a method, skip
    if not isinstance(body_item, ast.FunctionDef):
    continue
    # Check that it has a decorator
    for decorator in body_item.decorator_list:
    if (isinstance(decorator, ast.Name)
    and decorator.id == 'staticmethod'):
    # It's a static function, it's OK
    break
    else:
    # Function is not static, we do nothing for now
    pass

    Note that we use the special for/else form of Python, where the else is evaluated unless we used break to exit the for loop.
    for stmt in ast.walk(self.tree):
    # Ignore non-class
    if not isinstance(stmt, ast.ClassDef):
    continue
    # If it's a class, iterate over its body member to find methods
    for body_item in stmt.body:
    # Not a method, skip
    if not isinstance(body_item, ast.FunctionDef):
    continue
    # Check that it has a decorator
    for decorator in body_item.decorator_list:
    if (isinstance(decorator, ast.Name)
    and decorator.id == 'staticmethod'):
    # It's a static function, it's OK
    break
    else:
    try:
    first_arg = body_item.args.args[0]
    except IndexError:
    yield (
    body_item.lineno,
    body_item.col_offset,
    "H905: method misses first argument",
    "H905",
    )
    # Check next method
    continue

    We finally added some check! We grab the first argument from the method signature. Unless it fails, and in that case, we know there's a problem: you can't have a bound method without the self argument, therefore we raise the H905 code to signal a method that misses its first argument. Now you know why we registered this second pep8 code along with H904 in setup.cfg. We have here a good opportunity to kill two birds with one stone. The next step is to check if that first argument is used in the code of the method.
    for stmt in ast.walk(self.tree):
    # Ignore non-class
    if not isinstance(stmt, ast.ClassDef):
    continue
    # If it's a class, iterate over its body member to find methods
    for body_item in stmt.body:
    # Not a method, skip
    if not isinstance(body_item, ast.FunctionDef):
    continue
    # Check that it has a decorator
    for decorator in body_item.decorator_list:
    if (isinstance(decorator, ast.Name)
    and decorator.id == 'staticmethod'):
    # It's a static function, it's OK
    break
    else:
    try:
    first_arg = body_item.args.args[0]
    except IndexError:
    yield (
    body_item.lineno,
    body_item.col_offset,
    "H905: method misses first argument",
    "H905",
    )
    # Check next method
    continue
    for func_stmt in ast.walk(body_item):
    if six.PY3:
    if (isinstance(func_stmt, ast.Name)
    and first_arg.arg == func_stmt.id):
    # The first argument is used, it's OK
    break
    else:
    if (func_stmt != first_arg
    and isinstance(func_stmt, ast.Name)
    and func_stmt.id == first_arg.id):
    # The first argument is used, it's OK
    break
    else:
    yield (
    body_item.lineno,
    body_item.col_offset,
    "H904: method should be declared static",
    "H904",
    )

    To that end, we iterate using ast.walk again and we look for the use of the same variable named (usually self, but if could be anything, like cls for @classmethod) in the body of the function. If not found, we finally yield the H904 error code. Otherwise, we're good. Conclusion I've submitted this patch to hacking, and, finger crossed, it might be merged one day. If it's not I'll create a new Python package with that check for flake8. The actual submitted code is a bit more complex to take into account the use of abc module and include some tests. As you may have notice, the code walks over the module AST definition several times. There might be a couple of optimization to browse the AST in only one pass, but I'm not sure it's worth it considering the actual usage of the tool. I'll let that as an exercise for the reader interested in contributing to OpenStack. Happy hacking!
    The Hacker's Guide to Python
    A book I wrote talking about designing Python applications, state of the art, advice to apply when building your application, various Python tips, etc. Interested? Check it out.

    13 February 2015

    Daniel Leidert: Motion picture capturing: Debian + motion + Logitech C910 - part II

    In my recent attempt to setup a motion detection camera I was disappointed, that my camera, which should be able to record with 30 fps in 720p mode only reached 10 fps using the software motion. Now I got a bit further. This seems to be an issue with the format used by motion. I've check the output of v4l2-ctl ...
    $ v4l2-ctl -d /dev/video1 --list-formats-ext
    [..]
    ioctl: VIDIOC_ENUM_FMT
    Index : 0
    Type : Video Capture
    Pixel Format: 'YUYV'
    Name : YUV 4:2:2 (YUYV)
    [..]
    Size: Discrete 1280x720
    Interval: Discrete 0.100s (10.000 fps)
    Interval: Discrete 0.133s (7.500 fps)
    Interval: Discrete 0.200s (5.000 fps)
    [..]

    Index : 1
    Type : Video Capture
    Pixel Format: 'MJPG' (compressed)
    Name : MJPEG
    [..]
    Size: Discrete 1280x720
    Interval: Discrete 0.033s (30.000 fps)
    Interval: Discrete 0.042s (24.000 fps)
    Interval: Discrete 0.050s (20.000 fps)
    Interval: Discrete 0.067s (15.000 fps)
    Interval: Discrete 0.100s (10.000 fps)
    Interval: Discrete 0.133s (7.500 fps)
    Interval: Discrete 0.200s (5.000 fps)
    [..]
    ... and motion:
    $ motion
    [..]
    [1] [NTC] [VID] v4l2_set_pix_format: Config palette index 17 (YU12) doesn't work.
    [1] [NTC] [VID] v4l2_set_pix_format: Supported palettes:
    [1] [NTC] [VID] v4l2_set_pix_format: (0) YUYV (YUV 4:2:2 (YUYV))
    [1] [NTC] [VID] v4l2_set_pix_format: 0 - YUV 4:2:2 (YUYV) (compressed : 0) (0x56595559)
    [1] [NTC] [VID] v4l2_set_pix_format: (1) MJPG (MJPEG)
    [1] [NTC] [VID] v4l2_set_pix_format: 1 - MJPEG (compressed : 1) (0x47504a4d)
    [1] [NTC] [VID] v4l2_set_pix_format Selected palette YUYV
    [1] [NTC] [VID] v4l2_do_set_pix_format: Testing palette YUYV (1280x720)
    [1] [NTC] [VID] v4l2_do_set_pix_format: Using palette YUYV (1280x720) bytesperlines 2560 sizeimage 1843200 colorspace 00000008
    [..]
    Ok, so both formats YUYV and MJPG are supported and recognized and I can choose both via the v4l2palette configuration variable, citing motion.conf:
    # v4l2_palette allows to choose preferable palette to be use by motion
    # to capture from those supported by your videodevice. (default: 17)
    # E.g. if your videodevice supports both V4L2_PIX_FMT_SBGGR8 and
    # V4L2_PIX_FMT_MJPEG then motion will by default use V4L2_PIX_FMT_MJPEG.
    # Setting v4l2_palette to 2 forces motion to use V4L2_PIX_FMT_SBGGR8
    # instead.
    #
    # Values :
    # V4L2_PIX_FMT_SN9C10X : 0 'S910'
    # V4L2_PIX_FMT_SBGGR16 : 1 'BYR2'
    # V4L2_PIX_FMT_SBGGR8 : 2 'BA81'
    # V4L2_PIX_FMT_SPCA561 : 3 'S561'
    # V4L2_PIX_FMT_SGBRG8 : 4 'GBRG'
    # V4L2_PIX_FMT_SGRBG8 : 5 'GRBG'
    # V4L2_PIX_FMT_PAC207 : 6 'P207'
    # V4L2_PIX_FMT_PJPG : 7 'PJPG'
    # V4L2_PIX_FMT_MJPEG : 8 'MJPEG'
    # V4L2_PIX_FMT_JPEG : 9 'JPEG'
    # V4L2_PIX_FMT_RGB24 : 10 'RGB3'
    # V4L2_PIX_FMT_SPCA501 : 11 'S501'
    # V4L2_PIX_FMT_SPCA505 : 12 'S505'
    # V4L2_PIX_FMT_SPCA508 : 13 'S508'
    # V4L2_PIX_FMT_UYVY : 14 'UYVY'
    # V4L2_PIX_FMT_YUYV : 15 'YUYV'
    # V4L2_PIX_FMT_YUV422P : 16 '422P'
    # V4L2_PIX_FMT_YUV420 : 17 'YU12'
    #
    v4l2_palette 17
    Now motion uses YUYV as default mode as shown by its output. So it seems that all I have to do is to choose MJPEG in my motion.conf:
    v4l2_palette 8
    Testing again ...
    $ motion
    [..]
    [1] [NTC] [VID] vid_v4lx_start: Using V4L2
    [1] [NTC] [ALL] image_ring_resize: Resizing pre_capture buffer to 1 items
    [1] [NTC] [VID] v4l2_set_control: setting control "Brightness" to 25 (ret 0 )
    Corrupt JPEG data: 5 extraneous bytes before marker 0xd6
    [1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
    [1] [NTC] [VID] v4l2_set_control: setting control "Brightness" to 14 (ret 0 )
    Corrupt JPEG data: 1 extraneous bytes before marker 0xd5
    [1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
    [1] [NTC] [VID] v4l2_set_control: setting control "Brightness" to 36 (ret 0 )
    Corrupt JPEG data: 3 extraneous bytes before marker 0xd2
    [1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
    [1] [NTC] [VID] v4l2_set_control: setting control "Brightness" to 58 (ret 0 )
    Corrupt JPEG data: 1 extraneous bytes before marker 0xd7
    [1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
    [1] [NTC] [VID] v4l2_set_control: setting control "Brightness" to 80 (ret 0 )
    Corrupt JPEG data: 4 extraneous bytes before marker 0xd7
    [1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
    [1] [ERR] [ALL] motion_init: Error capturing first image
    [1] [NTC] [ALL] image_ring_resize: Resizing pre_capture buffer to 16 items
    Corrupt JPEG data: 4 extraneous bytes before marker 0xd1
    [1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
    Corrupt JPEG data: 11 extraneous bytes before marker 0xd1
    [1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
    Corrupt JPEG data: 3 extraneous bytes before marker 0xd4
    [1] [CRT] [VID] mjpegtoyuv420p: Corrupt image ... continue
    Corrupt JPEG data: 7 extraneous bytes before marker 0xd1
    [..]
    ... and another issue is turning up :( The output above goes on and on and on and there is no video capturing. So accordingly to $searchengine the above happens to a lot of people. I just found one often suggested fix: pre-load v4l2convert.so from libv4l-0:
    $ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libv4l/v4l2convert.so motion
    But the problem persists and I'm out of ideas :( So atm it lokks like I cannot use the MJPEG format and don't get 30 fps at 1280x720 pixels. During writing I then discovered a solution by good old trial-and-error: Leaving the v4l2_palette variable at its default value 17 (YU12) and pre-loading v4l2convert.so makes use of YU12 and the framerate at least raises to 24 fps:
    $ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libv4lg/v4l2convert.so motion
    [..]
    [1] [NTC] [VID] v4l2_do_set_pix_format: Testing palette YU12 (1280x720)
    [1] [NTC] [VID] v4l2_do_set_pix_format: Using palette YU12 (1280x720) bytesperlines 1280 sizeimage 1382400 colorspace 00000008
    [..]
    [1] [NTC] [EVT] event_new_video FPS 24
    [..]
    Finally! :) The results are nice. It would maybe even be a good idea to limit the framerate a bit, to e.g. 20. So that is a tested configuration for the Logitech C910 running at a resolution of 1280x720 pixels:
    v4l2_palette 17
    width 1280
    height 720
    framerate 20
    minimum_frame_time 0
    pre_capture 10 # 0,5 seconds pre-recording
    post_capture 50 # 2,5 seconds after-recording
    auto_brightness on
    ffmpeg_variable_bitrate 2 # best quality
    Now all this made me curious, which framerate is possible at a resolution of 1920x1080 pixels now and how the results look like. Although I get 24 fps too, the resulting movie suffers of jumps every few frames. So here I got pretty good results with a more conservative setting. By increasing framerate - tested up to 15 fps with good results - pre_capture needed to be decreased accordingly to values between 1..3 to minimize jumps:
    v4l2_palette 17
    width 1920
    height 1080
    framerate 12
    minimum_frame_time 0
    pre_capture 6 # 0,5 seconds pre-recording
    post_capture 30 # 2,5 seconds after-recording
    auto_brightness on
    ffmpeg_variable_bitrate 2 # best quality
    Both configurations lead to satisfying results. Of course the latter will easily fill your hardrive :) TODOI guess, the results can be optimzed further by playing around with ffmpeg_bps and ffmpeg_variable_bitrate. Maybe then it is possible to record without jumps at higher framerates too(?). I also didn't test the various norm settings (PAL, NTSC, etc.).

    11 February 2015

    Daniel Leidert: Issues with Server4You vServer running Debian Stable (Wheezy)

    I recently acquired a vServer hosted by Server4You and decided to install a Debian Wheezy image. Usually I boot any device in backup mode and first install a fresh Debian copy using debootstrap over the provided image, to have a clean system. In this case I did not and I came across a few glitches I want to talk about. So hopefully, if you are running the same system image, it saves you some time to figure out, why the h*ll some things don't work as expected :) Cron jobs not runningI installed unattended-upgrades and adjusted all configuration files to enable unattended upgrades. But I never received any mail about an update although looking at the system, I saw updates waiting. I checked with
    # run-parts --list /etc/cron.daily
    and apt was not listed although /etc/cron.daily/apt was there. After spending some time to figure out, what was going on, I found the rather simple cause: Several scripts were missing the executable bit, thus did not run. So it seems, for whatever reason, the image authors have tempered with file permissions and of course, not by using dpkg-statoverride :( It was easy to fix the file permissions for everything beyond /etc/cron*, but that still leaves a very bad feeling, that there are more files that have been tempered with! I'm not speaking about customizations. That are easy to find using debsums. I'm speaking about file permissions and ownership.Now there seems no easy way to either check for changed permissions or ownership. The only solution I found is to get a list of all installed packages on the system, install them into a chroot environment and get all permission and ownership information from this very fresh system. Then compare file permissions/ownership of the installed system with this list. Not fun. init from testing / upstart on holdToday I've discovered, that apt-get wanted to update the init package. Of course I was curious, why unattended-upgrades didn't yet already do so. Turns out, init is only in testing/unstable and essential there. I purged it, but apt-get keeps bugging me to update/install this package. I really began to wonder, what is going on here, because this is a plain stable system:
    • no sources listed for backports, volatile, multimedia etc.
    • sources listed for testing and unstable
    • only packages from stable/stable-updates installed
    • sets APT::Default-Release "stable";
    First I checked with aptitude:
    # aptitude why init
    Unable to find a reason to install init.
    Ok, so why:
    # apt-get dist-upgrade -u
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    Calculating upgrade... Done
    The following NEW packages will be installed:
    init
    0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
    Need to get 0 B/4674 B of archives.
    After this operation, 29.7 kB of additional disk space will be used.
    Do you want to continue [Y/n]?
    JFTR: I see a stable system bugging me to install systemd for no obvious reason. The issue might be similar! I'm still investigating. (not reproducible anymore)Now I tried to debug this:
    # apt-get -o  Debug::pkgProblemResolver="true" dist-upgrade -u
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    Calculating upgrade... Starting
    Starting 2
    Investigating (0) upstart [ amd64 ] < 1.6.1-1 1.11-5 > ( admin )
    Broken upstart:amd64 Conflicts on sysvinit [ amd64 ] < none -> 2.88dsf-41+deb7u1 2.88dsf-58 > ( admin )
    Conflicts//Breaks against version 2.88dsf-58 for sysvinit but that is not InstVer, ignoring
    Considering sysvinit:amd64 5102 as a solution to upstart:amd64 10102
    Added sysvinit:amd64 to the remove list
    Fixing upstart:amd64 via keep of sysvinit:amd64
    Done
    Done
    The following NEW packages will be installed:
    init
    0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
    Need to get 0 B/4674 B of archives.
    After this operation, 29.7 kB of additional disk space will be used.
    Do you want to continue [Y/n]?
    Eh, upstart?
    # apt-cache policy upstart
    upstart:
    Installed: 1.6.1-1
    Candidate: 1.6.1-1
    Version table:
    1.11-5 0
    500 http://ftp.de.debian.org/debian/ testing/main amd64 Packages
    500 http://ftp.de.debian.org/debian/ sid/main amd64 Packages
    *** 1.6.1-1 0
    990 http://ftp.de.debian.org/debian/ stable/main amd64 Packages
    100 /var/lib/dpkg/status
    # dpkg -l upstart
    Desired=Unknown/Install/Remove/Purge/Hold
    Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
    / Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
    / Name Version Architecture Description
    +++-=============================-===================-===================-===============================================================
    hi upstart 1.6.1-1 amd64 event-based init daemon
    Ok, at least one package is at hold. This is another questionable customization, but in case easy to fix. But I still don't understand apt-get and the difference to aptitude behaviour? Can someone please enlighten me? Customized filesThis isn't really an issue, but just for completion: several files have been customized. debsums easily shows which ones:
    # debsums -ac
    I don't have the original list anymore - please check yourself

    10 February 2015

    John Goerzen: Has modern Linux lost its way? (Some thoughts on jessie)

    For years, I used to run Debian sid (unstable) on all my personal machines. Laptops, workstations, sometimes even my personal servers years ago ran sid. Sid was, as its name implies, unstable. Sometimes things broke. But it wasn t a big deal, because I could always get in there and fix it fairly quickly, whatever it was. It was the price I paid for the latest and greatest. For the last number of months, I ve dealt with a small but annoying issue in jessie: None of Nautilus, Thunar, or digikam (yes, that represents Gnome, XFCE, and KDE) can mount USB drives I plug in anymore. I just get Not authorized to perform operation. I can, of course, still mount -o uid=1000 /dev/sdc1 /mnt, but I miss the convenience of doing it this way. One jessie system I switched to systemd specifically to get around this problem. It worked, but I don t know why. I haven t had the time to switch my workstation, and frankly I am concerned about it. Here s the crux of the issue: I don t even know where to start looking. I ve googled this issue, and found all sorts of answers pointing to polkit, or dbus, or systemd-shim, or cgmanager, or lightdm, or XFCE, or I found a bug report of this exact problem Debian #760281, but it s marked fixed, and nobody replied to my comment that I m still seeing it. Nowhere is it documented that a Digikam mounting issue should have me looking in polkit, let alone cgmanager. And even once I find those packages, their documentation suffers from Bad Unix Documentation Disease: talking about the nitty-gritty weeds view of something, without bothering to put it in context. Here is the mystifying heading for the cgmanager(8) manpage:
    cgmanager is a daemon to manage cgroups. Programs and users can make D-Bus requests to administer cgroups over which they have privilege. To ensure that users may not exceed their privilege in manipulating cgroups, the cgroup manager accepts regular D-Bus requests only from tasks within its own process-id and user namespaces. For tasks in private namespaces (such as containers), SCM-enhanced D-Bus calls are available. Using these manually is not recommended. Rather, each container is advised to run a cgproxy, which will forward plain D-Bus requests as SCM-enhanced D-Bus requests to the host cgmanager.
    That s about as comprehensible as Vogon poetry to me. How is cgmanager started? What does SCM-enhanced mean? And I even know what a cgroup is. This has been going on for months, which has me also wondering: is it only me? (Google certainly suggests it s not, and there are plenty of hits for this exact problem with many distros, and some truly terrible advice out there to boot.) And if not, why is something so basic and obvious festering for so long? Have we built something that s too complex to understand and debug? This is, in my mind, orthogonal to the systemd question. I used to be able to say Linux was clean, logical, well put-together, and organized. I can t really say this anymore. Users and groups are not really determinitive for permissions, now that we have things like polkit running around. (Yes, by the way, I am a member of plugdev.) Error messages are unhelpful (WHY was I not authorized?) and logs are nowhere to be found. Traditionally, one could twiddle who could mount devices via /etc/fstab lines and perhaps some sudo rules. Granted, you had to know where to look, but when you did, it was simple; only two pieces to fit together. I ve even spent time figuring out where to look and STILL have no idea what to do. systemd may help with some of this, and may hurt with some of it; but I see the problem more of an attitude of desktop environments to add features fast without really thinking of the implications. There is something to be said for slower progress if the result is higher quality. Then as I was writing this, of course, my laptop started insisting that it needed the root password to suspend when I close the lid. And it s running systemd. There s another quagmire Update: Part 2 with some reactions to this and further thoughts is now available.

    Dirk Eddelbuettel: RPushbullet 0.2.0

    A new releases of the RPushbullet package (interfacing the neat Pushbullet service) arrived on CRAN today. It brings several weeks of extensions, corrections and cleanups---with key contributions by Mike Birdgeneau and Henrik Bengtsson. RPushbullet now has support for channels (a reasonably new feature upstream). Setup, initialization and tests all got improved as well. Changes are summarized below based in the extract from the NEWS.Rd file.
    Changes in version 0.2.0 (2015-02-07)
    • Added support for Pushbullet 'channels' (once again thanks to Mike Birdgeneau for the initial push on this)
    • Support for pushes was solidified: proper choices of either device, email or channel should work in all cases
    • S3 methods are now properly exports (thanks to Henrik Bengtsson)
    • File transfer mode has been improved / corrected (thanks to Mike Birdgeneau)
    • The regression test suite was expanded and robustified
    • This NEWS file was added. Better late than never.
    Courtesy of CRANberries, there is also a diffstat report for this release. More details about the package are at the RPushbullet webpage and the RPushbullet GitHub repo.

    This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

    8 February 2015

    Diego Escalante Urrelo: Link Pack #06

    The story of Mel Blanc being saved by Bugs Bunny
    Such a fantastic and romantic story. Mel Blanc, voice of half the Looney Tunes characters was in an car crash and hanging from a thread in a comma. And it took his characters to save him. It s a very interestingly edited podcast too:
    ( ) a crazy story about Mel nearly dying in a crash on Dead Man s Curve on Hollywood Boulevard and about the moment two weeks later when Bugs Bunny emerged from Mel s coma before Mel did. In fact, according to neurosurgeon Louis Conway who attended to Mel at the time, it seemed as though Bugs Bunny was trying to save his life.
    The audacious rescue plan that might have saved space shuttle Columbia
    This is probably the best documentary/kinda fiction article I have read in a long time. It s a long read going over the hypothetical, and nail bitting, plan to rescue the Columbia if it ever got in trouble. It s really intense, and something to sit down and read fully focused, book style.
    But imagine an alternate timeline for the Columbia mission in which NASA quickly realized just how devastating the foam strike had been. Could the Columbia astronauts have been safely retrieved from orbit?
    Ocean Gravity: Riding sea currents
    Cool video of a sea diver running through seafloor currents, really cool and fascinating :). El chavo del ocho en el infierno
    This one is in Spanish, and it s a funny take on El chavo del ocho and how it follows some classic models and myths of literature. Chespirito was very fond of classics, so this would make sense. But, of course, the whole article is written on a tongue in cheek style.
    En El Chavo del Ocho, Bola os, o el Camus azteca, cre su propia versi n del mito de S sifo. El chavo y compa a est n condenados a empujar por una empinada colina todos los d as esta piedra enorme que siempre regresa, obligandolos al tormento del eterno retorno. La piedra de Quico es cuadrada, no rueda, se desliza. Es c mico, a pesar de tr gico.

    Go for some more Link Pack, it s sugar free

    Steve McIntyre: Yet another reason to not use Windows on your embedded devices...

    Seen on an ATM in Hong Kong airport.

    Broken ATM

    Next.

    Previous.