Search Results: "ag"

20 November 2015

Daniel Pocock: Databases of Muslims and homosexuals?

One US presidential candidate has said a lot recently, but the comments about making a database of Muslims may qualify as the most extreme. Of course, if he really wanted to, somebody with this mindset could find all the Muslims anyway. A quick and easy solution would involve tracing all the mobile phone signals around mosques on a Friday. Mr would-be President could compel Facebook and other social networks to disclose lists of users who identify as Muslim. Databases are a dangerous side-effect of gay marriage In 2014 there was significant discussion about Brendan Eich's donation to the campaign against gay marriage. One fact that never ranked very highly in the debate at the time is that not all gay people actually support gay marriage. Even where these marriages are permitted, not everybody who can marry now is choosing to do so. The reasons for this are varied, but one key point that has often been missed is that there are two routes to marriage equality: one involves permitting gay couples to visit the register office and fill in a form just as other couples do. The other route to equality is to remove all the legal artifacts around marriage altogether. When the government does issue a marriage certificate, it is not long before other organizations start asking for confirmation of the marriage. Everybody from banks to letting agents and Facebook wants to know about it. Many companies outsource that data into cloud CRM systems such as Salesforce. Before you know it, there are numerous databases that somebody could mine to make a list of confirmed homosexuals. Of course, if everybody in the world was going to live happily ever after none of this would be a problem. But the reality is different. While discrimination: either against Muslims or homosexuals - is prohibited and can even lead to criminal sanctions in some countries, this attitude is not shared globally. Once gay people have their marriage status documented in the frequent flyer or hotel loyalty program, or in the public part of their Facebook profile, there are various countries where they are going to be at much higher risk of prosecution/persecution. The equality to marry in the US or UK may mean they have less equality when choosing travel destinations. Those places are not as obscure as you might think: even in Australia, regarded as a civilized and laid-back western democracy, the state of Tasmania fought tooth-and-nail to retain the criminalization of virtually all homosexual conduct until 1997 when the combined actions of the federal government and high court compelled the state to reform. Despite the changes, people with some of the most offensive attitudes are able to achieve and retain a position of significant authority. The same Australian senator who infamously linked gay marriage with bestiality has successfully used his position to set up a Senate inquiry as a platform for conspiracy theories linking Halal certification with terrorism. There are many ways a database can fall into the wrong hands Ironically, one of the most valuable lessons about the risk of registering Muslims and homosexuals was an injustice against the very same tea-party supporters a certain presidential candidate is trying to woo. In 2013, it was revealed IRS employees had started applying a different process to discriminate against groups with Tea party in their name. It is not hard to imagine other types of rogue or misinformed behavior by people in positions of authority when they are presented with information that they don't actually need about somebody's religion or sexuality. Beyond this type of rogue behavior by individual officials and departments, there is also the more sinister proposition that somebody truly unpleasant is elected into power and can immediately use things like a Muslim database, surveillance data or the marriage database for a program of systematic discrimination. France had a close shave with this scenario in the 2002 presidential election when
Jean-Marie Le Pen, who has at least six convictions for racism or inciting racial hatred made it to the final round in a two-candidate run-off with Jacques Chirac. The best data security The best way to be safe- wherever you go, both now and in the future - is not to have data about yourself on any database. When filling out forms, think need-to-know. If some company doesn't really need your personal mobile number, your date of birth, your religion or your marriage status, don't give it to them.

Gergely Nagy: Looking for a keyboard

Even though I spend more time staring at the screen than typing, there are times when I - after lots and lots of prior brain work - sit down and start typing, a lot. A couple of years ago, I started to feel pain in my wrists, and there were multiple occasions when I had to completely stop writing for longer periods of time. These were situations I obviously did not want repeated, so I started to look for remedies. First, I bought a new keyboard, a TypeMatrix 2300, which while not ergonomic, was a huge relief for my hands and wrists. I also started to learn Dvorak, but that's still something that is kind-of in progress: my left hand can write Dvorak reasonably fast, but my right one seems to be Qwerty-wired, even after a month of typing Dvorak almost exclusively.This keyboard served me well for the past five year or so. But recently, I started to look for a replacement, partly triggered by a Clojure/conj talk I watched. I got as far as assembling a list of keyboards I'm interested in, but I have a hard time choosing. This blog post here serves two purposes then: first to make a clear pros/cons list for myself, second, to solicit feedback from others who may have more experience with any of the options below.Update: There is a [follow up post], with a few more keyboards explored, and a semi-final verdict. Thanks everyone for the feedback and help, much appreciated!Lets start with the current keyboard!TypeMatrix 2030TypeMatrix 2030Pros Cons SummaryAll in all, this is a keyboard I absolutely love, and am very happy with. Yet, I feel I'm ready to try something different. With my skins aging, and the aforementioned Clojure/conj talk, the desire to switch has been growing for a while now.Desired propertiesThere are a few desired properties of the keyboard I want next. The perfect keyboard need not have all of these, but the more the merrier. I plan to buy one keyboard for a start, but may end up buying another to bring to work (like I did with the TypeMatrix, except my employer at the time bought the second one for me). At work, I will continue using the TypeMatrix, most likely, but I'm not sure yet.Anyhow, there are a number of things I do with my computer that require a keyboard: I am looking for a keyboard that helps me do these things. A keyboard that will stay with me not for five years or a decade, but pretty much forever.The optionsUltimate Hacking KeyboardUltimate Hacking KeyboardPros Cons SummaryThe keyboard looks nice, has a lot of appealing features. It is programmable, so much so that by the looks of it, I could emulate the hardware dvorak switch my TypeMatrix has. However, I'm very unhappy with the addons, so there's that too.All in all, this would cost me about $304 (base keyboard, modules, palm rest and shipping). Not too bad, certainly a strong contender, despite the shortcomings.ErgoDoxErgoDoxPros Cons SummaryThe keyboard looks interesting, primarily due to the thumb keys. From the ErgoDox EZ campaign, I'm looking at $270. That's friendly, and makes ErgoDox a viable option! (Thanks @miffe!)Kinesis AdvantageKinesis AdvantagePros Cons SummaryThe key wells look interesting, but it's not a split keyboard, nor is it open source. The cost come out about $325 plus shipping and VAT and so on, so I'm probably looking at something closer to $400. Nah. I'm pretty sure I can rule this out.Kinesis FreeStyle2Kinesis FreeStyle2Pros Cons SummaryWhile a split keyboard, at a reasonably low cost ($149 + shipping + VAT), it lacks too many things to be considered a worthy contender.MaltronMaltronPros Cons SummaryWithout shipping, I'm looking at 450. That's a very steep price. I love the wells, and the thumb keys, but it's not split, and customisability is a big question here.AtreusAtreusPros Cons SummaryWhile not a split keyboard, it does look very interesting, and the price is much lower than the rest: $149 + shipping ($50 or so). It is similar - in spirit - to my existing TypeMatrix. It wouldn't take much to get used to, and is half the price of the alternatives. A strong option, for sure.Keyboardio M01Keyboardio Model 01Pros Cons SummaryWith shipping cost and whatnot, I'm looking at something in the $370 ballpark, which is on the more expensive side. On the other hand, I get a whole lot of bang for my buck: LEDs, two center bars (tripod mounting sounds really awesome!), hardwood body, and a key layout that is very similar to what I came to love on the TypeMatrix.I also have a thing for wooden stuff. I like the look of it, the feel of it.The Preference ListAfter writing this all up, I think I prefer the Model 01, but the UHK and ErgoDox come close too!The UHK is cheaper, but not by a large margin. It lacks the thumb keys and the palm key the M01 has. It also looks rather dull (sorry). They'd both ship about the same time, but, the M01 is already funded, while the UHK is not (mind you, there's a pretty darn high chance it will be).The ErgoDox has thumb keys, split keyboard, and is open source. Compared to the UHK, we have the thumb keys, and less distraction, for a better price. But the case is not so nice. Compared to the Model 01: no leds, or center bar, and an inferior case. But, much better price, which is an important factor too.Then, there's the Atreus. While it's a DIY kit, it is much more affordable than the rest, and I could have it far sooner. Yet... it doesn't feel like a big enough switch from my current keyboard. I might as well continue using the TypeMatrix then, right?The rest, I ruled out earlier, while I was reviewing them anyway.So, the big question is: should I invest close to $400 into a keyboard that looks stunning, and will likely grow old with me? Or should I give up some of the features, and settle for one of the $300 ones, that'll also grow old with me. Or is there an option I did not consider, that may match my needs and preferences better?If you, my dear reader, got this far, and have a suggestion, please either tweet at me, or write an email, or reach me over any other medium I am reachable at (including IRC, hanging out as algernon on FreeNode and OFTC).Thank you in advance, to all of you who contact me, and help me choose a keyboard!

Sylvain Beucler: Mini-sendmail... in bash

I recently faced an environment where there is no MTA. WTF? The reason is that people who work there get security audits on a regular basis, and the security people are usually mo...deratly skilled guys who blindly run a set of scripts, e.g. by ordering to disable Apache modules that "where seen enabled in /etc/apache2/mods-available/"... To avoid spending days arguing with them and nitpicking with non-technical managers, the system is trimmed to the minimum - and there is no MTA. No MTA, so no cron output, so difficulty to understand why last night's cron job failed miserably. Since it was not my role to reshape the whole business unit, I decided to hack a super-light, but functional way to get my cron output:
cat <<'EOF' > /usr/sbin/sendmail
#!/bin/bash
(
    echo "From me  $(LANG=C date)"
    cat
    echo
) >> /var/mail/all
EOF
chmod 755 /usr/sbin/sendmail
It works! :) There is a companion logrotate script, to avoid filling the file system:
cat <<'EOF' > /etc/logrotate.d/mail-all
/var/mail/all  
  daily
  rotate 10
  compress
  delaycompress
  notifempty
  create 622 root mail
 
EOF
Bootstrap with:
touch /var/mail/all
logrotate -f /var/mail/all
You now can check your sys-mails with:
mutt -f /var/mail/all
;)

Sylvain Beucler: Meritous: Free game ported on Android

Base attack Meritous is a nice, addictive action-adventure dungeon crawl game. Each new game is unique since the dungeon is built in a semi-random fashion. Last but not least, the engine, graphics and sound effects are GPL'd :) The game is based on SDL 1.2, which has an unofficial Android variant, so I decided to try and port it on my cell phone! The port was surprinsingly smooth and only non-SDL fixes (move big stack allocation to heap) were necessary. Who said it was difficult to program in C on Android? ;) It was also an opportunity to study the build system for F-Droid, an app market for free software apps, where APKs are rebuilt from source. The spec-like file is here. The game packaging is also being ressurected for Debian but is being distressfully held hostage in the NEW queue for 2 weeks! You can download the very first (aka beta) Android version: Comments welcome!

Sylvain Beucler: New free OpenGL ES documentation

Great news! The Learn OpenGL ES website recently switched its licensing to Creative Commons BY-SA 3.0 :) http://www.learnopengles.com/announcing-the-new-qa-and-forums-and-a-roundup/ It provides tutorials for OpenGL ES using Java/Android and WebGL, and is focusing on a more community-oriented creative process. Give them cheers!

Sylvain Beucler: No to ACTA - Paris

Today, there were events all around Europe to block ACTA. In Paris, the protest started at Place de la Bastille : APRIL was present, with in particular its president Lionel Allorge, and two members who wore the traditional anti-DRM suit : J r mie Zimmermann from La Quadrature du Net gave a speech and urged people to contact their legal representatives, in addition to protesting in the street : The protest was cheerful and free of violence : It got decent media coverage : Notable places it crossed include Place des Victoires : and Palais Royal, where it ended : Next protest is in 2 weeks, on March 10th. Update your agenda!

Sylvain Beucler: Rebuilding Android proprietary SDK binaries

Going back to Android recently, I saw that all tools binaries from the Android project are now click-wrapped by a quite ugly proprietary license, among others an anti-fork clause (details below). Apparently those T&C are years old, but the click-wrapping is newer. This applies to the SDK, the NDK, Android Studio, and all the essentials you download through the Android SDK Manager. Since I keep my hands clean of smelly EULAs, I'm working on rebuilding the Android tools I need.
We're talking about hours-long, quad-core + 8GB-RAM + 100GB-disk-eating builds here, so I'd like to publish them as part of a project who cares. As a proof-of-concept, the Replicant project ships a 4.2 SDK and I contributed build instructions for ADT and NDK (which I now use daily). (Replicant is currently stuck to a 2013 code base though.) I also have in-progress instructions on my hard-drive to rebuild various newer versions of the SDK/API levels, and for the NDK whose releases are quite hard to reproduce (no git tags, requires fixes committed after the release, updates are partial rebuilds, etc.) - not to mention that Google doesn't publish the source code until after the official release (closed development) :/ And in some cases like Android Support Repository [not Library] I didn't even find the proper source code, only an old prebuilt. Would you be interested in contributing, and would you recommend a structure that would promote Free, rebuilt Android *DK? The legalese Anti-fork clause:
3.4 You agree that you will not take any actions that may cause or result in the fragmentation of Android, including but not limited to distributing, participating in the creation of, or promoting in any way a software development kit derived from the SDK.
So basically the source is Apache 2 + GPL, but the binaries are non-free. By the way this is not a GPL violation because right after:
3.5 Use, reproduction and distribution of components of the SDK licensed under an open source software license are governed solely by the terms of that open source software license and not this License Agreement.
Still, AFAIU by clicking "Accept" to get the binary you still accept the non-free "Terms and Conditions". (Incidentally, if Google wanted SDK forks to spread and increase fragmentation, introducing an obnoxious EULA is probably the first thing I'd have recommended. What was its legal team thinking?) Indemnification clause:
12.1 To the maximum extent permitted by law, you agree to defend, indemnify and hold harmless Google, its affiliates and their respective directors, officers, employees and agents from and against any and all claims, actions, suits or proceedings, as well as any and all losses, liabilities, damages, costs and expenses (including reasonable attorneys fees) arising out of or accruing from (a) your use of the SDK, (b) any application you develop on the SDK that infringes any copyright, trademark, trade secret, trade dress, patent or other intellectual property right of any person or defames any person or violates their rights of publicity or privacy, and (c) any non-compliance by you with this License Agreement.
Usage restriction:
3.1 Subject to the terms of this License Agreement, Google grants you a limited, worldwide, royalty-free, non-assignable and non-exclusive license to use the SDK solely to develop applications to run on the Android platform. 3.3 You may not use the SDK for any purpose not expressly permitted by this License Agreement. Except to the extent required by applicable third party licenses, you may not: (a) copy (except for backup purposes), modify, adapt, redistribute, decompile, reverse engineer, disassemble, or create derivative works of the SDK or any part of the SDK; or (b) load any part of the SDK onto a mobile handset or any other hardware device except a personal computer, combine any part of the SDK with other software, or distribute any software or device incorporating a part of the SDK.
If you know the URLs, you can still direct-download some of the binaries which don't embed the license, but all this feels fishy. GNU licensing didn't answer me (yet). Maybe debian-legal has an opinion? In any case, the difficulty to reproduce the *DK builds is worrying enough to warrant an independent rebuild. Did you notice this?

Timo Jyrinki: Converting an existing installation to LUKS using luksipc

This is a burst of notes that I wrote in an e-mail in June when asked about it, and I'm not going to have any better steps since I don't remember even that amount as back then. I figured it's better to have it out than not.

So... if you want to use LUKS In-Place Conversion Tool, the notes below on converting a shipped-with-Ubuntu Dell XPS 13 Developer Edition (2015 Intel Broadwell model) may help you. There were a couple of small learnings to be had...

The page http://www.johannes-bauer.com/linux/luksipc/ itself is good and without errors, although funnily uses reiserfs as an example. It was only a bit unclear why I did save the initial_keyfile.bin since it was then removed in the next step (I guess it's for the case you want to have a recovery file hidden somewhere in case you forget the passphrase).

For using the tool I booted from a 14.04.2 LTS USB live image and operated there, including downloading and compiling luksipc in the live session. The exact reason of resizing before luksipc was a bit unclear to me at first so I simply indeed resized the main rootfs partition and left unallocated space in the partition table.


Then finally I ran ./luksipc -d /dev/sda4 etc.


I realized I want /boot to be on an unencrypted partition to be able to load the kernel + initrd from grub before entering into LUKS unlocking. I couldn't resize the luks partition anymore since it was encrypted... So I resized what I think was the empty small DIAGS partition (maybe used for some system diagnostic or something, I don't know), or possibly the next one that is the actual recovery partition one can reinstall the pre-installed Ubuntu from. And naturally I had some problems because it seems vfatresize tool didn't do what I wanted it to do and gparted simply crashed when I tried to use it first to do the same. Anyway, when done with getting some extra free space somewhere, I used the remaining 350MB for /boot where I copied the rootfs's /boot contents to.

After adding the passphrase in luks I had everything encrypted etc and decryptable, but obviously I could only access it from a live session by manual cryptsetup luksOpen + mount /dev/mapper/myroot commands. I needed to configure GRUB, and I needed to do it with the grub-efi-amd64 which was a bit unfamiliar to me. There's also grub-efi-amd64-signed I have installed now but I'm not sure if it was required for the configuration. Secure boot is not enabled by default in BIOS so maybe it isn't needed.


I did GRUB installation I think inside rootfs chroot where I also mounted /dev/sda6 as /boot (inside the rootfs chroot), ie mounted dev, sys with -o bind to under the chroot (from outside chroot) and mount -t proc proc proc too. I did a lot of trial and effort so I surely also tried from outside the chroot, in the live session, using some parameters to point to the mounted rootfs's directories...


I needed to definitely install cryptsetup etc inside the encrypted rootfs with apt, and I remember debugging for some time if they went to the initrd correctly after I executed mkinitramfs/update-initramfs inside the chroot.


At the end I had grub asking for the password correctly at bootup. Obviously I had edited the rootfs's /etc/fstab to include the new /boot partition, I changed / to be "UUID=/dev/mapper/myroot / ext4 errors=remount-ro 0 ", kept /boot/efi as coming from the /dev/sda1 and so on. I had also added "myroot /dev/sda4 none luks" to /etc/crypttab. I seem to also have GRUB_CMDLINE_LINUX="cryptdevice=/dev/sda4:myroot root=/dev/mapper/myroot" in /etc/default/grub.

The only thing I did save from the live session was the original partition table if I want to revert.


So the original was:

Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 500118192 sectors, 238.5 GiB
Logical sector size: 512 bytes
...
First usable sector is 34, last usable sector is 500118158
Partitions will be aligned on 2048-sector boundaries
Total free space is 6765 sectors (3.3 MiB)

Number Start (sector) End (sector) Size Code Name
1 2048 1026047 500.0 MiB EF00 EFI system partition
2 1026048 1107967 40.0 MiB FFFF Basic data partition
3 1107968 7399423 3.0 GiB 0700 Basic data partition
4 7399424 467013631 219.2 GiB 8300
5 467017728 500117503 15.8 GiB 8200

And I now have:


Number Start (sector) End (sector) Size Code Name

1 2048 1026047 500.0 MiB EF00 EFI system partition
2 1026048 1107967 40.0 MiB FFFF Basic data partition
3 1832960 7399423 2.7 GiB 0700 Basic data partition
4 7399424 467013631 219.2 GiB 8300
5 467017728 500117503 15.8 GiB 8200
6 1107968 1832959 354.0 MiB 8300

So it seems I did not edit DIAGS (and it was also originally just 40MB) but did something with the recovery partition while preserving its contents. It's a FAT partition so maybe I was able to somehow resize it after all.


The 16GB partition is the default swap partition. I did not encrypt it at least yet, I tend to not run into swap anyway ever in my normal use with the 8GB RAM.


If you go this route, good luck! :D

Pau Garcia i Quiles: Desktops DevRoom @ FOSDEM 2016: Have you submitted your talk yet?

FOSDEM 2016 is going to be great (again!) and you still have the chance to be one of the stars. Have you submitted your talk to the Desktops DevRoom yet? No? Remember: we will only accept proposals until December 6th. After that, the Organization Team will get busy and vote and choose the talks. Here is the full Call for Participation, in case you need to check the details on how to submit: FOSDEM Desktops DevRoom 2016 Call for Participation Topics include anything related to the Desktop: desktop environments, software development for desktop/cross-platform, applications, UI, etc

19 November 2015

Matthew Garrett: If it's not practical to redistribute free software, it's not free software in practice

I've previously written about Canonical's obnoxious IP policy and how Mark Shuttleworth admits it's deliberately vague. After spending some time discussing specific examples with Canonical, I've been explicitly told that while Canonical will gladly give me a cost-free trademark license permitting me to redistribute unmodified Ubuntu binaries, they will not tell me what Any redistribution of modified versions of Ubuntu must be approved, certified or provided by Canonical if you are going to associate it with the Trademarks. Otherwise you must remove and replace the Trademarks and will need to recompile the source code to create your own binaries actually means.

Why does this matter? The free software definition requires that you be able to redistribute software to other people in either unmodified or modified form without needing to ask for permission first. This makes it clear that Ubuntu itself isn't free software - distributing the individual binary packages without permission is forbidden, even if they wouldn't contain any infringing trademarks[1]. This is obnoxious, but not inherently toxic. The source packages for Ubuntu could still be free software, making it fairly straightforward to build a free software equivalent.

Unfortunately, while true in theory, this isn't true in practice. The issue here is the apparently simple phrase you must remove and replace the Trademarks and will need to recompile the source code. "Trademarks" is defined later as being the words "Ubuntu", "Kubuntu", "Juju", "Landscape", "Edubuntu" and "Xubuntu" in either textual or logo form. The naive interpretation of this is that you have to remove trademarks where they'd be infringing - for instance, shipping the Ubuntu bootsplash as part of a modified product would almost certainly be clear trademark infringement, so you shouldn't do that. But that's not what the policy actually says. It insists that all trademarks be removed, whether they would embody an infringement or not. If a README says "To build this software under Ubuntu, install the following packages", a literal reading of Canonical's policy would require you to remove or replace the word "Ubuntu" even though failing to do so wouldn't be a trademark infringement. If an @ubuntu.com email address is present in a changelog, you'd have to change it. You wouldn't be able to ship the juju-core package without renaming it and the application within. If this is what the policy means, it's so impractical to be able to rebuild Ubuntu that it's not free software in any meaningful way.

This seems like a pretty ludicrous interpretation, but it's one that Canonical refuse to explicitly rule out. Compare this to Red Hat's requirements around Fedora - if you replace the fedora-logos, fedora-release and fedora-release-notes packages with your own content, you're good. A policy like this satisfies the concerns that Dustin raised over people misrepresenting their products, but still makes it easy for users to distribute modified code to other users. There's nothing whatsoever stopping Canonical from adopting a similarly unambiguous policy.

Mark has repeatedly asserted that attempts to raise this issue are mere FUD, but he won't answer you if you ask him direct questions about this policy and will insist that it's necessary to protect Ubuntu's brand. The reality is that if Debian had had an identical policy in 2004, Ubuntu wouldn't exist. The effort required to strip all Debian trademarks from the source packages would have been immense[2], and this would have had to be repeated for every release. While this policy is in place, nobody's going to be able to take Ubuntu and build something better. It's grotesquely hypocritical, especially when the Ubuntu website still talks about their belief that people should be able to distribute modifications without licensing fees.

All that's required for Canonical to deal with this problem is to follow Fedora's lead and isolate their trademarks in a small set of packages, then tell users that those packages must be replaced if distributing a modified version of Ubuntu. If they're serious about this being a branding issue, they'll do it. And if I'm right that the policy is deliberately obfuscated so Canonical can encourage people to buy licenses, they won't. It's easy for them to prove me wrong, and I'll be delighted if they do. Let's see what happens.

[1] The policy is quite clear on this. If you want to distribute something other than an unmodified Ubuntu image, you have two choices:
  1. Gain approval or certification from Canonical
  2. Remove all trademarks and recompile the source code
Note that option 2 requires you to rebuild even if there are no trademarks to remove.

[2] Especially when every source package contains a directory called "debian"

comment count unavailable comments

Vincent Fourmond: Purely shell way to extract a numbered line from a file

I feel almost shameful to write it down, but as it took me a long time to realize this, I'll write it down anyway. Here's the simplest portable shell one-liner I've found to extract only the, say, 5th line from file:
~ cat file   tail -n +5   head -n1
Hope it helps...Update: following the comments to this post, here are a couple of other solutions. Thanks to all who contributed !
~ cat file   sed -ne 5p
~ cat file   sed -ne '5 p;q ' 
The second solution has the advantage of closing the input after line 5, so if you have an expensive command, you'll kill it with a SIGPIPE soon after it produces line 5. Other ones:
~ cat file   awk 'NR==5  print; exit '
~ cat file   head -n5   tail -n1 
The last one, while simpler, is slightly more expensive because all the lines before the one you're interested in are copied twice (first from cat to head and then from head to tail). This happens less with the first solution because, even if tail passes on all the lines after the one you're interested in, it is killed by a SIGPIPE when head closes.

Miriam Ruiz: Projects, Conflicts and Emotions

The Debian Project includes many people, groups and teams with different goals, priorities and ways of doing things. Diversity is a good thing, and the results of the continuous interaction, cooperation and competition among different points of view and components make up a successful developing framework both in Debian and in other Free / Libre / Open Source Software communities. The cost of this evolutionary paradigm is that sometimes there are subprojects that might have been extremely successful and useful that are surpassed by newer approaches, or that have to compete with alternative approaches that were not there before, and which might pursue different goals or have a different way of doing things that their developers find preferable in terms of modularity, scalability, stability, maintenance, aesthetics or any other reason. Whenever this happens, the emotional impact on the person or group of people that are behind the established component (or process, or organizational structure), that is being questioned and put under test by the newer approach can be important, particularly when they have invested a lot of time and effort and a considerable amount of emotional energy doing a great job for many years. Something they should be thanked for. This might be particularly hard when -for whatever reason- the communication between both teams is not too fluent or constant, and sometimes the author or authors of the solution that was considered mainstream until then might feel left out and their territory stolen. As generally development teams and technical people in the Free / Libre / Open Source world are more focused on results than on relationships, projects are generally not too good at managing this (emotional, relational) situations, even though they (we) are gradually learning and improving. What has happened with the Debian Live Project is indeed a hurtful situation, even though it s probably an unavoidable one. The Debian Live Project has done a great job for many years and it is sad to see it dying abruptly. A new competing approach is on its way with a different set of priorities and different way of doing things, and all that can be done at the moment is to thank Daniel for all his work, as well as everyone who has made the Debian Live Project successful for so many years, also thank the people who are investing their time and effort in developing something that might be even better. Lets wait and see. Source of the image: Conflict Modes and Managerial Styles by Ed Batista

18 November 2015

Daniel Pocock: Improving DruCall and JSCommunicator user interface

DruCall is one of the easiest ways to get up and running with WebRTC voice and video calling on your own web site or blog. It is based on 100% open source and 100% open standards - no binary browser plugins and no lock-in to a specific service provider or vendor. On Debian or Ubuntu, just running a command such as
# apt-get install -t jessie-backports drupal7-mod-drucall
will install Drupal, Apache, MySQL, JSCommunicator, JsSIP and all the other JavaScript library packages and module dependencies for DruCall itself. The user interface Most of my experience is in server-side development, including things like the powerful SIP over WebSocket implementation in the reSIProcate SIP proxy repro. In creating DruCall, I have simply concentrated on those areas related to configuring and bringing up the WebSocket connection and creating the authentication tokens for the call. Those things provide a firm foundation for the module, but it would be nice to improve the way it is presented and optimize the integration with other Drupal features. This is where the projects (both DruCall and JSCommunicator) would really benefit from feedback and contributions from people who know Drupal and web design in much more detail. Benefits for collaboration If anybody wants to collaborate on either or both of these projects, I'd be happy to offer access to a pre-configured SIP WebSocket server in my lab for more convenient testing. The DruCall source code is a Drupal.org hosted project and the JSCommunicator source code is on Github. When you get to the stage where you want to run your own SIP WebSocket server as well then free community support can also be provided through the repro-user mailing list. The free, online RTC Quick Start Guide gives a very comprehensive overview of everything you need to do to run your own WebRTC SIP infrastructure.

17 November 2015

Norbert Preining: Debian/TeX Live 2015.20151116-1

One month has passed since the big multiarch update, and not one bug report concerning it did come in, that are good news. So here is a completely boring update with nothing more than the usual checkout from the TeX Live tlnet distribution as of yesterday. Debian - TeX Live 2015 I cannot recall anything particular to mention here, so this time let me go with the list of updated and new packages only: Updated packages alegreya, algorithm2e, animate, archaeologie, articleingud, attachfile, bankstatement, beebe, biber, biblatex, biblatex-manuscripts-philology, biblatex-opcit-booktitle, bidi, br-lex, bytefield, catcodes, chemfig, chemformula, chemgreek, comprehensive, computational-complexity, ctable, datetime2, dowith, dvipdfmx, dvipdfmx-def, dynamicnumber, e-french, epspdf, etoc, fetamont, findhyph, fix2col, gitinfo2, gradstudentresume, indextools, kotex-oblivoir, kotex-utils, kpathsea, l3build, l3experimental, l3kernel, l3packages, latex, latex2e-help-texinfo, lisp-on-tex, ltxfileinfo, luatexja, makedtx, mathastext, mathtools, mcf2graph, media9, medstarbeamer, morehype, nameauth, nicetext, ocgx2, perltex, preview, prftree, pst-eucl, pstricks, reledmac, resphilosophica, selnolig, substances, tcolorbox, tetex, teubner, tex4ht, texdoc, texinfo, texlive-scripts, toptesi, translations, turabian-formatting, uptex, xepersian, xetex, xetex-def, xint. New packages asciilist, babel-macedonian, bestpapers, bibtexperllibs, fixcmex, iffont, nucleardata, srcredact, texvc, xassoccnt. Enjoy.

Sven Hoexter: Failing with F5: assign a http profile and an irule at the same time

Beside of an upgrade to TMOS 11.4.1HF9 I wanted to use a maintenance today to assign some specific irule to a VS. Within the irule I use some HTTP functions so when I tried to add the irule to the already existing VS the tmsh correctly told me that I also need a http profile on this VS. Thanks tmsh you're right, oversight by myself. So what I did was:
tmsh modify ltm virtual mySpecialVS rules   mySpecialiRule   profiles add   company-http-profile  
tmsh accepted but all my tests ended at the VS. I could connect but got no reply at all. That was strange because I tested this irule extensively. So I reverted back to the known good state with just plain tcp forwarding. My next try was to assign only the http profile without the irule.
tmsh modify ltm virtual mySpecialVS profiles add   company-http-profile  
Tested that and it worked. So what on earth was wrong with my irule? I added some debug statements and readded the irule like this:
tmsh modify ltm virtual mySpecialVS rules   mySpecialiRule  
And now it worked as intended. So I went on and removed my debug statements, tested again and it still works. Let's see if I can reproduce that case some time later this week to fill a proper bugreport with F5. Update: Turns out it was all my fault. Due to a misunderstanding about RULE_INIT and the static namespace, I managed to overwrite important variables globally. Lesson learned: Be very careful if you use "static::" or better avoid it. Also think twice if you start to set things on the RULE_INIT event. Since it's only called on saving an irule or restarts of the device, your errors might show only later when you do not expect that.

Petter Reinholdtsen: PGP key transition statement for key EE4E02F9

I've needed a new OpenPGP key for a while, but have not had time to set it up properly. I wanted to generate it offline and have it available on a OpenPGP smart card for daily use, and learning how to do it and finding time to sit down with an offline machine almost took forever. But finally I've been able to complete the process, and have now moved from my old GPG key to a new GPG key. See the full transition statement, signed with both my old and new key for the details. This is my new key:
pub   3936R/111D6B29EE4E02F9 2015-11-03 [expires: 2019-11-14]
      Key fingerprint = 3AC7 B2E3 ACA5 DF87 78F1  D827 111D 6B29 EE4E 02F9
uid                  Petter Reinholdtsen <pere@hungry.com>
uid                  Petter Reinholdtsen <pere@debian.org>
sub   4096R/87BAFB0E 2015-11-03 [expires: 2019-11-02]
sub   4096R/F91E6DE9 2015-11-03 [expires: 2019-11-02]
sub   4096R/A0439BAB 2015-11-03 [expires: 2019-11-02]
The key can be downloaded from the OpenPGP key servers, signed by my old key. If you signed my old key (DB4CCC4B2A30D729), I'd very much appreciate a signature on my new key, details and instructions in the transition statement. I m happy to reciprocate if you have a similarly signed transition statement to present.

Dirk Eddelbuettel: RcppAnnoy 0.0.7

A new version of RcppAnnoy, our Rcpp-based R integration of the nifty Annoy library by Erik, is now on CRAN. Annoy is a small, fast, and lightweight C++ template header library for approximate nearest neighbours. This release mostly just catches up with the Annoy release 1.6.2 of last Friday. No new features were added on our side. Courtesy of CRANberries, there is also a diffstat report for this release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

16 November 2015

Daniel Pocock: Quick start using Blender for video editing

Updated 2015-11-16 for WebM Although it is mostly known for animation, Blender includes a non-linear video editing system that is available in all the current stable versions of Debian, Ubuntu and Fedora. Here are some screenshots showing how to start editing a video of a talk from a conference. In this case, there are two input files:
  • A video file from a DSLR camera, including an audio stream from a microphone on the camera
  • A separate audio file with sound captured by a lapel microphone attached to the speaker's smartphone. This is a much better quality sound and we would like this to replace the sound included in the video file.
Open Blender and choose the video editing mode Launch Blender and choose the video sequence editor from the pull down menu at the top of the window: Now you should see all the video sequence editor controls: Setup the properties for your project Click the context menu under the strip editor panel and change the panel to a Properties panel: The video file we are playing with is 720p, so it seems reasonable to use 720p for the output too. Change that here: The input file is 25fps so we need to use exactly the same frame rate for the output, otherwise you will either observe the video going at the wrong speed or there will be a conversion that is CPU intensive and degrades the quality. Also check that the resolution_percentage setting under the picture dimensions is 100%: Now specify output to PNG files. Later we will combine them into a WebM file with a script. Specify the directory where the files will be placed and use the # placeholder to specify the number of digits to use to embed the frame number in the filename: Now your basic rendering properties are set. When you want to generate the output file, come back to this panel and use the Animation button at the top. Editing the video Use the context menu to change the properties panel back to the strip view panel: Add the video file: and then right click the video strip (the lower strip) to highlight it and then add a transform strip: Audio waveform Right click the audio strip to highlight it and then go to the properties on the right hand side and click to show the waveform: Rendering length By default, Blender assumes you want to render 250 frames of output. Looking in the properties to the right of the audio or video strip you can see the actual number of frames. Put that value in the box at the bottom of the window where it says 250: Enable AV-sync Also at the bottom of the window is a control to enable AV-sync. If your audio and video are not in sync when you preview, you need to set this AV-sync option and also make sure you set the frame rate correctly in the properties: Add the other sound strip Now add the other sound file that was recorded using the lapel microphone: Enable the waveform display for that sound strip too, this will allow you to align the sound strips precisely: You will need to listen to the strips to make an estimate of the time difference. Use this estimate to set the "start frame" in the properties for your audio strip, it will be a negative value if the audio strip starts before the video. You can then zoom the strip panel to show about 3 to 5 seconds of sound and try to align the peaks. An easy way to do this is to look for applause at the end of the audio strips, the applause generates a large peak that is easily visible. Once you have synced the audio, you can play the track and you should not be able to hear any echo. You can then silence the audio track from the camera by right clicking it, look in the properties to the right and change volume to 0. Make any transforms you require For example, to zoom in on the speaker, right click the transform strip (3rd from the bottom) and then in the panel on the right, click to enable "Uniform Scale" and then set the scale factor as required: Render the video output to PNG Click the context menu under the Curves panel and choose Properties again. Click the Animation button to generate a sequence of PNG files for each frame. Render the audio output On the Properties panel, click the Audio button near the top. Choose a filename for the generated audio file. Look on the bottom left-hand side of the window for the audio file settings, change it to the ogg container and Vorbis codec: Ensure the filename has a .ogg extension Now look at the top right-hand corner of the window for the Mixdown button. Click it and wait for Blender to generate the audio file. Combine the PNG files and audio file into a WebM video file You will need to have a few command line tools installed for manipulating the files from scripts. Install them using the package manager, for example, on a Debian or Ubuntu system: # apt-get install mjpegtools vpx-tools mkvtoolnix
Now create a script like the following:
#!/bin/bash -e
# Set this to match the project properties
FRAME_RATE=25
# Set this to the rate you desire:
TARGET_BITRATE=1000
WORK_DIR=$ HOME /video1
PNG_DIR=$ WORK_DIR /frames
YUV_FILE=$ WORK_DIR /video.yuv
WEBM_FILE=$ WORK_DIR /video.webm
AUDIO_FILE=$ WORK_DIR /audio-mixed.ogg
NUM_FRAMES= find $ PNG_DIR  -type f   wc -l 
png2yuv -I p -f $FRAME_RATE -b 1 -n $NUM_FRAMES \
    -j $ PNG_DIR /%08d.png > $ YUV_FILE 
vpxenc --good --cpu-used=0 --auto-alt-ref=1 \
   --lag-in-frames=16 --end-usage=vbr --passes=2 \
   --threads=2 --target-bitrate=$ TARGET_BITRATE  \
   -o $ WEBM_FILE -noaudio $ YUV_FILE 
rm $ YUV_FILE 
mkvmerge -o $ WEBM_FILE  -w $ WEBM_FILE -noaudio $ AUDIO_FILE 
rm $ WEBM_FILE -noaudio
Next steps There are plenty of more comprehensive tutorials, including some videos on Youtube, explaining how to do more advanced things like fading in and out or zooming and panning dynamically at different points in the video. If the lighting is not good (faces too dark, for example), you can right click the video strip, go to the properties panel on the right hand side and click Modifiers, Add Strip Modifier and then select "Color Balance". Use the Lift, Gamma and Gain sliders to adjust the shadows, midtones and highlights respectively.

Steve Kemp: lumail2 nears another release

I'm pleased with the way that Lumail2 development is proceeding, and it is reaching a point where there will be a second source-release. I've made a lot of changes to the repository recently, and most of them boil down to moving code from the C++ side of the application, over to the Lua side. This morning, for example, I updated the handing of index.limit to be entirely Lua based.
When you open a Maildir folder you see the list of messages it contains, as you would expect. The notion of the index.limit is that you can limit the messages displayed, for example:
  • See all messages: Config:set( "index.limit", "all")
  • See only new/unread messages: Config:set( "index.limit", "new")
  • See only messages which arrived today: Config:set( "index.limit", "today")
  • See only messages which contain "Steve" in their formatted version: Config:set( "index.limit", "steve")
These are just examples that are present as defaults, but they give an idea of how things can work. I guess it isn't so different to Mutt's "limit" facilities - but thanks to the dynamic Lua nature of the application you can add your own with relative ease.
One of the biggest changes, recently, was the ability to display coloured text! That was always possible before, but a single line could only be one colour. Now colours can be mixed within a line, so this works as you might imagine: Panel:append( "$[RED]This is red, $[GREEN]green, $[WHITE]white, and $[CYAN]cyan!" ) Other changes include a persistant cache of "stuff", which is Lua-based, the inclusion of at least one luarocks library to parse Date: headers, and a simple API for all our objects. All good stuff. Perhaps time for a break in the next few weeks, but right now I think I'm making useful updates every other evening or so.

Julien Danjou: Profiling Python using cProfile: a concrete case

Writing programs is fun, but making them fast can be a pain. Python programs are no exception to that, but the basic profiling toolchain is actually not that complicated to use. Here, I would like to show you how you can quickly profile and analyze your Python code to find what part of the code you should optimize. What's profiling? Profiling a Python program is doing a dynamic analysis that measures the execution time of the program and everything that compose it. That means measuring the time spent in each of its functions. This will give you data about where your program is spending time, and what area might be worth optimizing. It's a very interesting exercise. Many people focus on local optimizations, such as determining e.g. which of the Python functions range or xrange is going to be faster. It turns out that knowing which one is faster may never be an issue in your program, and that the time gained by one of the functions above might not be worth the time you spend researching that, or arguing about it with your colleague. Trying to blindly optimize a program without measuring where it is actually spending its time is a useless exercise. Following your guts alone is not always sufficient. There are many types of profiling, as there are many things you can measure. In this exercise, we'll focus on CPU utilization profiling, meaning the time spent by each function executing instructions. Obviously, we could do many more kind of profiling and optimizations, such as memory profiling which would measure the memory used by each piece of code something I talk about in The Hacker's Guide to Python. cProfile Since Python 2.5, Python provides a C module called cProfile which has a reasonable overhead and offers a good enough feature set. The basic usage goes down to:
>>> import cProfile
>>> cProfile.run('2 + 2')
2 function calls in 0.000 seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 method 'disable' of '_lsprof.Profiler' objects

Though you can also run a script with it, which turns out to be handy:
$ python -m cProfile -s cumtime lwn2pocket.py
72270 function calls (70640 primitive calls) in 4.481 seconds

Ordered by: cumulative time

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.004 0.004 4.481 4.481 lwn2pocket.py:2(<module>)
1 0.001 0.001 4.296 4.296 lwn2pocket.py:51(main)
3 0.000 0.000 4.286 1.429 api.py:17(request)
3 0.000 0.000 4.268 1.423 sessions.py:386(request)
4/3 0.000 0.000 3.816 1.272 sessions.py:539(send)
4 0.000 0.000 2.965 0.741 adapters.py:323(send)
4 0.000 0.000 2.962 0.740 connectionpool.py:421(urlopen)
4 0.000 0.000 2.961 0.740 connectionpool.py:317(_make_request)
2 0.000 0.000 2.675 1.338 api.py:98(post)
30 0.000 0.000 1.621 0.054 ssl.py:727(recv)
30 0.000 0.000 1.621 0.054 ssl.py:610(read)
30 1.621 0.054 1.621 0.054 method 'read' of '_ssl._SSLSocket' objects
1 0.000 0.000 1.611 1.611 api.py:58(get)
4 0.000 0.000 1.572 0.393 httplib.py:1095(getresponse)
4 0.000 0.000 1.572 0.393 httplib.py:446(begin)
60 0.000 0.000 1.571 0.026 socket.py:410(readline)
4 0.000 0.000 1.571 0.393 httplib.py:407(_read_status)
1 0.000 0.000 1.462 1.462 pocket.py:44(wrapped)
1 0.000 0.000 1.462 1.462 pocket.py:152(make_request)
1 0.000 0.000 1.462 1.462 pocket.py:139(_make_request)
1 0.000 0.000 1.459 1.459 pocket.py:134(_post_request)
[ ]

This prints out all the function called, with the time spend in each and the number of times they have been called. Advanced visualization with KCacheGrind While being useful, the output format is very basic and does not make easy to grab knowledge for complete programs. For more advanced visualization, I leverage KCacheGrind. If you did any C programming and profiling these last years, you may have used it as it is primarily designed as front-end for Valgrind generated call-graphs. In order to use, you need to generate a cProfile result file, then convert it to KCacheGrind format. To do that, I use pyprof2calltree.
$ python -m cProfile -o myscript.cprof myscript.py
$ pyprof2calltree -k -i myscript.cprof

And the KCacheGrind window magically appears!
Concrete case: Carbonara optimization I was curious about the performances of Carbonara, the small timeserie library I wrote for Gnocchi. I decided to do some basic profiling to see if there was any obvious optimization to do. In order to profile a program, you need to run it. But running the whole program in profiling mode can generate a lot of data that you don't care about, and adds noise to what you're trying to understand. Since Gnocchi has thousands of unit tests and a few for Carbonara itself, I decided to profile the code used by these unit tests, as it's a good reflection of basic features of the library. Note that this is a good strategy for a curious and naive first-pass profiling. There's no way that you can make sure that the hotspots you will see in the unit tests are the actual hotspots you will encounter in production. Therefore, a profiling in conditions and with a scenario that mimics what's seen in production is often a necessity if you need to push your program optimization further and want to achieve perceivable and valuable gain. I activated cProfile using the method described above, creating a cProfile.Profile object around my tests (I actually started to implement that in testtools). I then run KCacheGrind as described above. Using KCacheGrind, I generated the following figures.
The test I profiled here is called test_fetch and is pretty easy to understand: it puts data in a timeserie object, and then fetch the aggregated result. The above list shows that 88 % of the ticks are spent in set_values (44 ticks over 50). This function is used to insert values into the timeserie, not to fetch the values. That means that it's really slow to insert data, and pretty fast to actually retrieve them. Reading the rest of the list indicates that several functions share the rest of the ticks, update, _first_block_timestamp, _truncate, _resample, etc. Some of the functions in the list are not part of Carbonara, so there's no point in looking to optimize them. The only thing that can be optimized is, sometimes, the number of times they're called.
The call graph gives me a bit more insight about what's going on here. Using my knowledge about how Carbonara works, I don't think that the whole stack on the left for _first_block_timestamp makes much sense. This function is supposed to find the first timestamp for an aggregate, e.g. with a timestamp of 13:34:45 and a period of 5 minutes, the function should return 13:30:00. The way it works currently is by calling the resample function from Pandas on a timeserie with only one element, but that seems to be very slow. Indeed, currently this function represents 25 % of the time spent by set_values (11 ticks on 44). Fortunately, I recently added a small function called _round_timestamp that does exactly what _first_block_timestamp needs that without calling any Pandas function, so no resample. So I ended up rewriting that function this way:
def _first_block_timestamp(self):
- ts = self.ts[-1:].resample(self.block_size)
- return (ts.index[-1] - (self.block_size * self.back_window))
+ rounded = self._round_timestamp(self.ts.index[-1], self.block_size)
+ return rounded - (self.block_size * self.back_window)

And then I re-run the exact same test to compare the output of cProfile.
The list of function seems quite different this time. The number of time spend used by set_values dropped from 88 % to 71 %.
The call stack for set_values shows that pretty well: we can't even see the _first_block_timestamp function as it is so fast that it totally disappeared from the display. It's now being considered insignificant by the profiler. So we just speed up the whole insertion process of values into Carbonara by a nice 25 % in a few minutes. Not that bad for a first naive pass, right?

Next.

Previous.