Search Results: "rul"

13 September 2015

Russ Allbery: Review: The Sea Thy Mistress

Review: The Sea Thy Mistress, by Elizabeth Bear
Series: Edda of Burdens #3
Publisher: Tor
Copyright: February 2011
ISBN: 0-7653-1884-9
Format: Hardcover
Pages: 334
The Sea Thy Mistress is the third book in the Edda of Burdens series, but the second book (By the Mountain Bound) was a prequel to the first, telling the story of an earlier world and the betrayal that ended it. This one is a direct sequel to All the Windwracked Stars, a complex and idea-rich book that I read more than six years ago. A challenging property of Bear's fiction is that it's short on explanation and doesn't hold your hand. This makes it lean, sharp, and effective if you're fully invested in it and can keep the context in your head. It becomes very hard to understand if you've mostly forgotten the plot of an earlier book. (Once again, I bemoan the death of the previous book synopsis in series fiction.) The Sea Thy Mistress is all about aftermath, loss, despair, unhealed wounds, old grudges, and complicated personal histories, and it relies very heavily on the reader being invested enough in the characters to care about all of those things. I wish I'd read this entire series closer together so that I could retain more of the state. The start of this story was therefore somewhat frustrating, as I scrambled to rebuild my memory of the first book and piece together the large cast. Thankfully, most of the characters are easy to care about. Aethelred reliably brought a smile to my face, and the opening scene of him finding a baby on the sea shore is a great piece of writing. Selene is back; she was one of my favorite characters in the first book. And Bear pulls off something I didn't think she could: the Grey Wolf is also back, for the third book, and he actually won me over. I think he grew up a bit; there's at least less whining and angst, and more effective action. It probably also helped that less of the story was told from his perspective. The Sea Thy Mistress is primarily the story of Cathoair and his son, Cathmar, and their encounter with a truly nasty threat: a goddess who has way more knowledge of how their world works than they do, and is out to destroy it. This review will necessarily be sketchy on details; most of them would partly spoil All the Windwracked Stars. (I suppose I'm spoiling it a bit anyway just by listing characters who are still around.) Cathmar is... not too irritating. He's young and stupid, and the first half of the book is a bit painful as he plays into the hands of the plot with much less information than the reader has available. But I was okay with him by the end of the book. Cathoair is a more complicated matter. I suspect whether you like this book will depend mostly on whether you can find some sympathy for his personal angst and self-destructive behavior, or at least tolerate him long enough to get to the ending. Quite a lot of this book consists of people behaving stupidly and horribly while those who love them stand by helplessly, unable to figure out how to break the cycle. If that sounds frustrating, you have the right impression. I really struggled with the first two-thirds of the book and started regretting reading it. But there is a payoff: one that's uncomfortable, raw, perilous, and sad, but also beautiful and honest and effective. Muire's role in this book may be minor, but it's very memorable. Despite the difficulties of the early book, I was very glad to have read it for the payoff. I'm not sure whether to recommend this or not, but I think it's worth reading if you enjoyed All the Windwracked Stars. If you do read it, though, try to read the whole series in close proximity. You'll want to remember the details between books. I'm now somewhat tempted to go back and re-read the whole series to experience it in context. Rating: 6 out of 10

11 September 2015

Daniel Pocock: Payment for shadow work

A significant court ruling yesterday means that companies in the EU have to pay employees for time spent traveling to and from work. If something seems too good to be true, it usually is. Once you get past the headline, you find that it only applies to workers who are going to appointments outside their workplace. Some companies have been careless about scheduling such appointments, sometimes leaving an employee more than 2 hours from his home at the end of the day and this new ruling may just force them to either try a little harder to schedule appointments with consideration for the employee or pay for the extra time if excessive travel sometimes occurs. The ruling may be good news for IT workers, road congestion and the environment. With companies now becoming more efficient in managing travel time, they may well need to improve the software they use for scheduling the employee's time. With certain types of employee potentially having a shorter journey after the last appointment, this means less road congestion and less pollution. Professionals who have to fly may also be taking note: if you work in Paris but have to fly to London one day for meetings, then your working time starts when you leave your home at 05:30 and finishes at 22:00 when you get home. Shadow work comes in many forms It is good to see society taking note of the hidden cost of shadow work. What about all those other nasty and inefficient practices that take time but you don't get paid for them? Have you ever wondered why some companies send bills and statements to you by email or post, but others expect you to waste 5 minutes of your time logging in to their web site to download the bill or statement? Should people be paid for this time too and would that force these companies to stop being so lazy? Some companies have been trying to avoid doing the right thing by arguing that email is not secure, but this just doesn't add up: the original PGP MIME specs for email encryption were published through the IETF process in 1996. Barclaycard recently began offering to send credit card statements using encrypted PDF, proving that many of the excuses given by other companies are just that: excuses. For developers For people who are interested in improving the way their organization sends communications to customers, a popular choice is the Apache Camel integration framework. I've deployed Camel for customer trade notification and other purposes at several large and well known firms in financial services. It has a rich set of features for transforming messages, including a PGP data format for encryption and it is able to send messages out using a vast number of connectivity options, including many of the basic ones like SMTP, SMPP (for SMS), XMPP/Jabber and message queues.

10 September 2015

Antoine Beaupr : Switched to bootstrap theme

I finally gave up and drank the cool-aid of the Bootstrap theme. The main reason was that I noticed the site was basically unusable on any mobile device, including tablet computers like the iPad, which are unfortunately very, very common. Phones wouldn't render the site in a legible way either, unfortunately. The change is rather drastic, so I figured it was important to mention it here. I am not so satisfied pretty exctied with the result: the theme is very basic, if not absent. I actually like that purified form now, and it works across devices now much better. But I do feel I just made my blog look like everyone else that uses Bootstrap. We seem to enter an era where non-graphic designers (like me) are back to building web pages that all look the same, not very different, in a way, from the old days of plain HTML, like the default ikiwiki theme. I did some work to change the look at least minimally, but the top black navbar is really a killer giveaway this is a bootstrap theme. Bootstrap does a lot for the typography, at least when checking the presslabs checklist or the practical typography checklist, but I still had to change the main body width, which helped a lot I think. Anyways, the old Night City theme is still available for download and I could still flip it back on here if I need to.

What changed The new theme keeps the distractions away, but ironically, it's slower than the previous theme even though there are no images and basically no colors at all. On my tests, it now loads in about 3.42 seconds on the "Regular 2G (250Kbps)" simulation of Chromium with 72KB in 12 requests. Whereas my tests with the previous theme were taking 3.25 seconds with 43.5KB in 17 requests. This is due to the Bootstrap CSS, which takes a whopping 19KB itself, but especially because I now load JQuery, which takes a whopping 32.8KB! And that is probably gzip-compressed, as the original is more around 90KB. But at least the thing is readable on phones and tablets now. Compare:

Samsung S3, before Screenshot of a fake rendering on the Samsung Galaxy S3 in night city

Samsung S3, after Screenshot of a fake rendering on the Samsung Galaxy S3 in bootstrap I like this: much simpler, purer version on smaller devices. And we see what counts: the freaking text. Of course, on tablets, it doesn't fare as well: the top menu gets wrapped around...

Apple iPad, before Screenshot of a fake rendering on the iPad in night city

Apple iPad, after Screenshot of a fake rendering on the iPad in bootstrap Apparently, the fix would be for me to "customize the @grid-float-breakpoint variable" in LESS (which I thought was a pager, but seems to be a CSS preprocessor, graaah). Still, I think it's better that way, more readable, and less crufty.

How it was done After doing a thorough evaluation of all the Bootstrap ikiwiki themes I could find (there are more than 4 at least!), I figured I prefered the Jak Linux one. It seemed to be better implemented than the others, cleaner, and I liked the mean black border on top. Black is cool. Oh, and I liked the subtle footer as well. Subtle is cool. Unfortunately, that theme originally required a custom plugin to have a menu on top, which i thought was silly. So I patched it to make it work with the sidebar plugin, which turned out to be trivially simple: just dump the sidebar content, and make the sidebar page have explicit HTML tags in it that match Bootstrap's required classes. I also added the regular action links in the top navbar, but that does overload it quite a bit. I also did some code cleanups and various other small changes, all of which are available in my personnal git repo.

What doesn't work So of course, there are always problems. The first problem is the extra bandwidth usage. Not sure that can be fixed, other than switching to the upcoming Bootstrap 4, which is smaller than Bootstrap 3, bizarrely. The more annoying problems are weird alignment issues. If the screen gets two narrow without kicking some collapse rules in place (the iPad bug above), the navbar rows overflow and look ugly. Even more bizarre, in some cases the right navbar can just completely disappear, in fact, that's the only fix I could find: to assign the hidden-xs and hidden-sm classes to the navbar-right <UL> element. But then it just goes away, which is pretty darn stupid. Still - I assigned the classes to a few actions that seemed low priority, both in the theme and in the sidebar page. Similarly, the search form at the bottom of the page doesn't seem to want to fit in the footer properly. No idea why and I can't bother to figure that one out. I ended up merging this with the backlinks, tags and trails navigation items. Oh, and amazingly enough: comments are simply not rendered at all right now. Oops. I seemed to have picked the single bootstrap Ikiwiki theme that does *not* have comments rendering... Aaaargh. For now I just copy-pasted stuff from the [ramseydsilva][] theme and it looks pretty ugly, but at least it works. I ended up theming comments the same way i did for Night City, which looks pretty good, but is not in sync with the LESS stuff in Bootstrap. I also add to add back trails, backlinks and favicon... looks like the Jak theme wasn't made for a blog after all... but that's now fixed! :) Another improvement would be to change the font from the canonical Helvetica to something prettier, like suggested in Practical Typography, in the font recommendations section, or the presslabs checklist. So far I have settled on Mozilla's Fira font (which means, yes, one set of objects is actually loading from the CDN, sorry). Feedback welcome.

8 September 2015

Dirk Eddelbuettel: It was twenty years ago today...

sgt pepper Hm, wasn't there a catchy tune that started that way? Just kidding. But about twenty years ago today I sent this email about a new Debian package upload -- and that makes it (as best as I can tell) the earliest trace of me doing Debian work. It so happened that I did upload two packages in July of 1995 as well, but it was so early in the project that we did not have a mailing list archive for such things yet (!!). And I have a vague recollection that the two in July were an adoption---whereas the post referenced above announced efax as my first new package added to the distribution. And there were more to come ... Anyway, thanks for all the fish! Twenty years with Debian has been a great, great ride. I learned a lot from a lot of fantastic people, and I hope I helped a few people along the way with a package or two I still maintain. Debian remains a truly fabulous project which I hope will go strongly for another 20 (or even 22).

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Eriberto Mota: Debian: how to use blhc to solve hardening issues when packaging

Implementing the hardening When packaging in Debian, is very common to see the lintian messages 'hardening-no-relro' and 'hardening-no-fortify-functions' in some softwares written in C or C++. To solve these issues, we can use the 'blhc' tool (apt-get install blhc). Please, get the revision 1.11-9 of the icmpinfo package. You can get this revision from http://snapshot.debian.org or from http://eriberto.pro.br/debian/icmpinfo. As a shortcut, you can use the following command:
$ dget -u http://eriberto.pro.br/debian/icmpinfo/icmpinfo_1.11-9.dsc
The icmpinfo 1.11-9 is almost clean for lintian (in 2015-09-07, Standards-Version 3.9.6). The only problem is:
W: icmpinfo: hardening-no-relro usr/sbin/icmpinfo
To track the problem I will use blhc over the .build file:
$ blhc icmpinfo_1.11-9_amd64.build
LDFLAGS missing (-Wl,-z,relro): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -o icmpinfo recvping.o print.o err.o icmpinfo.o pid.o
Note that the problem is some missing options (-Wl,-z,relro) for LDFLAGS when building icmpinfo (for newbies, in GCC, -o is used to indicate the name to be used for the final binary after the compilation). If you are using the DebHelper compat 9 (debian/compat=9) and the DebHelper 9 (debhelper >= 9 in Build-Depends field in d/control), some variables as CFLAGS, LDFLAGS, CPPFLAGS and CXXFLAGS will be automatically passed during calls to dh_auto_* programs (yes, you should use the new and reduced d/rules format - see as example the debian/rules of the icmpinfo 1.11-9; if you still have doubts, $ man dh). Now, we need discover the reason why the LDFLAGS is being changed between its generation by the Debian build system and its utilization by the upstream's source code. So, we need to check the upstream Makefile. There is in Makefile (after a 'quilt push -a', to apply all current patches):
LDFLAGS= $(CFLAGS)
OBJECTS= recvping.o print.o err.o icmpinfo.o pid.o
TARGET = icmpinfo
$(TARGET): $(OBJECTS)
 $(CC) $(LDFLAGS) -o $@ $(OBJECTS) $(LDLIBS)
Hummm... The LDFLAGS content generated by Debian is being dropped by Makefile because it is saying that "LDFLAGS = CFLAGS content". This line is a problem because the upstream Makefile needs to take and use the CFLAGS and LDFLAGS independently. To fix the issue, you can use this patch:
--- icmpinfo-1.11.orig/Makefile
+++ icmpinfo-1.11/Makefile
@@ -20,13 +20,13 @@ VERS = 1.11
 
 RM = rm -f
 
-LDFLAGS= $(CFLAGS)
+#LDFLAGS= $(CFLAGS)
 
 OBJECTS= recvping.o print.o err.o icmpinfo.o pid.o
 TARGET = icmpinfo
 
 $(TARGET): $(OBJECTS)
- $(CC) $(LDFLAGS) -o $@ $(OBJECTS) $(LDLIBS)
+ $(CC) $(LDFLAGS) $(CFLAGS) -o $@ $(OBJECTS) $(LDLIBS)
 
 tgz: clean
 rm -f CHECKSUMS.asc
After a 'debuild' is a fact the problem is solved and the lintian is happy. See the blhc results:
$ blhc ../icmpinfo_1.11-9_amd64.build
$
Now, we can improve the hardening. To see the current status, we can use the 'blhc --all' command. See here:
blhc --all ../icmpinfo_1.11-9_amd64.build
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o recvping.o recvping.c
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o print.o print.c
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o err.o err.c
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o icmpinfo.o icmpinfo.c
CFLAGS missing (-fPIE): cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -c -o pid.o pid.c
LDFLAGS missing (-fPIE -pie -Wl,-z,now): cc -Wl,-z,relro -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -o icmpinfo recvping.o print.o err.o icmpinfo.o pid.o
Well, we know that CFLAGS and LDFLAGS are present. Now, we can force the DebHelper to pass some extra options to make hardening better. Generally, is only needed to add the following line to debian/rules:
export DEB_BUILD_MAINT_OPTIONS = hardening=+all
See the results (after a new debuild command):
$ blhc --all ../icmpinfo_1.11-9_amd64.build
$
More examples Let me to show other example. I will use the mac-robber 1.02-3 (however, I disabled the Makefile.patch in debian/patches/series). After a debuild, the following lintian messages are presented:
W: mac-robber: hardening-no-relro usr/bin/mac-robber
I: mac-robber: hardening-no-fortify-functions usr/bin/mac-robber
Using blhc:
$ blhc ../mac-robber_1.02-3_amd64.build 
CFLAGS missing (-g -O2 -fstack-protector-strong -Wformat -Werror=format-security): gcc -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
CPPFLAGS missing (-D_FORTIFY_SOURCE=2): gcc -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
LDFLAGS missing (-Wl,-z,relro): gcc -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
We need to verify what is the problem in Makefile with CFLAGS, CPPFLAGS and LDFLAGS when generating the binary 'mac-robber' (just recalling, -o mac-robber in GCC command). See:
linux_notstatic: 
 $(CC) -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
There are no references to CFLAGS, CPPFLAGS and LDFLAGS. To solve the problem, we need patch the Makefile to make this:
linux_notstatic: 
 $(CC) $(CFLAGS) $(LDFLAGS) $(CPPFLAGS) -D_FILE_OFFSET_BITS=64 -o mac-robber mac-robber.c
As last example, is possible that the Makefile is overriding the content sent by DebHelper when building. See this line from a hypothetical Makefile:
CFLAGS = -g -Wall
As you can see, the Makefile is redefining CFLAGS; consequently, it is discarding any previous content sent by DebHelper. To solve this issue, we can use the following patch:
-CFLAGS = -g -Wall
+CFLAGS += -g -Wall
So, the content received from DebHelper will be added to '-g -Wall'. Default parameters As curiosity, to see the basic parameters created by DebHelper as hardening, use the command:
$ dpkg-buildflags
To see the all parameters, use the command:
$ DEB_BUILD_MAINT_OPTIONS=hardening=+all dpkg-buildflags
More information More information about the hardening can be viewed at two places: https://wiki.debian.org/Hardening https://wiki.debian.org/HardeningWalkthrough I hope this help. Enjoy!

5 September 2015

Christoph Berg: 10 Years Debian Developer

I knew it was about this time of the year 10 years ago when my Debian account was created, but I couldn't remember the exact date until I looked it up earlier this evening: today :). Rene Engelhard had been my advocate, and Marc Brockschmidt my AM. Thanks guys! A lot of time has passed since then, and I've worked in various parts of the project. I became an application manager almost immediately, and quickly got into the NM front desk as well, revamping parts of the NM process which had become pretty bureaucratic (I think we are now, 10 years later, back where we should be, thanks to almost all of the paperwork being automated, thanks Enrico!). I've processed 37 NMs, most of them between 2005 and 2008, later I was only active as front desk and eventually Debian account manager. I've recently picked up AMing again, which I still find quite refreshing as the AM will always also learn new things. Quality Assurance was and is the other big field. Starting by doing QA uploads of orphaned packages, I attended some QA meetings around Germany, and picked up maintenance of the DDPO pages, which I still maintain. The link between QA and NM is the MIA team where I was active for some years until they kindly kicked me out because I was MIA there myself. I'm glad they are still using some of the scripts I was writing to automate some things. My favorite MUA is mutt, of which I became co-maintainer in 2007, and later maintainer. I'm still listed in the uploaders field, but admittedly I haven't really done anything there lately. Also in 2007 I started working at credativ, after having been a research assistant at the university, which meant making my Debian work professional. Of course it also meant more real work and less time for the hobby part, but I was still very active around that time. Later in 2010 I was marrying, and we got two kids, at which point family was of course much more important, so my Debian involvement dropped to a minimum. (Mostly lurking on IRC ;) Being a PostgreSQL consultant at work, it was natural to start looking into the packaging, so I started submitting patches to postgresql-common in 2011, and became a co-maintainer in 2012. Since then, I've mostly been working on PostgreSQL-related packages, of which far too many have my (co-)maintainer stamp on them. To link the Debian and PostgreSQL worlds together, we started an external repository (apt.postgresql.org) that contains packages for the PostgreSQL major releases that Debian doesn't ship. Most of my open source time at the moment is spent on getting all PostgreSQL packages in shape for Debian and this repository. According to minechangelogs, currently 844 changelog entries in Debian mention my name, or were authored by me. Scrolling back yields memories of packages that are long gone again from unstable, or I passed on to other maintainers. There are way too many people in Debian that I enjoy(ed) working with to list them here, and many of them are my friends. Debian is really the extended family on the internet. My last DebConf before this year had been in Mar del Plata - I had met some people at other conferences like FOSDEM, but meeting (almost) everyone again in Heidelberg was very nice. I even remembered all basic Mao rules :D. So, thanks to everyone out there for making Debian such a wonderful place to be!

Niels Thykier: I accidentally dak

So, yesterday, I unbroke dak twice even! It is of course slightly less awesome that one of the broken parts was in code written by yours truly. Anyhow: Unbreaking the dak auto-decrufter You may remember the auto-decrufter, which I added to dak. As a safety measure, it bails out when in doubt about which removal breaks what package. Turns out it was often in doubt, because the code had a bug. Of course, nothing that could not be solved with a patch. Thanks to Ansgar for merging this. :) Unbreaking dak generate-releases As a part of migrating apt-file to use APTs new acquire system (from experimental), I learned APT really likes having checksums for everything. Now including checksums for both the compressed file and the uncompressed file. Sadly, dak had optimised out the uncompressed checksums for Contents files, but even after removing that optimisation (and Ganneff unbreaking my dinstall breakage) some Contents files still did not have an checksum for the uncompressed Contents file. After some sophisticated debugging (read: printf-debugging ), I finally discovered the issue and submitted a patch. Thanks to Ansgar and Ganneff for merging (and fixing my dinstall breakage).
Filed under: Debian

4 September 2015

Norbert Preining: How to rename a local OfflineIMAP managed folder

Since quite some time I am using OfflineIMAP to keep my mailbox status from various servers locally available on my machine. I did more or less follow the excellent guide A unix style mail setup, but didn t use folder name translation right from the beginning. So I got stuck with four folders named INBOX: acc1/INBOX, acc2/INBOX, etc. This wouldn t be a big problem, but when using Mutt with the folder sidebar patch (as available in the mutt-patched package in Debian), there is the problem that only the INBOX part is shown. offlineimap So I decided to rename all the local folders to the scheme described in the above guide. At the end I want to have a folder layout as follows:
  Maildir/acc1/acc1
  Maildir/acc1/acc1.folder1
  Maildir/acc1/acc1.folder2
where the acc1 related to the INBOX on the remote account acc1, and the others are folders on the remote end. Unfortunately, renaming is not supported in offlineimap, it would simply download everything again at best! So I checked what needs to be done, and it turned out it is not as complicated as I thought. The necessary steps are: To make this more specific, let us assume I want to reorganize the account acc1 in this way, and there are two folders synced. That is, before we start the Maildir folder contains:
Maildir/acc1/INBOX
Maildir/acc1/folder1
Maildir/acc1/folder2
and we want to achieve that the layout is
Maildir/acc1/acc1
Maildir/acc1/acc1.folder1
Maildir/acc1/acc1.folder2
Due to the fact that both the LocalStatus files and the FolderValidity files are naned in the same way as the folders, the following trivial script does all the necessary renaming for this simple case:
for d in ~/Maildir/acc1 ~/.offlineimap/Account-acc1/LocalStatus ~/.offlineimap/Repository-acc1local/FolderValidity ; do
  mv $d/INBOX $d/acc1
  mv $d/folder1 $d/acc1.folder1
  mv $d/folder2 $d/acc1.folder2
done
Finally, the correct renaming has to be done by adding nametrans rules to the helper script. The functions oimaptransfolder_acc1 and localtransfolder_acc1 as defined in the above tutorial can be used as is, and need only be added to the respective Repository sections in the configuration file. After having done the necessary renaming, what remains is to give it a good test. Best way to do it is with
offlineimap --dry-run -o -a acc1
which runs the syncronization in dry-run mode, so nothing is actually copied, and only once and only for acc1. If during this test run nothing strange appears, like many many emails are deleted on either end, then you are fine. Enjoy.

3 September 2015

John Goerzen: There s still a chance to save WiFi

You may not know it, but wifi is under assault in the USA due to proposed FCC regulations about modifications to devices with modular radios. In short, it would make it illegal for vendors to sell devices with firmware that users can replace. This is of concern to everyone, because Wifi routers are notoriously buggy and insecure. It is also of special concern to amateur radio hobbyists, due to the use of these devices in the Amateur Radio Service (FCC Part 97). I submitted a comment to the FCC about this, which I am pasting in here. This provides a background and summary of the issues for those that are interested. Here it is: My comment has two parts: one, the impact on the Amateur Radio service; and two, the impact on security. Both pertain primarily to the 802.11 ( Wifi ) services typically operating under Part 15. The Amateur Radio Service (FCC part 97) has long been recognized by the FCC and Congress as important to the nation. Through it, amateurs contribute to scientific knowledge, learn skills that bolster the technological competitiveness of the United States, and save lives through their extensive involvement in disaster response. Certain segments of the 2.4GHz and 5GHz Wifi bands authorized under FCC Part 15 also fall under the frequencies available to licensed amateurs under FCC Part 97 [1]. By scrupulously following the Part 97 regulations, many amateur radio operators are taking devices originally designed for Part 15 use and modifying them for legal use under the Part 97 Amateur Radio Service. Although the uses are varied, much effort is being devoted to fault-tolerant mesh networks that provide high-speed multimedia communications in response to a disaster, even without the presence of any traditional infrastructure or Internet backbone. One such effort [2] causes users to replace the firmware on off-the-shelf Part 15 Wifi devices, reconfiguring them for proper authorized use under Part 97. This project has many vital properties, particularly the self-discovery of routes between nodes and self-healing nature of the mesh network. These features are not typically available in the firmware of normal Part 15 devices. It should also be noted that there is presently no vendor of Wifi devices that operate under Part 97 out of the box. The only route available to amateurs is to take Part 15 devices and modify them for Part 97 use. Amateur radio users of these services have been working for years to make sure they do not cause interference to Part 15 users [3]. One such effort takes advantage of the modular radio features of consumer Wifi gear to enable communication on frequencies that are within the Part 97 allocation, but outside (though adjacent) to the Part 15 allocation. For instance, the chart at [1] identifies frequencies such as 2.397GHz or 5.660GHz that will never cause interference to Part 15 users because they lie entirely outside the Part 15 Wifi allocation. If the FCC prevents the ability of consumers to modify the firmware of these devices, the following negative consequences will necessarily follow: 1) The use of high-speed multimedia or mesh networks in the Amateur Radio service will be sharply curtailed, relegated to only outdated hardware. 2) Interference between the Amateur Radio service which may use higher power or antennas with higher gain and Part 15 users will be expanded, because Amateur Radio service users will no longer be able to intentionally select frequencies that avoid Part 15. 3) The culture of inventiveness surrounding wireless communication will be curtailed in an important way. Besides the impact on the Part 97 Amateur Radio Service, I also wish to comment on the impact to end-user security. There have been a terrible slew of high-profile situations where very popular consumer Wifi devices have had incredible security holes. Vendors have often been unwilling to address these issues [4]. Michael Horowitz maintains a website tracking security bugs in consumer wifi routers [5]. Sadly these bugs are both severe and commonplace. Within just the last month, various popular routers have been found vulnerable to remote hacking [6] and platforms for launching Distributed Denial-of-Service (DDoS) attacks [7]. These impacted multiple models from multiple vendors. To make matters worse, most of these issues should have never happened in the first place, and were largely the result of carelessness or cluelessness on the part of manufacturers. Consumers should not be at the mercy of vendors to fix their own networks, nor should they be required to trust unauditable systems. There are many well-regarded efforts to provide better firmware for Wifi devices, which still keep them operating under Part 15 restrictions. One is OpenWRT [8], which supports a wide variety of devices with a system built upon a solid Linux base. Please keep control of our devices in the hands of consumers and amateurs, for the benefit of all. References: [1] https://en.wikipedia.org/wiki/High-speed_multimedia_radio [2] http://www.broadband-hamnet.org/just-starting-read-this.html [3] http://www.qsl.net/kb9mwr/projects/wireless/modify.html [4] http://www.tomsitpro.com/articles/netgear-routers-security-hole,1-2461.html [5] http://routersecurity.org/bugs.php [6] https://www.kb.cert.org/vuls/id/950576 [7] https://www.stateoftheinternet.com/resources-cloud-security-2015-q2-web-security-report.html [8] https://openwrt.org/

1 September 2015

Lunar: Reproducible builds: week 18 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Aur lien Jarno uploaded glibc/2.21-0experimental1 which will fix the issue were locales-all did not behave exactly like locales despite having it in the Provides field. Lunar rebased the pu/reproducible_builds branch for dpkg on top of the released 1.18.2. This made visible an issue with udebs and automatically generated debug packages. The summary from the meeting at DebConf15 between ftpmasters, dpkg mainatainers and reproducible builds folks has been posted to the revelant mailing lists. Packages fixed The following 70 packages became reproducible due to changes in their build dependencies: activemq-activeio, async-http-client, classworlds, clirr, compress-lzf, dbus-c++, felix-bundlerepository, felix-framework, felix-gogo-command, felix-gogo-runtime, felix-gogo-shell, felix-main, felix-shell-tui, felix-shell, findbugs-bcel, gco, gdebi, gecode, geronimo-ejb-3.2-spec, git-repair, gmetric4j, gs-collections, hawtbuf, hawtdispatch, jack-tools, jackson-dataformat-cbor, jackson-dataformat-yaml, jackson-module-jaxb-annotations, jmxetric, json-simple, kryo-serializers, lhapdf, libccrtp, libclaw, libcommoncpp2, libftdi1, libjboss-marshalling-java, libmimic, libphysfs, libxstream-java, limereg, maven-debian-helper, maven-filtering, maven-invoker, mochiweb, mongo-java-driver, mqtt-client, netty-3.9, openhft-chronicle-queue, openhft-compiler, openhft-lang, pavucontrol, plexus-ant-factory, plexus-archiver, plexus-bsh-factory, plexus-cdc, plexus-classworlds2, plexus-component-metadata, plexus-container-default, plexus-io, pytone, scolasync, sisu-ioc, snappy-java, spatial4j-0.4, tika, treeline, wss4j, xtalk, zshdb. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: Chris Lamb also noticed that binaries shipped with libsilo-bin did not work. Documentation update Chris Lamb and Ximin Luo assembled a proper specification for SOURCE_DATE_EPOCH in the hope to convince more upstreams to adopt it. Thanks to Holger it is published under a non-Debian domain name. Lunar documented easiest way to solve issues with file ordering and timestamps in tarballs that came with tar/1.28-1. Some examples on how to use SOURCE_DATE_EPOCH have been improved to support systems without GNU date. reproducible.debian.net armhf is finally being tested, which also means the remote building of Debian packages finally works! This paves the way to perform the tests on even more architectures and doing variations on CPU and date. Some packages even produce the same binary Arch:all packages on different architectures (1, 2). (h01ger) Tests for FreeBSD are finally running. (h01ger) As it seems the gcc5 transition has cooled off, we schedule sid more often than testing again on amd64. (h01ger) disorderfs has been built and installed on all build nodes (amd64 and armhf). One issue related to permissions for root and unpriviliged users needs to be solved before disorderfs can be used on reproducible.debian.net. (h01ger) strip-nondeterminism Version 0.011-1 has been released on August 29th. The new version updates dh_strip_nondeterminism to match recent changes in debhelper. (Andrew Ayer) disorderfs disorderfs, the new FUSE filesystem to ease testing of filesystem-related variations, is now almost ready to be used. Version 0.2.0 adds support for extended attributes. Since then Andrew Ayer also added support to reverse directory entries instead of shuffling them, and arbitrary padding to the number of blocks used by files. Package reviews 142 reviews have been removed, 48 added and 259 updated this week. Santiago Vila renamed the not_using_dh_builddeb issue into varying_mtimes_in_data_tar_gz_or_control_tar_gz to align better with other tag names. New issue identified this week: random_order_in_python_doit_completion. 37 FTBFS issues have been reported by Chris West (Faux) and Chris Lamb. Misc. h01ger gave a talk at FrOSCon on August 23rd. Recordings are already online. These reports are being reviewed and enhanced every week by many people hanging out on #debian-reproducible. Huge thanks!

29 August 2015

Francois Marier: Letting someone ssh into your laptop using Pagekite

In order to investigate a bug I was running into, I recently had to give my colleague ssh access to my laptop behind a firewall. The easiest way I found to do this was to create an account for him on my laptop and setup a pagekite frontend on my Linode server and a pagekite backend on my laptop.

Frontend setup Setting up my Linode server in order to make the ssh service accessible and proxy the traffic to my laptop was fairly straightforward. First, I had to install the pagekite package (already in Debian and Ubuntu) and open up a port on my firewall by adding the following to both /etc/network/iptables.up.rules and /etc/network/ip6tables.up.rules:
-A INPUT -p tcp --dport 10022 -j ACCEPT
Then I created a new CNAME for my server in DNS:
pagekite.fmarier.org.   3600    IN  CNAME   fmarier.org.
With that in place, I started the pagekite frontend using this command:
pagekite --clean --isfrontend --rawports=virtual --ports=10022 --domain=raw:pagekite.fmarier.org:Password1

Backend setup After installing the pagekite and openssh-server packages on my laptop and creating a new user account:
adduser roc
I used this command to connect my laptop to the pagekite frontend:
pagekite --clean --frontend=pagekite.fmarier.org:10022 --service_on=raw/22:pagekite.fmarier.org:localhost:22:Password1

Client setup Finally, my colleague needed to add the folowing entry to ~/.ssh/config:
Host pagekite.fmarier.org
  CheckHostIP no
  ProxyCommand /bin/nc -X connect -x %h:10022 %h %p
and install the netcat-openbsd package since other versions of netcat don't work. On Fedora, we used netcat-openbsd-1.89 successfully, but this newer package may also work. He was then able to ssh into my laptop via ssh roc@pagekite.fmarier.org.

Making settings permanent I was quite happy settings things up temporarily on the command-line, but it's also possible to persist these settings and to make both the pagekite frontend and backend start up automatically at boot. See the documentation for how to do this on Debian and Fedora.

19 August 2015

Russell Coker: The Purpose of a Code of Conduct

On a private mailing list there have been some recent discussions about a Code of Conduct which demonstrate some great misunderstandings. The misunderstandings don t seem particular to that list so it s worthy of a blog post. Also people tend to think more about what they do when their actions will be exposed to a wider audience so hopefully people who read this post will think before they respond. Jokes The first discussion concerned the issue of making jokes . When dealing with the treatment of other people (particularly minority groups) the issue of jokes is a common one. It s fairly common for people in positions of power to make jokes about people with less power and then complain if someone disapproves. The more extreme examples of this concern hate words which are strongly associated with violence, one of the most common is a word used to describe gay men which has often been associated with significant violence and murder. Men who are straight and who conform to the stereotypes of straight men don t have much to fear from that word while men who aren t straight will associate it with a death threat and tend not to find any amusement in it. Most minority groups have words that are known to be associated with hate crimes. When such words are used they usually send a signal that the minority groups in question aren t welcome. The exception is when the words are used by other members of the group in question. For example if I was walking past a biker bar and heard someone call out geek or nerd I would be a little nervous (even though geeks/nerds have faced much less violence than most minority groups). But at a Linux conference my reaction would be very different. As a general rule you shouldn t use any word that has a history of being used to attack any minority group other than one that you are a member of, so black rappers get to use a word that was historically used by white slave-owners but because I m white I don t get to sing along to their music. As an aside we had a discussion about such rap lyrics on the Linux Users of Victoria mailing list some time ago, hopefully most people think I m stating the obvious here but some people need a clear explanation. One thing that people should consider jokes is the issue of punching-down vs punching-up [1] (there are many posts about this topic, I linked to the first Google hit which seems quite good). The basic concept is that making jokes about more powerful people or organisations is brave while making jokes about less powerful people is cowardly and serves to continue the exclusion of marginalised people. When I raised this issue in the mailing list discussion a group of men immediately complained that they might be bullied by lots of less powerful people making jokes about them. One problem here is that powerful people tend to be very thin skinned due to the fact that people are usually nice to them. While the imaginary scenario of less powerful people making jokes about rich white men might be unpleasant if it happened in person, it wouldn t compare to the experience of less powerful people who are the target of repeated jokes in addition to all manner of other bad treatment. Another problem is that the impact of a joke depends on the power of the person who makes it, EG if your boss makes a joke about you then you have to work on your CV, if a colleague or subordinate makes a joke then you can often ignore it. Who does a Code of Conduct Protect One member of the mailing list wrote a long and very earnest message about his belief that the CoC was designed to protect him from off-topic discussions. He analysed the results of a CoC on that basis and determined that it had failed due to the number of off-topic messages on the mailing lists he subscribes to. Being so self-centered is strongly correlated with being in a position of power, he seems to sincerely believe that everything should be about him, that he is entitled to all manner of protection and that any rule which doesn t protect him is worthless. I believe that the purpose of all laws and regulations should be to protect those who are less powerful, the more powerful people can usually protect themselves. The benefit that powerful people receive from being part of a system that is based on rules is that organisations (clubs, societies, companies, governments, etc) can become larger and achieve greater things if people can trust in the system. When minority groups are discouraged from contributing and when people need to be concerned about protecting themselves from attack the scope of an organisation is reduced. When there is a certain minimum standard of treatment that people can expect then they will be more willing to contribute and more able to concentrate on their contributions when they don t expect to be attacked. The Public Interest When an organisation declares itself to be acting in the public interest (EG by including Public Interest in the name of the organisation) I think that we should expect even better treatment of minority groups. One might argue that a corporation should protect members of minority groups for the sole purpose of making more money (it has been proven that more diverse groups produce better quality work). But an organisation that s in the Public Interest should be expected to go way beyond that and protect members of minority groups as a matter of principle. When an organisation is declared to be operating in the Public Interest I believe that anyone who s so unable to control their bigotry that they can t refrain from being bigoted on the mailing lists should not be a member.

18 August 2015

Enrico Zini: standup-comedy-notes

My semi serious stand up comedy notes Video Disclaimers
Someone has said that it requires less mental effort to condemn than to think. (Emma Goldman, on several things including mailing list flamewars)
Fascinating A da's "Dogging" song. Look for "dogging etiquette" for more examples of code of conducts. Just don't take your computer for repair immediately afterwards . Introduction
Every daring attempt to make a great change in existing conditions, every lofty vision of new possibilities for the human race, has been labeled Utopian. (Emma Goldman, on the Debian Social Contract)
I am going to talk about many topics that we all know have so much in common: They are all, after all: BDSM
A person is no less a slave because they are allowed to choose a new master once in a term of years. (Lysander Spooner about proprietary cloud service providers)
If you thought you've seen it all with recursive acronyms, here's a chain acronym: Bondage Discipline, Dominance Submission, Sado Masochism. Why I think BDSM is interesting: not (just) because of whips, but for having a lot of awareness about power releationships. Why should one accept from a coworker a level of abuse that would be considered a hard limit when negotiating with a trusted dom? The BDSM Free Software definition: "I refuse to be bound by software I cannot negotiate with". YKINMKBYKIOK (Your Kink Is Not My Kink But Your Kink Is Okay) is a nice example of dealing with diversity, and it also definitely solves the emacs vs vi debate. Comfort zones, safewords, traffic light flow control, safety. "No means no", unless there has been a long discussion first, and a safeword is in place, in which case "Fuzzy purple unicorn" means "no" "No means no", and if someone insists after a "no", it becomes harassment. "No means no" is a precondition for being able to say "yes": http://pervocracy.blogspot.de/2011/03/no-and-no-and-no-and-yes.html Aftercare! Aftercare! Release parties! High fives! Solidarity after flamewars or votes! Poliamory
If love does not know how to give and take without restrictions, it is not love, but a transaction that never fails to lay stress on a plus and a minus. (Emma Goldman, on volunteer projects)
Polyamory is the practice, desire, or acceptance of intimate relationships that are not exclusive with respect to other sexual or intimate relationships, with knowledge and consent of everyone involved. Compersion, n: the feeling you get when someone else also takes good care of one of your packages. We currently allow only one value in the Maintainer field: * takeover is traumatic, because values can only be replaced * if values could be added instead, and removed when they don't make sense anymore... What is your definition of love? My current one is: my world is better with you in it. Relationship anarchy is the practice of forming relationships which are not bound by rules aside from what the people involved mutually agree on. How do you call a relationship that is bound by rules that the people involved do not agree on? From discussions after the talk New Relationship Energy, the excitement when you start to maintain a new package, and the risk of been carried away by the excitement and neglecting all the other ones. Consent
Anarchism, to me, means not only the denial of authority, not only a new economy, but a revision of the principles of morality. It means the development of the individual as well as the assertion of the individual. It means self-responsibility, and not leader worship. (Voltairine de Cleyre about trusting lintian warnings)
You need to know what you are doing, and what situation you're putting yourself into. You need to know that the person asking a question really is able to accept any answer, and take it seriously. You need to feel that you have alternatives. Be selfish when you ask, honest when you reply, and when others reply, take them seriously. If any of this doesn't stand, I find it hard to trust that we are in a consensual situation. When is one supposed to learn about consent? Consent explained with tea. Practical advice
Anarchism has but one infallible, unchangeable motto, Freedom. Freedom to discover any truth, freedom to develop, to live naturally and fully. (Lucy Parsons about the DFSG)
Relationship advice and work advice have a lot in common: Relationship advice from 99 ways to ruin an open source project Online participation advice from How to Screw Up Your Relationship (and make everyone miserable while you re at it) Packaging advice from BDSM Basics: 20 Unsolicited Tips for New Dominants Advice about joining a new community from Advice to a newbie submissive about dominants Dear Debian, and dear everyone contributing to it: my world is better with you in it. I love you all :* <3

16 August 2015

Lunar: Reproducible builds: week 16 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Valentin Lorentz sent a patch for ispell to initialize memory structures before dumping their content. In our experimental repository, qt4-x11 has been rebased on the latest version (Dhole), as was doxygen (akira). Packages fixed The following packages became reproducible due to changes in their build dependencies: backup-manager, cheese, coinor-csdp, coinor-dylp, ebook-speaker, freefem, indent, libjbcrypt-java, qtquick1-opensource-src, ruby-coffee-script, ruby-distribution, schroot, twittering-mode. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: akira found another embedded code copy of texi2html in maxima. reproducible.debian.net Work on testing several architectures has continued. (Mattia/h01ger) Package reviews 29 reviews have been removed, 187 added and 34 updated this week. 172 new FTBFS reports were filled, 137 solely by Chris West (Faux). josch spent time investigating the issue with fonts in PDF files. Chris Lamb documented the issue affecting documentation generated by ocamldoc. Misc. Lunar presented a general Reproducible builds HOWTO talk at the Chaos Communication Camp 2015 in Germany on August 13th. Recordings are already available, as well as slides and script. h01ger and Lunar also used CCCamp15 as an opportunity to have discussions with members of several different projects about reproducible builds. Good news should be coming soon.

14 August 2015

Dirk Eddelbuettel: Rblpapi: Connecting R to Bloomberg

big deal Whit, John and I are thrilled to announce Rblapi, a new CRAN package which connects R to the Bloomberg backends. Rebuilt from scratch using only the Bloomberg C++ API and the Rcpp and BH packages, it offers efficient and direct access from R to a truly vast number of financial data series, pricing tools and more. The package has been tested on Windows, OS X and Linux. As is standard for CRAN packages, binaries for Windows and OS X are provided (or will be once the builders caught up). Needless to say, a working Bloomberg installation is required to use the package. Please see the Rblapi package page for more details, including a large part of the introductory vignette document. As a teaser, here are just three of the core functions:
## Bloomberg Data Point Query
bdp(c("ESA Index", "SPY US Equity"), c("PX_LAST", "VOLUME"))
## Bloomberg Data Set Query
bds("GOOG US Equity", "TOP_20_HOLDERS_PUBLIC_FILINGS")
## Bloomberg Data History Query
bdh("SPY US Equity", c("PX_LAST", "VOLUME"), start.date=Sys.Date()-31)
## Get OHLCV bars (by default hourly and just six of them)
getBars("ES1 Index")
## Get Tick Data (by default last hour)
getTicks("ES1 Index")
Source code for the package is at the Rblpapi GitHub repo where issue tickets can be filed as well. The sibbling blp GitHub repo contains the Bloomberg code required to build and link the package (which is automated during the build of the CRAN package). Last but not least the Rblpapi package page has more details about the package.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

12 August 2015

Daniel Pocock: Recording live events like a pro (part 2: video)

In the first blog in this series, we looked at how to capture audio effectively for a range of different events. While recording audio appears less complicated than video, it remains fundamental to a good recording. For some types of event, like a speech or a debate, you can have quite a bad video with poor lighting and other defects but people will still be able to watch it if the audio is good. Therefore, if you haven't already looked at the previous blog, please do so now. As mentioned in the earlier blog, many people now have high quality equipment for recording both audio and video and a wide range of opportunities to use it, whether it is a talk at a conference, a wedding or making a Kickstarter video. The right camera for video recording The fundamental piece of equipment is the camera itself. You may have a DSLR camera that can record video or maybe you have a proper handheld video camera. The leading DSLR cameras, combined with a good lens, make higher quality recordings than many handheld video cameras. Unfortunately, although you pay good money to buy a very well engineered DSLR that could easily record many hours of video, most DSLRs are crippled to record a maximum of 30 minutes in one recording. This issue and some workarounds are discussed later in this blog. If you don't have any camera at all you need to think carefully about which type to buy. If you are only going to use it once you may want to consider renting or borrowing or asking for other people attending the event to bring whatever cameras they have to help make multiple recordings (the crowdsourcing solution). If you are a very keen photographer then you will probably have a preference for a DSLR. Accessories Don't just look at the cost of your camera and conclude that is all the budget you need. For professional quality video recording, you will almost certainly need some accessories. You may find they are overpriced at the retail store where you bought your camera, but you still need some of them, so have a look online.
Recording a talk at a free software event with a Nikon D800 on a very basic tripod with Rode VideoMic Pro, headphones (white cable) and external power (black cable) If you want to capture audio with the camera and record it in the video file (discussed in more detail below), you will need to purchase a microphone that mounts on the camera. The built-in microphones on cameras are often quite bad, even on the most expensive cameras. If you are just using the built-in microphone for reference audio (to help with time synchronization when you combine different audio files with the video later) then the built-in microphone may be acceptable. Camera audio is discussed in more detail below. If your camera has a headphone socket, get some headphones for listening to the audio. Make sure you have multiple memory cards. Look carefully at the speed of the memory cards, slow ones are cheaper but they can't keep up with the speed of writing 1080p video. At a minimum, you should aim to buy memory cards that can handle one or two days worth of data for whatever it is you do. A tripod is essential for most types of video. If you use a particularly heavy camera or lens or if you are outdoors and it may be windy you will need a heavier tripod for stability. For video, it is very useful to have a tripod with a handle for panning left and right but if the camera will be stationary for the whole recording then the handle is not essential. Carrying at least one spare battery is another smart move. On one visit to the Inca Trail in Peru, we observed another member of our group hiking up and down the Andes with a brand new DSLR that they couldn't use because the battery was flat. For extended periods of recording, batteries will not be sufficient and you will need to purchase a mains power supply (PSU). These are available for most types of DSLR and video camera. The camera vendors typically design cameras with unusual power sockets so that you can only use a very specific and heavily overpriced PSU from the same company. Don't forget a surge protector too. There are various smartphone apps that allow you to remotely control the camera from the screen of your phone, such as the qDslrDashboard app. These often give a better preview than the screen built-in to the camera and may even allow you to use the touch screen to focus more quickly on a specific part of the picture. A regular USB cable is not suitable for this type of app, you need to buy a USB On-The-Go (OTG) cable.
Screenshot of qDslrDashboard app on a smartphone, controlling a DSLR camera If you plan to copy the video from the camera to a computer at the event, you will need to make sure you have a fast memory card reader. The memory card readers in some laptops are quite slow and others can be quite fast so you may not need to buy an extra card reader. Camera audio Most cameras, including DSLRs, have a built-in microphone and a socket for connecting an external microphone. The built-in microphones obtain very poor quality sound. For many events, it is much better to have independent microphones, such as a lapel microphone attached to a smartphone or wireless transmitter. Those solutions are described in part one of this blog series. Nonetheless, there are still some benefits of capturing audio in the camera. The biggest benefit is the time synchronization: if you have audio recordings in other devices, you will need to align them with the video using post-production software. If the camera recorded an audio stream too, even if the quality is not very good, you can visualize the waveform on screen and use it to align the other audio recordings much more easily and precisely. If the camera will be very close to the people speaking then it may be acceptable to use a microphone mounted on the camera. This will be convenient for post-production because the audio will be synchronized with the video. It may still not be as good as a lapel microphone though, but the quality of these camera-mounted microphones is still far higher than the built-in microphones. I've been trying the Rode VideoMic Pro, it is definitely better than recording with the built-in microphone on the camera and also better than the built-in microphone on my phone. One problem that most people encounter is the sound of the lens autofocus mechanism being detected by the microphone. This occurs with both the built-in microphone and any other microphone you mount on the camera. A microphone mounted on top of the camera doesn't detect this noise with the same intensity as the built-in microphone but it is still present in the recordings. If using a camera-mounted microphone to detect the audio from an audience, you may need to have an omnidirectional microphone. Many camera-mounted microphones are directional and will not detect very much sound from the sides or behind the camera. When using any type of external microphone with the camera, it is recommend to disable automatic gain control (AGC) in the camera settings and then manually adjust the microphone sensitivity/volume level. Use headphones A final word on audio - most good cameras have an audio output socket. Connect headphones and wear them, to make sure you are always capturing audio. Otherwise, if the microphone's battery goes flat or if a wireless microphone goes out of range you may not notice. Choosing a lens The more light you get, the better. Bigger and more expensive lenses allow more light into the camera. Many of the normal lenses sold with a DSLR camera are acceptable but if it is a special occasion you may want to rent a more expensive lens for the day. If you already have a lens, it is a very good idea to test it in conditions similar to those you expect for the event you want to record. Recording duration limits Most DSLR cameras with video capability impose a 30 minute maximum recording duration This is basically the result of a friendly arrangement between movie studios and politicians to charge an extra tax on video recording technology and potentially make the movie studio bosses richer, supposedly justified by the fact that a tiny but exaggerated number of people use their cameras to record movies at the cinema. As a consequence, most DSLR manufacturers limit the duration of video recording so their product won't be subject to the tax, ensuring the retail price is lower and more attractive. On top of this, many DSLR cameras also have a 4GB file size limit if they use the FAT filesystem. Recording 1080p video at a high frame rate may hit the file size limit in 10 minutes, well before you encounter the 30 minute maximum recording duration. To deal with the file size issue, you can record at 720p instead of 1080p and use the frame rate 24fps. For events longer than 30 minutes or where you really want 1080p or a higher frame rate, there are some other options you can consider:
  • Buy or rent a proper video camera instead of using a DSLR camera
  • Using multiple cameras that can stop and be restarted at different times.
  • Manually stopping and restarting the camera if there are breaks in the event where it is safe to do so.
  • Use an app to control the camera and program it to immediately restart the recording each time it stops
  • Extract the raw output from the camera's HDMI socket and record into some other device or computer. There are several purpose-built devices
    that can be used this way with an embedded SSD for storage.
  • There are also some people distributing unofficial/alternative firmware images that remove the artificial 30 minute recording limit.
Camera settings There are many online tutorials and demonstration videos on YouTube that will help you optimize the camera settings for video. You may have already made recordings using the automatic mode. Adjusting some or all of the settings manually may help you create a more optimal recording. You will need to spend some time familiarizing yourself with the settings first. The first thing to check is white balance. This tells the camera the type of lighting in the location. If you set this incorrectly then the colours will be distorted. Many cameras have the ability to automatically set the white balance. For video, you may be able to change one or more settings that control the recording quality. These settings control the file compression ratio and image size. Typical image size settings are 720p and 1080p. Compression ratio may be controlled by a high/medium/low quality setting. Choosing the highest quality and biggest picture requires more space on the memory card and also means you reach the 4GB file size limit more quickly. A higher quality setting also implies a faster memory card is required, because the rate of megabytes per second written to the memory card is higher. Next you need to think about the frame rate. Events that involve fast moving subjects, such as sports, typically benefit from a higher frame rate. For other events it is quite acceptable to use 24 frames per second (fps). Higher frame rates also imply bigger file size and a requirement for a faster memory card. Once you have decided on the frame rate, the next thing to do is set the shutter speed. Use a shutter speed that is double the frame rate. For example, if using 24fps or 25fps, use a 1/50 shutter speed. The final two settings you need to adjust are the ISO and aperture. Set these based on the lighting conditions and extent to which the subjects are moving. For example, if the setting is dark or if you are trying to record fast moving subjects like athletes, vehicles or animals, use an ISO value of 800 or higher. Once you have chosen ISO, adjust the aperture to ensure the picture is sufficiently illuminated. Aperture also has a significant impact on the depth of field. Operating the camera: zoom and focus Many people use zoom lenses. It is not always necessary to change the zoom while recording a video, you can use software to zoom in and out on individual parts of the picture when editing it in post-production. If you do change the zoom while recording, it may be more difficult to maintain focus. Almost all lenses support manual focus (turning the focus ring by hand) and many support automatic focus. When shooting photographs with a DSLR, the mirror is down and the camera can use dedicated sensors for focus and light sensing. When shooting video, the mirror is up and the camera can not use the same focus sensors that are used in photography. Video recording uses a digital focussing algorithm based on contrast in the picture. If you take a lot of photos you are probably quite accustomed to the fast and precise autofocus for photography and you will notice that keeping a video in focus is more challenging. As mentioned already, one of the first things you can do to keep focus simple is to avoid zooming while recording. Record in a higher resolution than you require and then zoom with software later. Some people record using 4k resolution even when they only want to produce a 720p video, as they can digitally zoom in to different parts of the 4k recording without losing detail. If the subject is very stationary (people sitting at a desk for an interview is a typical example), you may be able to set the camera to manual focus and not change it at all while recording. If you choose to enable autofocus while recording, any built-in camera microphone or microphone mounted on the camera is likely to detect sounds from the motorized focus system. Ultimately, the autofocus mechanism is not accurate for all subjects and you may be unable to stop them moving around so you will need to change the focus manually while recording. It requires some practice to be able to do this quickly without overshooting the right focus. To make life more tricky, Nikon and Canon focus rings rotate in the opposite direction, so if you are proficient using one brand you may feel awkward if you ever have to use the other. A good way to practice this skill is to practice while in the car or on the train, pointing at different subjects outside the window and trying to stay in focus as you move from one subject to the next. Make a trial run Many events, from weddings right up to the Olympic Games opening ceremony, have a trial run the day before. One reason for that is to test the locations and settings of all the recording and broadcasting equipment. If a trial run isn't possible for your event, you may find some similar event to practice recording and test your equipment. For example, if you are planning to record a wedding, you could try and record a Sunday mass in the same church. Backup and duplicate the recordings before leaving the event If you only have one copy of the recordings and the equipment is stolen or damaged you may be very disappointed. Before your event, make a plan to duplicate the raw audio and video recordings so that several people can take copies away with them. Decide in advance who will be responsible for this, ensure there will be several portable hard disks and estimate how much time it will take to prepare the copies and factor this into the schedule. Conclusion All the products described can be easily purchased from online retailers. You may not need every accessory that is mentioned as it depends on the type of event you record. The total cost of buying or renting the necessary accessories may be as much as the cost of the camera itself so if you are new to this you may need to think carefully about making a budget with a spreadsheet to do it correctly. Becoming familiar with the camera controls and practicing the techniques for manual focus and zoom can take weeks or months. If you enjoy photography this can be time well spent but if you don't enjoy it then you may not want to commit the time necessary to make good quality video. Don't rule out options like renting equipment instead of buying it or crowdsourcing, asking several participants or friends to help make recordings with their own equipment. For many events, audio is far more indispensable than video and as emphasizing at the beginning of this article, it is recommended that you should be one hundred percent confident in your strategy for recording audio before you start planning to record video.

10 August 2015

Norbert Preining: Debian/TeX Live 2015.20150810-1

For those who are starving for some updates while the big gcc5 transitions brings the rest of Debian/sid to a halt, here is some fresh meat, a new TeX Live checkout. Nothing spectacular new here, just the usual big bunch of updates of and several new packages. Maybe worthwhile mentioning is that luasseq has been reincorporated into the TeX Live packages. Thanks to the maintainer for his work till now! Debian - TeX Live 2015 From the long list of changes let me pick one update and one new package: pgf has been updated to 3.0.1, which is just a small change in number, but incorporates 1.5 years of fixes, and the list of fixes is long. And a very interesting newcomer is pdfpagediff by C. V. Rad hakr ish nan. It allows visual comparison of two pdf files by overlaying them in a transparent way, making even slight changes in the layout immediately visible. Of course, these are not the only updates and newcomers, see the list below. Updated packages academicons, addlines, algorithms, archaic, babel, babel-estonian, biber, biblatex-source-division, bidi, bxjscls, chronology, cjk-ko, conteq, csplain, csquotes, curve2e, datatool, doclicense, dvips, eledmac, enotez, esami, eso-pic, etex-pkg, etoolbox, exsheets, fandol, fithesis, fontawesome, fontspec, forest, glossaries, greek-inputenc, jslectureplanner, kotex-oblivoir, kotex-utf, l3build, l3experimental, l3kernel, l3packages, latex, leadsheets, ledmac, lshort-english, luamplib, luasseq, luatexko, maths-symbols, media9, memoir, metrix, mhchem, minitoc, moderncv, modiagram, morefloats, mptopdf, msu-thesis, nameauth, ndsu-thesis, newpx, newtx, newtxsf, newtxtt, notes, ocgx2, pageslts, pdfpages, pdftex, pgf, phonrule, plain, polyglossia, pstricks, ptex, pxchfon, reflectgraphics, rsfso, sectionbox, siunitx, skrapport, standalone, tcolorbox, tetex, tex4ht, texfot, texinfo, texlive-scripts, todonotes, translations, tudscr, ucbthesis, unicode-math, xcharter, xgreek. New packages alertmessage, bewerbung, bidihl, bxpdfver, cloze, comicneue, copyedit, fcavtex, gradstudentresume, make4ht, mcf2graph, multiaudience, nmbib, pdfpagediff, quran, reledmac, roundrect, screenplay-pkg, shapes, tex4ebook. Enjoy.

9 August 2015

Hideki Yamane: xz multithreaded compression

$ dpkg -s liblzma5
Package: liblzma5
Status: install ok installed
(snip)
Version: 5.1.1alpha+20120614-2.1
$ time sudo cowbuilder --build fonts-horai-umefont_530-1.dsc
real 1m8.378s
user 1m0.084s
sys 0m3.244s
real 1m9.842s
user 1m0.964s
sys 0m3.856s
real 1m9.355s
user 1m0.824s
sys 0m3.380s

xz 5.2 supports multithread, with local build (remove --disable-thread option in debian/rules)

$ dpkg -s liblzma5
Package: liblzma5
Status: install ok installed
(snip)
Version: 5.2.1-0.1
real 1m7.946s
user 0m59.628s
sys 0m3.228s
real 1m8.794s
user 1m0.072s
sys 0m3.556s
real 1m7.580s
user 0m59.152s
sys 0m3.368s

Umm, not affected.

$ time xz -T 0 fonts-horai-umefont_530.orig.tar
real 0m15.488s
user 0m47.044s
sys 0m0.252s
$ time xz -T 1 fonts-horai-umefont_530.orig.tar
real 0m40.988s
user 0m40.804s
sys 0m0.124s
multithread itself is enabled.

3 August 2015

Sven Hoexter: Failing with F5: CMP - Clustered Multiprocessing

The last Whiteboard Wednesday tackled a few of the gotchas of the F5 CMP implementation. I take that as an opportunity to finish a blog post I started to write six month ago on this topic. I've run into two scenarios where CMP got in my way: All of this happened on TMOS 11.4.1+HF run on BigIP 2K systems. Since F5 is sometimes moving very quickly you might not experience those issues on other hardware or with a different software release. applying low and strict connection limits Apparently connection limits are counted on a per tmm level and are not shared between the tmm processes running on different CPUs. So if you've two CPU cores with HT you'll see four tmm processes that will all enforce the connection limit. If you have a max connection limit set to two you might still see a maximum of eight connections. That is a very bad thing if you abuse the F5 to limit your outbound connections because your counter part is not able to enforce connection limits, but forces you via a contract to not open more then two connections. Disable CMP and it will all end on the same tmm process and suddenly your connection limits work as expected. If your connection limit is big enough you can cheat, say it's twelve connections, you can configure a limit of three. With four processes that makes a max of twelve connections as required. Though due to the hashing and distribution among the tmm processes it could happen that you reject connections on one process while others are idle. Another option would be to enter the dirty land of iRules and tables to maintain your own connection count table. There are examples given in the F5 devcentral community, but the best resolution you can get, without starting to scan the whole table on every request, is X requests per minute. active health checks Active monitoring checks were added to detect the operational state of our application servers. Now imagine you have some realtime processing with over a hundred requests per minute. So in case you've issues with one instance you'd like to remove it from the pool rather sooner then later, which requires frequent checking. So here is a simple active http check:
defaults-from http-appmon
destination *:*
interval 9
recv "^HTTP/1\\.1 200 OK"
send "GET /mon/state HTTP/1.1\\r\\nHost: MON\\r\\nConnection: Close\\r\\n\\r\\n"
time-until-up 0
timeout 2
up-interval 1
(Configuration options are described here) With an up-interval of 1 second and a timeout of 2 seconds someone naive like me expected that a fatal issue like a crashed application would be detected within three seconds and no more requests should hit the application after second 3. That turned out to be slightly longer, at least 4s for this case, often longer. Also for a regular maintenance we had to instruct our scripts to flip the application monitor to unavailable and wait for interval X tmm process count + timeout + 1s + max allowed request duration. That is not an instant off. If you use tcpdump (on the F5 where it's patched to show you the tmm that received the request) to look at the health check traffic you'll notice that the tmm threads pass around the handle to do a health check. So my guess is the threads do only sync the health check state of the application when they're due to run the health check. Makes sense performance wise but if you operate with longer timeouts and bigger intervals you will definetly loose a notable amount of requests. Even in our low end case we already loose a notable amount of requests in 4s. While you could resent them you never know exactly if a request was already processed or not. So you have to ensure that somewhere else in your stack. One option is to disable CMP but I guess that's not an option for most people that are required to scale out to handle the load and not only for the sake of redundancy. If you're running the balancer only for the sake of redundancy the choice is easy. You find a hint on this kind of issue if you lookup the documentation for the inband monitor: Note: Systems with multiple tmm processes use a per-process number to calculate failures, depending on the specified load balancing method. That case is a bit different due to the nature of the monitor but points in the same direction. The actual check is executed by bigd according to this document, so I'm pretty sure I'm still missing some pieces of the picture. Just be warned that you've to really test your outage scenarios and you should double check it if you run a transaction based service that is sensitive to lost requests.

1 August 2015

Russ Allbery: Review: The Pyramid Waltz

Review: The Pyramid Waltz, by Barbara Ann Wright
Series: Katya and Starbride #1
Publisher: Bold Strokes
Copyright: September 2012
ISBN: 1-60282-792-3
Format: Kindle
Pages: 264
Princess Katya Nar Umbriel is publicly a bored, womanizing, and difficult daughter to the rulers of Farraday. It's all an act, though, with the full knowledge of her parents. As the second child, she's the leader of the Order of Vestra: the equivalent of the Secret Service, devoted to protecting the royal family and, by extension, the kingdom, particularly against magical attacks. Starbride is new to court and entirely out of place. From a northern neighboring country, and far more comfortable in practical clothing than the frilled court dresses that her mother wants her to wear, she has been sent to court to make contacts. Her people are getting the bad side of various trade contracts and desperately need some political maneuvering space of their own. Starbride's best hope for this is to study law in the palace library when she can manage to avoid the other courtiers. But then she and Katya stumble across each other, outside of the roles they're playing, and might have an opportunity for a deeper connection. One that neither of them want to entangle in their personal worries. This is the last of a set of books I picked up while looking for lesbian romance with fantasy or science fiction elements. On the romance front, it's one of the better entries in that set. Both Katya and Starbride are likeable, in large part due to their mutual exasperation with the trappings of the court. (Making the protagonists more serious, thoughtful, and intelligent than the surrounding characters is an old trick, but it works.) Wright has a good ear for banter, particularly the kind when two people of good will are carefully feeling each other out. And despite Katya's need to keep a deep secret from Starbride for some of the book, The Pyramid Waltz mostly avoids irritating communication failures as a plot driver. The fantasy portion and the plot drivers, alas, are weaker. The world building is not exactly bad, but it's just not that interesting. There are a couple of moderately good ideas, in the form of pyramid magic and secret (and dangerous) magical powers that run in the royal family, but they're not well-developed. Pyramid magic turns out to look much like any other generic fantasy magic system, with training scenes that could have come from a Valdemar or Wheel of Time novel (and without as much dramatic tension). And the royal family's secret, while better-developed and integral to the plot, still felt rather generic and one-sided. Maybe that's something Wright develops better in future novels in this series, but that was another problem: the ending of The Pyramid Waltz was rather weak. Partly, I think, this is because the cast is too large and not well-developed. I cared about Katya and Starbird, and to a lesser extent their servants and one of the Order members. (Wright has a moderately interesting bit of worldbuilding about how servants work in Starbride's culture, which I wish we'd seen more of.) But there are a bunch of other Order of Vesta members, Katya's family, and various other bits of history and hinted world views, none of which seemed to get much depth. The ending climax involved a lot of revelations and twists that primarily concerned characters I didn't care about. It lost something in the process. This book is clearly set up for a sequel. There is an ending, but it's not entirely satisfying. Unfortunately, despite liking Katya and Starbird a lot, the rest of the story wasn't compelling enough to make me want to buy it, particularly since the series apparently goes through another three books before reaching a real ending. I enjoyed parts of this book, particularly Katya and Starbird feeling each other out and discovering similarities in their outlook. Katya teasing Starbird, and Starbird teasing herself, over her mother's choice of her clothing was probably the best part. It's not bad for what it's trying to do, but I think it's a bit too generic and not satisfying enough to really recommend. Followed by For Want of a Fiend. Rating: 6 out of 10

Next.

Previous.