Search Results: "pm"

20 August 2015

Axel Beckert: German-written Debian Package Management Book

Thursday was our big day: After more than 2.5 years of working in the hidden, ups and downs, Frank Hofmann and myself were able to announce the availability of our book project Debian Package Management under a free license (Creative Commons Attribution ShareAlike 4.0 International License, short CC BY-SA 4.0 ) during a Lightning Talk at DebConf15 in Heidelberg. This became possible because we found Onyx Neon, a publishing company which is specialised on books with contents under free licenses. Its founder does not only have a faible for Perl but also for Debian. (Since the question already came up: We also thought about self-publishing, e.g. via Lulu or Epubli and it would have been our fallback solution , but we prefer the professionalism and services of a real publisher. I m though happy to share what I found out about self-publishing in the past few months.) The source code of the book is written in the AsciiDoc format and available on GitHub. The book is still work in progress. But if you want, you can already build an e-book out of the publically available source code:
sudo apt-get install asciidoc dblatex git
git clone git://github.com/dpmb/dpmb.git
cd dpmb
make
(Works fine on Debian 7 Wheezy, Debian 8 Jessie and Ubuntu 14.04 LTS Trusty. Does not work on Ubuntu 12.04 LTS Precise.) If you find an error in the book, please file an issue on GitHub. If you also know how to fix the error, please for the Git repository on GitHub, fix the error in your Git repository and file a pull request. (The first pull request already happenend and has been applied.) Initially there will be only a German written issue as e-book (at least in HTML, PDF and EPUB formats, maybe also KF8/MOBI and EPUB3) and at some point in the future also as printed book at Onyx Neon. But we re also planning a translation to English as well as a Debian package. If your want to get informed when we publish a printed book, a translation or an official e-book release, please subscribe to one of our mailing lists: There s one in German and one in English.

Simon Kainz: vim in Heidelberg

Following the tradition of Love Locks, apparently there is someone really in love with vim in Heidelberg! Valerie Found at the Old Bridge in Heidelberg during DebConf15.

18 August 2015

Enrico Zini: standup-comedy-notes

My semi serious stand up comedy notes Video Disclaimers
Someone has said that it requires less mental effort to condemn than to think. (Emma Goldman, on several things including mailing list flamewars)
Fascinating A da's "Dogging" song. Look for "dogging etiquette" for more examples of code of conducts. Just don't take your computer for repair immediately afterwards . Introduction
Every daring attempt to make a great change in existing conditions, every lofty vision of new possibilities for the human race, has been labeled Utopian. (Emma Goldman, on the Debian Social Contract)
I am going to talk about many topics that we all know have so much in common: They are all, after all: BDSM
A person is no less a slave because they are allowed to choose a new master once in a term of years. (Lysander Spooner about proprietary cloud service providers)
If you thought you've seen it all with recursive acronyms, here's a chain acronym: Bondage Discipline, Dominance Submission, Sado Masochism. Why I think BDSM is interesting: not (just) because of whips, but for having a lot of awareness about power releationships. Why should one accept from a coworker a level of abuse that would be considered a hard limit when negotiating with a trusted dom? The BDSM Free Software definition: "I refuse to be bound by software I cannot negotiate with". YKINMKBYKIOK (Your Kink Is Not My Kink But Your Kink Is Okay) is a nice example of dealing with diversity, and it also definitely solves the emacs vs vi debate. Comfort zones, safewords, traffic light flow control, safety. "No means no", unless there has been a long discussion first, and a safeword is in place, in which case "Fuzzy purple unicorn" means "no" "No means no", and if someone insists after a "no", it becomes harassment. "No means no" is a precondition for being able to say "yes": http://pervocracy.blogspot.de/2011/03/no-and-no-and-no-and-yes.html Aftercare! Aftercare! Release parties! High fives! Solidarity after flamewars or votes! Poliamory
If love does not know how to give and take without restrictions, it is not love, but a transaction that never fails to lay stress on a plus and a minus. (Emma Goldman, on volunteer projects)
Polyamory is the practice, desire, or acceptance of intimate relationships that are not exclusive with respect to other sexual or intimate relationships, with knowledge and consent of everyone involved. Compersion, n: the feeling you get when someone else also takes good care of one of your packages. We currently allow only one value in the Maintainer field: * takeover is traumatic, because values can only be replaced * if values could be added instead, and removed when they don't make sense anymore... What is your definition of love? My current one is: my world is better with you in it. Relationship anarchy is the practice of forming relationships which are not bound by rules aside from what the people involved mutually agree on. How do you call a relationship that is bound by rules that the people involved do not agree on? From discussions after the talk New Relationship Energy, the excitement when you start to maintain a new package, and the risk of been carried away by the excitement and neglecting all the other ones. Consent
Anarchism, to me, means not only the denial of authority, not only a new economy, but a revision of the principles of morality. It means the development of the individual as well as the assertion of the individual. It means self-responsibility, and not leader worship. (Voltairine de Cleyre about trusting lintian warnings)
You need to know what you are doing, and what situation you're putting yourself into. You need to know that the person asking a question really is able to accept any answer, and take it seriously. You need to feel that you have alternatives. Be selfish when you ask, honest when you reply, and when others reply, take them seriously. If any of this doesn't stand, I find it hard to trust that we are in a consensual situation. When is one supposed to learn about consent? Consent explained with tea. Practical advice
Anarchism has but one infallible, unchangeable motto, Freedom. Freedom to discover any truth, freedom to develop, to live naturally and fully. (Lucy Parsons about the DFSG)
Relationship advice and work advice have a lot in common: Relationship advice from 99 ways to ruin an open source project Online participation advice from How to Screw Up Your Relationship (and make everyone miserable while you re at it) Packaging advice from BDSM Basics: 20 Unsolicited Tips for New Dominants Advice about joining a new community from Advice to a newbie submissive about dominants Dear Debian, and dear everyone contributing to it: my world is better with you in it. I love you all :* <3

Russell Coker: BTRFS Training

Some years ago Barwon South Water gave LUV 3 old 1RU Sun servers for any use related to free software. We gave one of those servers to the Canberra makerlab and another is used as the server for the LUV mailing lists and web site and the 3rd server was put aside for training. The servers have hot-swap 15,000rpm SAS disks IE disks that have a replacement cost greater than the budget we have for hardware. As we were given a spare 70G disk (and a 140G disk can replace a 70G disk) the LUV server has 2*70G disks and the 140G disks (which can t be replaced) are in the server for training. On Saturday I ran a BTRFS and ZFS training session for the LUV Beginners SIG. This was inspired by the amount of discussion of those filesystems on the mailing list and the amount of interest when we have lectures on those topics. The training went well, the meeting was better attended than most Beginners SIG meetings and the people who attended it seemed to enjoy it. One thing that I will do better in future is clearly documenting commands that are expected to fail and documenting how to login to the system. The users all logged in to accounts on a Xen server and then ssh d to root at their DomU. I think that it would have saved a bit of time if I had aliased commands like btrfs to echo you must login to your virtual server first or made the shell prompt at the Dom0 include instructions to login to the DomU. Each user or group had a virtual machine. The server has 32G of RAM and I ran 14 virtual servers that each had 2G of RAM. In retrospect I should have configured fewer servers and asked people to work in groups, that would allow more RAM for each virtual server and also more RAM for the Dom0. The Dom0 was running a BTRFS RAID-1 filesystem and each virtual machine had a snapshot of the block devices from my master image for the training. Performance was quite good initially as the OS image was shared and fit into cache. But when many users were corrupting and scrubbing filesystems performance became very poor. The disks performed well (sustaining over 100 writes per second) but that s not much when shared between 14 active users. The ZFS part of the tutorial was based on RAID-Z (I didn t use RAID-5/6 in BTRFS because it s not ready to use and didn t use RAID-1 in ZFS because most people want RAID-Z). Each user had 5*4G virtual disks (2 for the OS and 3 for BTRFS and ZFS testing). By the end of the training session there was about 76G of storage used in the filesystem (including the space used by the OS for the Dom0), so each user had something like 5G of unique data. We are now considering what other training we can run on that server. I m thinking of running training on DNS and email. Suggestions for other topics would be appreciated. For training that s not disk intensive we could run many more than 14 virtual machines, 60 or more should be possible. Below are the notes from the BTRFS part of the training, anyone could do this on their own if they substitute 2 empty partitions for /dev/xvdd and /dev/xvde. On a Debian/Jessie system all that you need to do to get ready for this is to install the btrfs-tools package. Note that this does have some risk if you make a typo. An advantage of doing this sort of thing in a virtual machine is that there s no possibility of breaking things that matter.
  1. Making the filesystem
    1. Make the filesystem, this makes a filesystem that spans 2 devices (note you must use the-f option if there was already a filesystem on those devices):
      mkfs.btrfs /dev/xvdd /dev/xvde
    2. Use file(1) to see basic data from the superblocks:
      file -s /dev/xvdd /dev/xvde
    3. Mount the filesystem (can mount either block device, the kernel knows they belong together):
      mount /dev/xvdd /mnt/tmp
    4. See a BTRFS df of the filesystem, shows what type of RAID is used:
      btrfs filesystem df /mnt/tmp
    5. See more information about FS device use:
      btrfs filesystem show /mnt/tmp
    6. Balance the filesystem to change it to RAID-1 and verify the change, note that some parts of the filesystem were single and RAID-0 before this change):
      btrfs balance start -dconvert=raid1 -mconvert=raid1 -sconvert=raid1 force /mnt/tmp
      btrfs filesystem df /mnt/tmp
    7. See if there are any errors, shouldn t be any (yet):
      btrfs device stats /mnt/tmp
    8. Copy some files to the filesystem:
      cp -r /usr /mnt/tmp
    9. Check the filesystem for basic consistency (only checks checksums):
      btrfs scrub start -B -d /mnt/tmp
  2. Online corruption
    1. Corrupt the filesystem:
      dd if=/dev/zero of=/dev/xvdd bs=1024k count=2000 seek=50
    2. Scrub again, should give a warning about errors:
      btrfs scrub start -B /mnt/tmp
    3. Check error count:
      btrfs device stats /mnt/tmp
    4. Corrupt it again:
      dd if=/dev/zero of=/dev/xvdd bs=1024k count=2000 seek=50
    5. Unmount it:
      umount /mnt/tmp
    6. In another terminal follow the kernel log:
      tail -f /var/log/kern.log
    7. Mount it again and observe it correcting errors on mount:
      mount /dev/xvdd /mnt/tmp
    8. Run a diff, observe kernel error messages and observe that diff reports no file differences:
      diff -ru /usr /mnt/tmp/usr/
    9. Run another scrub, this will probably correct some errors which weren t discovered by diff:
      btrfs scrub start -B -d /mnt/tmp
  3. Offline corruption
    1. Umount the filesystem, corrupt the start, then try mounting it again which will fail because the superblocks were wiped:
      umount /mnt/tmp
      dd if=/dev/zero of=/dev/xvdd bs=1024k count=200
      mount /dev/xvdd /mnt/tmp
      mount /dev/xvde /mnt/tmp
    2. Note that the filesystem was not mountable due to a lack of a superblock. It might be possible to recover from this but that s more advanced so we will restore the RAID.
      Mount the filesystem in a degraded RAID mode, this allows full operation.
      mount /dev/xvde /mnt/tmp -o degraded
    3. Add /dev/xvdd back to the RAID:
      btrfs device add /dev/xvdd /mnt/tmp
    4. Show the filesystem devices, observe that xvdd is listed twice, the missing device and the one that was just added:
      btrfs filesystem show /mnt/tmp
    5. Remove the missing device and observe the change:
      btrfs device delete missing /mnt/tmp
      btrfs filesystem show /mnt/tmp
    6. Balance the filesystem, not sure this is necessary but it s good practice to do it when in doubt:
      btrfs balance start /mnt/tmp
    7. Umount and mount it, note that the degraded option is not needed:
      umount /mnt/tmp
      mount /dev/xvdd /mnt/tmp
  4. Experiment
    1. Experiment with the btrfs subvolume create and btrfs subvolume delete commands (which act like mkdir and rmdir).
    2. Experiment with btrfs subvolume snapshot SOURCE DEST and btrfs subvolume snapshot -r SOURCE DEST for creating regular and read-only snapshots of other subvolumes (including the root).

17 August 2015

Iustin Pop: Nikkor 300mm f/4E tests and happy dogs!

I recently got a new lens, which I'm still trying to learn how to properly use . The new lens is the Nikon 300m f/4E PF ED VR (wow, that's a mouthful of acronyms ) and it's as good as I expected, except that it seems that it's too good for the person using it (me). Physically, the lens is indeed very light and very easy to hand-hold; it's barely heavier and longer than the 70-300mm f/4-5.6 zoom, although a bit more bulky. I was fearing something "bigger" and heavier for a fixed-aperture lens, even after hearing all the good reports about it, but it was better than I expected. I was also slightly concerned, this being a prime lens, about finding the subject fast, especially if smaller or far away. It turned out that if the lens is focused at somewhat the right distance (not at the opposite end), it is not a real problem. So, as the first test/learning exercise, I did a couple of hour long walks outdoor, photographing random subjects. First day I used the lens by itself, on the second day I added the Nikon TC-14E(III) tele-converter, for a 420mm focal length. All pictures below except the Huskies are non-cropped, just down-sampled (they have the original field of view); the Husky pictures were cropped. In all cases, camera was set to semi-manual mode, mostly at 1/1000s or 1/2000s, various apertures between f/4 and f/8; this means wide open or stopped down one (when with the tele-converter) or two stops. Semi-manual as auto-ISO was on. Lens focus limiter on, VR either active or sport (to help with the viewfinder image mostly, at this speed), camera set to AF-C and group mode most of the time. Also, all pictures were shot hand-held, with bad technique - until somebody stopped me, turns out he was a photographer, and showed me how to properly hold a telephoto lens; thanks, whoever you were! The "random subjects" parts worked only somewhat, as (at first) the subjects were not too interesting; air-plane flying high above - check, fast focus (although it is easy to focus on a plane in clear sky), or close, static subjects: Airplane in the sky Test of close focusing Birds flying around - almost check, fast focus (both with just the lens and with the TC), although I learned that shooting flying birds while panning needs 1/2000s - my initial attempt at 1/1000s was not enough to eliminate either panning blur or wing-tip blur (both depend on the speed of the respective action); in any case, initial focus acquisition is speedy enough for me. A couple of examples (first without, second with the TC; in the second one, I was unprepared when the I saw the bird from the corner of my eye, and managed to bring up the camera and take exactly one photo before the bird was gone): Not enough speed Too much blur Another attempt was to focus on dogs jumping in the water. Here the problem was different: focusing on the correct object! The splashing around meant that there were lots of water droplets both in front and behind the dog, the dog being partially obscured by them. I don't know what AF settings would have helped here (maybe 3D tracking pre-focused on the black dog)? I had a low success rate on the first attempt, so I'll have to try again. The lens did correctly track the wrong thing, though . Examples: Almost focused Wrong focus Once the dog caught its target, the lens focused well enough, of course, but not perfect; it might need some AF fine-tuning or photographer upgrade : Not quite perfect focus At one point, I was lucky to have a dog running towards me; focus tracking on a subject approaching the camera is a good test for the lens (and camera) auto-focus performance. I was pleasantly surprised, given my failure to track left-to-right flying birds (which should be easier!): Run! Not yet tired! In any case, I was walking and thinking on how to improve on my technique, when I saw in the distance a pair of dogs (Husky's, I believe) which were very active. This case was relatively easy for the camera and lens, since they were moving only left-to-right, instead of forward-backward, so the focus distance was mostly constant, and I just had to pan. Full gallery here, two examples: Jump! Play over, going home These two dogs had an education value beyond the "how is the lens performing"; they were so fast, active (they kept going at it non-stop for a good number of minutes) and playful that they made my day brighter . So, that concludes my first test of this lens and the 1.4 tele-converter. Conclusion: equipment good, need to upgrade the photographer!

14 August 2015

Neil Williams: The problem of SD mux

I keep being asked about automated bootloader testing and the phrase which crops up is SD mux hardware to multiplex SD card access (typically microSD). Each time, the questioner comes up with a simple solution which can be built over a weekend, so I ve decided to write out the actual objective, requirements and constraints to hopefully illustrate that this is not a simple problem and the solution needs to be designed to a fully scalable, reliable and maintainable standard. The objective Support bootloader testing by allowing fully automated tests to write a custom, patched, bootloader to the principal boot media of a test device, hard reset the board and automatically recover if the bootloader fails to boot the device by switching the media from the test device to a known working support device with full write access to overwrite everything on the card and write a known working bootloader. The environment 100 test devices, one SD mux each (potentially), in a single lab with support for all or any to be switched simultaneously and repeatedly (maybe a hundred times a day to and fro) with 99.99% reliability. The history First attempt was a simplistic solution which failed to operate reliably. Next attempt was a complex solution (LMP) which failed to operate as designed in a production environment (partially due to a reliance on USB) and has since suffered from a lack of maintenance. The most recent attempt was another simplistic solution which delivered three devices for test with only one usable and even that became unreliable in testing. The requirements (None of these are negotiable and all are born from real bugs or real failures of previous solutions in the above environment.)
  1. Ethernet yes, really. Physical, cat5/6 RJ45 big, bulky, ugly gigabit ethernet port. No wifi. This is not about design elegance, this is about scalability, maintenance and reliability. Must have a fully working TCP/IP stack with stable and reliable DHCP client. Stable, predictable, unique MAC addresses for every single board guaranteed. No dynamic MAC addresses, no hard coded MAC addresses which cannot be modified. Once modified, retain permanence of the required MAC address across reboots.
  2. No USB involement yes, really. The server writing to the media to recover a bricked device usually has only 2 USB ports but supports 20 devices. Powered hubs are not sufficiently reliable.
  3. Removable media only eMMC sounds useful but these are prototype development boards and some are already known to intermittently fry SD card controller chips causing permanent and irreversible damage to the SD card. If that happened to eMMC, the entire device would have to be discarded.
  4. Cable connections to the test device. This is a solved problem, the cables already exist due to the second attempt at a fix for this problem which resulted in a serviceable design for just the required cables. Do not consider any fixed connection, the height of the connector will never match all test device requirements and will be a constant source of errors when devices are moved around within the rack.
  5. Guaranteed unique, permanent and stable serial numbers for every device. With 100 devices in a lab, it is absolutely necessary that every single one is uniquely addressable.
  6. Interrogation there must be an interface for the control device to query the status of the SD mux and be assured that the results reflect reality at all times. The device must allow the control device to read and write to the media without requiring the test device to acknowledge the switch or even be powered on.
  7. No feature creep. There is no need to make this be able to switch ethernet or HDMI or GPIO as well as SD. Follow the software principle of pick one job and do it properly.
  8. Design for scalability this is not a hobbyist project, this is a serious task requiring genuine design. The problem is not simple, it is not acceptable to make a simple solution.
  9. Power the device must boot directly from power-on without requiring manual intervention of any kind and boot into a default safe mode where the media is only accessible to the control device. 5V power with a barrel connector is preferable definitely not power over USB. Device must raise the TCP/IP control interface automatically and be prepared to react to commands immediately that the interface is available.
  10. Software: some logic to prevent queued requests from causing repeated switching without any interval in between, e.g. if the device had to be power cycled.
  11. Ongoing support and maintenance of hardware, firmware and software. Test devices continue to develop and will require further changes or fixes as time goes on.
  12. Mounting holes sounds obvious but the board needs to be mounted in a sensible manner. Dangling off the end of a cat5 cable is not acceptable.
If any of those seem insurmountable or awkward or unappealing, please go back to the drawing board or leave well alone. Beyond the absolutes, there are other elements. The device is likely to need some kind of CPU and something ARM would be preferable, Cortex-M or Cortex-A if relevant, but creating a cape for a beaglebone-black is likely to be overkill. The available cables are short and the device will need to sit quite close to the test device. Test devices never put the SD card slot in the same place twice or in any location which is particularly accessible. Wherever possible, the components on the device should be commodity parts, replaceable and serviceable. The device would be best not densely populated there is no need for the device to be any particular size and overly small boards tend to be awkward to position correctly once cables are connected. There are limits, of course, so boards which end up bigger than typical test devices would seem excessive. So these are the reasons why we don t have automated bootloader testing and won t have it any time soon. If you ve got this far, maybe there is a design which meets all the criteria so contact me and let s see if this is a fixable problem after all.

13 August 2015

Matthew Palmer: Your "Infrastructure as Code" is still code!

Whether you re a TDD zealot, or you just occasionally write a quick script to reproduce some bug, it s a rare coder who doesn t see value in some sort of automated testing. Yet, somehow, in all of the new-age Infrastructure as Code mania, we appear to have forgotten this, and the tools that are commonly used for implementing Infrastructure as Code have absolutely woeful support for developing your Infrastructure Code. I believe this has to change. At present, the state of the art in testing system automation code appears to be, spin up a test system, run the manifest/state/whatever, and then use something like serverspec or testinfra to SSH in and make sure everything looks OK . It s automated, at least, but it isn t exactly a quick process. Many people don t even apply that degree of rigour to their system config systems, and rely on manual testing, or even just doing it live! , to shake out the bugs they ve introduced. Speed in testing is essential. As the edit-build-test-debug cycle gets longer, frustration grows exponentially. If it takes two minutes to get a something went wrong report out of my tests, I m not going to run them very often. If I m not running my tests very often, then I m not likely to write tests much, and suddenly oops. Everything s on fire. In traditional software development, the unit tests are the backbone of the fast feedback cycle. You re running thousands of tests per second, ideally, and an entire test run might take 5-10 seconds. That s the sweet spot, where you can code and test in a rapid cycle of ever-increasing awesomeness. Interestingly, when I ask the users of most infrastructure management systems about unit testing, they either get a blank look on their face, or, at best, point me in the direction of the likes of Test Kitchen, which is described quite clearly as an integration platform, not a unit testing platform. Puppet has rspec-puppet, which is a pretty solid unit testing framework for Puppet manifests although it isn t widely used. Others, though nobody seems to have any ideas. The blank look is near-universal. If infrastructure developers want to be taken seriously, we need to learn a little about what s involved in the development part of the title we ve bestowed upon ourselves. This means knowing what the various types of testing are, and having tools which enable and encourage that testing. It also means things like release management, documentation, modularity and reusability, and considering backwards compatibility. All of this needs to apply to everything that is within the remit of the infrastructure developer. You don t get to hand-wave away any of this just because it s just configuration! . This goes double when your just configuration! is a hundred lines of YAML interspersed with templating directives (SaltStack, I m looking at you).

12 August 2015

Daniel Pocock: Recording live events like a pro (part 2: video)

In the first blog in this series, we looked at how to capture audio effectively for a range of different events. While recording audio appears less complicated than video, it remains fundamental to a good recording. For some types of event, like a speech or a debate, you can have quite a bad video with poor lighting and other defects but people will still be able to watch it if the audio is good. Therefore, if you haven't already looked at the previous blog, please do so now. As mentioned in the earlier blog, many people now have high quality equipment for recording both audio and video and a wide range of opportunities to use it, whether it is a talk at a conference, a wedding or making a Kickstarter video. The right camera for video recording The fundamental piece of equipment is the camera itself. You may have a DSLR camera that can record video or maybe you have a proper handheld video camera. The leading DSLR cameras, combined with a good lens, make higher quality recordings than many handheld video cameras. Unfortunately, although you pay good money to buy a very well engineered DSLR that could easily record many hours of video, most DSLRs are crippled to record a maximum of 30 minutes in one recording. This issue and some workarounds are discussed later in this blog. If you don't have any camera at all you need to think carefully about which type to buy. If you are only going to use it once you may want to consider renting or borrowing or asking for other people attending the event to bring whatever cameras they have to help make multiple recordings (the crowdsourcing solution). If you are a very keen photographer then you will probably have a preference for a DSLR. Accessories Don't just look at the cost of your camera and conclude that is all the budget you need. For professional quality video recording, you will almost certainly need some accessories. You may find they are overpriced at the retail store where you bought your camera, but you still need some of them, so have a look online.
Recording a talk at a free software event with a Nikon D800 on a very basic tripod with Rode VideoMic Pro, headphones (white cable) and external power (black cable) If you want to capture audio with the camera and record it in the video file (discussed in more detail below), you will need to purchase a microphone that mounts on the camera. The built-in microphones on cameras are often quite bad, even on the most expensive cameras. If you are just using the built-in microphone for reference audio (to help with time synchronization when you combine different audio files with the video later) then the built-in microphone may be acceptable. Camera audio is discussed in more detail below. If your camera has a headphone socket, get some headphones for listening to the audio. Make sure you have multiple memory cards. Look carefully at the speed of the memory cards, slow ones are cheaper but they can't keep up with the speed of writing 1080p video. At a minimum, you should aim to buy memory cards that can handle one or two days worth of data for whatever it is you do. A tripod is essential for most types of video. If you use a particularly heavy camera or lens or if you are outdoors and it may be windy you will need a heavier tripod for stability. For video, it is very useful to have a tripod with a handle for panning left and right but if the camera will be stationary for the whole recording then the handle is not essential. Carrying at least one spare battery is another smart move. On one visit to the Inca Trail in Peru, we observed another member of our group hiking up and down the Andes with a brand new DSLR that they couldn't use because the battery was flat. For extended periods of recording, batteries will not be sufficient and you will need to purchase a mains power supply (PSU). These are available for most types of DSLR and video camera. The camera vendors typically design cameras with unusual power sockets so that you can only use a very specific and heavily overpriced PSU from the same company. Don't forget a surge protector too. There are various smartphone apps that allow you to remotely control the camera from the screen of your phone, such as the qDslrDashboard app. These often give a better preview than the screen built-in to the camera and may even allow you to use the touch screen to focus more quickly on a specific part of the picture. A regular USB cable is not suitable for this type of app, you need to buy a USB On-The-Go (OTG) cable.
Screenshot of qDslrDashboard app on a smartphone, controlling a DSLR camera If you plan to copy the video from the camera to a computer at the event, you will need to make sure you have a fast memory card reader. The memory card readers in some laptops are quite slow and others can be quite fast so you may not need to buy an extra card reader. Camera audio Most cameras, including DSLRs, have a built-in microphone and a socket for connecting an external microphone. The built-in microphones obtain very poor quality sound. For many events, it is much better to have independent microphones, such as a lapel microphone attached to a smartphone or wireless transmitter. Those solutions are described in part one of this blog series. Nonetheless, there are still some benefits of capturing audio in the camera. The biggest benefit is the time synchronization: if you have audio recordings in other devices, you will need to align them with the video using post-production software. If the camera recorded an audio stream too, even if the quality is not very good, you can visualize the waveform on screen and use it to align the other audio recordings much more easily and precisely. If the camera will be very close to the people speaking then it may be acceptable to use a microphone mounted on the camera. This will be convenient for post-production because the audio will be synchronized with the video. It may still not be as good as a lapel microphone though, but the quality of these camera-mounted microphones is still far higher than the built-in microphones. I've been trying the Rode VideoMic Pro, it is definitely better than recording with the built-in microphone on the camera and also better than the built-in microphone on my phone. One problem that most people encounter is the sound of the lens autofocus mechanism being detected by the microphone. This occurs with both the built-in microphone and any other microphone you mount on the camera. A microphone mounted on top of the camera doesn't detect this noise with the same intensity as the built-in microphone but it is still present in the recordings. If using a camera-mounted microphone to detect the audio from an audience, you may need to have an omnidirectional microphone. Many camera-mounted microphones are directional and will not detect very much sound from the sides or behind the camera. When using any type of external microphone with the camera, it is recommend to disable automatic gain control (AGC) in the camera settings and then manually adjust the microphone sensitivity/volume level. Use headphones A final word on audio - most good cameras have an audio output socket. Connect headphones and wear them, to make sure you are always capturing audio. Otherwise, if the microphone's battery goes flat or if a wireless microphone goes out of range you may not notice. Choosing a lens The more light you get, the better. Bigger and more expensive lenses allow more light into the camera. Many of the normal lenses sold with a DSLR camera are acceptable but if it is a special occasion you may want to rent a more expensive lens for the day. If you already have a lens, it is a very good idea to test it in conditions similar to those you expect for the event you want to record. Recording duration limits Most DSLR cameras with video capability impose a 30 minute maximum recording duration This is basically the result of a friendly arrangement between movie studios and politicians to charge an extra tax on video recording technology and potentially make the movie studio bosses richer, supposedly justified by the fact that a tiny but exaggerated number of people use their cameras to record movies at the cinema. As a consequence, most DSLR manufacturers limit the duration of video recording so their product won't be subject to the tax, ensuring the retail price is lower and more attractive. On top of this, many DSLR cameras also have a 4GB file size limit if they use the FAT filesystem. Recording 1080p video at a high frame rate may hit the file size limit in 10 minutes, well before you encounter the 30 minute maximum recording duration. To deal with the file size issue, you can record at 720p instead of 1080p and use the frame rate 24fps. For events longer than 30 minutes or where you really want 1080p or a higher frame rate, there are some other options you can consider:
  • Buy or rent a proper video camera instead of using a DSLR camera
  • Using multiple cameras that can stop and be restarted at different times.
  • Manually stopping and restarting the camera if there are breaks in the event where it is safe to do so.
  • Use an app to control the camera and program it to immediately restart the recording each time it stops
  • Extract the raw output from the camera's HDMI socket and record into some other device or computer. There are several purpose-built devices
    that can be used this way with an embedded SSD for storage.
  • There are also some people distributing unofficial/alternative firmware images that remove the artificial 30 minute recording limit.
Camera settings There are many online tutorials and demonstration videos on YouTube that will help you optimize the camera settings for video. You may have already made recordings using the automatic mode. Adjusting some or all of the settings manually may help you create a more optimal recording. You will need to spend some time familiarizing yourself with the settings first. The first thing to check is white balance. This tells the camera the type of lighting in the location. If you set this incorrectly then the colours will be distorted. Many cameras have the ability to automatically set the white balance. For video, you may be able to change one or more settings that control the recording quality. These settings control the file compression ratio and image size. Typical image size settings are 720p and 1080p. Compression ratio may be controlled by a high/medium/low quality setting. Choosing the highest quality and biggest picture requires more space on the memory card and also means you reach the 4GB file size limit more quickly. A higher quality setting also implies a faster memory card is required, because the rate of megabytes per second written to the memory card is higher. Next you need to think about the frame rate. Events that involve fast moving subjects, such as sports, typically benefit from a higher frame rate. For other events it is quite acceptable to use 24 frames per second (fps). Higher frame rates also imply bigger file size and a requirement for a faster memory card. Once you have decided on the frame rate, the next thing to do is set the shutter speed. Use a shutter speed that is double the frame rate. For example, if using 24fps or 25fps, use a 1/50 shutter speed. The final two settings you need to adjust are the ISO and aperture. Set these based on the lighting conditions and extent to which the subjects are moving. For example, if the setting is dark or if you are trying to record fast moving subjects like athletes, vehicles or animals, use an ISO value of 800 or higher. Once you have chosen ISO, adjust the aperture to ensure the picture is sufficiently illuminated. Aperture also has a significant impact on the depth of field. Operating the camera: zoom and focus Many people use zoom lenses. It is not always necessary to change the zoom while recording a video, you can use software to zoom in and out on individual parts of the picture when editing it in post-production. If you do change the zoom while recording, it may be more difficult to maintain focus. Almost all lenses support manual focus (turning the focus ring by hand) and many support automatic focus. When shooting photographs with a DSLR, the mirror is down and the camera can use dedicated sensors for focus and light sensing. When shooting video, the mirror is up and the camera can not use the same focus sensors that are used in photography. Video recording uses a digital focussing algorithm based on contrast in the picture. If you take a lot of photos you are probably quite accustomed to the fast and precise autofocus for photography and you will notice that keeping a video in focus is more challenging. As mentioned already, one of the first things you can do to keep focus simple is to avoid zooming while recording. Record in a higher resolution than you require and then zoom with software later. Some people record using 4k resolution even when they only want to produce a 720p video, as they can digitally zoom in to different parts of the 4k recording without losing detail. If the subject is very stationary (people sitting at a desk for an interview is a typical example), you may be able to set the camera to manual focus and not change it at all while recording. If you choose to enable autofocus while recording, any built-in camera microphone or microphone mounted on the camera is likely to detect sounds from the motorized focus system. Ultimately, the autofocus mechanism is not accurate for all subjects and you may be unable to stop them moving around so you will need to change the focus manually while recording. It requires some practice to be able to do this quickly without overshooting the right focus. To make life more tricky, Nikon and Canon focus rings rotate in the opposite direction, so if you are proficient using one brand you may feel awkward if you ever have to use the other. A good way to practice this skill is to practice while in the car or on the train, pointing at different subjects outside the window and trying to stay in focus as you move from one subject to the next. Make a trial run Many events, from weddings right up to the Olympic Games opening ceremony, have a trial run the day before. One reason for that is to test the locations and settings of all the recording and broadcasting equipment. If a trial run isn't possible for your event, you may find some similar event to practice recording and test your equipment. For example, if you are planning to record a wedding, you could try and record a Sunday mass in the same church. Backup and duplicate the recordings before leaving the event If you only have one copy of the recordings and the equipment is stolen or damaged you may be very disappointed. Before your event, make a plan to duplicate the raw audio and video recordings so that several people can take copies away with them. Decide in advance who will be responsible for this, ensure there will be several portable hard disks and estimate how much time it will take to prepare the copies and factor this into the schedule. Conclusion All the products described can be easily purchased from online retailers. You may not need every accessory that is mentioned as it depends on the type of event you record. The total cost of buying or renting the necessary accessories may be as much as the cost of the camera itself so if you are new to this you may need to think carefully about making a budget with a spreadsheet to do it correctly. Becoming familiar with the camera controls and practicing the techniques for manual focus and zoom can take weeks or months. If you enjoy photography this can be time well spent but if you don't enjoy it then you may not want to commit the time necessary to make good quality video. Don't rule out options like renting equipment instead of buying it or crowdsourcing, asking several participants or friends to help make recordings with their own equipment. For many events, audio is far more indispensable than video and as emphasizing at the beginning of this article, it is recommended that you should be one hundred percent confident in your strategy for recording audio before you start planning to record video.

11 August 2015

Matthias Klumpp: AppStream/DEP-11 for everyone! (beta!)

One of the things we discussed at this year s Akademy conference is making AppStream work on Kubuntu.appstream-logo On Debian-based systems, we use a YAML-based implementation of AppStream, called DEP-11 . DEP-11 exists for historical reasons (the DEP-11 YAML format was a superset of AppStream once) and because YAML, unlike XML, is one accepted file format by the Debian FTPMasters team, who we want to have on board when adding support for AppStream. So I ve spent the last few days on setting up the DEP-11 generator for Kubuntu, as well as improving it greatly to produce more meaningful error messages and to generate better output. It became a bit slower in the process, but the greatly improved diagnostics data is worth it. For example, maintainers of Debian packages will now get a more verbose explanation of issues found with metadata in their packages, making them easier to fix for people who didn t get in contact with AppStream yet. At time, we generate AppStream metadata for Tanglu, Kubuntu and Debian, but so far only Tanglu makes real use of it by shipping it in a .deb package. Shipping the data as package is only a workaround though, for a proper implementation, the data will be downloaded by Apt. To achieve that, the data needs to reach the archive first, which is something that I can hopefully discuss and implement with the FTPMasters team of Debian at this year s Debconf. When this is done, the new metadata will automatically become available in tools like GNOME-Software or Muon Discover. How can I see if there are issues with my package? The dep11-generator tool will return HTML pages to show both the extracted metadata, as well as issues with it. You can find the information for the respective distribution here: Each issue tag will contain a detailed explanation of what went wrong. Errors generally lead to ignoring the metadata, so it will not be processed. Warnings usually concern things which might reduce the amount of metadata or make it less useful, while Info-type hints contain information on how to improve the metadata or make it more useful. Can I use the data already? Yes, you can. You just need to place the compressed YAML files in /var/cache/app-info/yaml and the icons in /var/cache/app-info/icons/<suite>-<component>/<size>, for example: /var/cache/app-info/icons/jessie-amd64/64 64 I think I found a bug in the generator In that case, please report the issue against the appstream-dep11 package at Debian, or file an issue at Github.. The only reason why I announce this feature now is to find remaining generator bugs, before officially announcing the feature on debian-devel-announce. When will this be officially announced? I want to give this feature a little bit more testing, and ideally have the integration into the archive ready, so people can see how the metadata looks like when rendered in GNOME-Software/Discover. I also want to write a bit more documentation to help Debian developers and upstreams to improve their metadata. Ideally, I also want to incorporate some feedback at Debconf when announcing full AppStream support in Debian. So take all the stuff you ve read above as a little sneak peek  Debconf15goingI will also give a talk at Debconf, titled AppStream, Limba, XdgApp Where we are going. The aim of this talk is to give an insight i tnto the new developments happening in the software distribution area, and what the goal of these different projects is (the talk should give an overview of what s in the oven and how it will impact Debian). So if you are interested, please drop by :-) Maybe setting up a BOF would also be a good idea.

10 August 2015

Matthias Klumpp: Akademy 2015

I am very late with this, but I still wanted to write a few words about Akademy 2015. First of all: It was an awesome conference! Meeting all the great people involved with KDE and seeing who I am working with (as in: face to face, not via email) was awesome. We had some very interesting discussions on a wide variety of topics, and also enjoyed quite some beer together. Particularly important to me was of course talking to Daniel Vr til who is working on XdgApp, and Aleix Pol of Muon Discover fame.
Also meeting with the other Kubuntu members was awesome I haven t seen some of them for about 3 years, and also met many cool people for the first time. My talk on AppStream/Limba went well, except that I got slightly confused by the timer showing that I had only 2 minutes left, after I had just completed the first half of my talk. It turned out that the timer was wrong  Another really nice aspect was to be able to get an insight into areas where I am usually not involved with, like visual design. It was really interesting to learn about the great work others are doing and to talk to people about their work and I also managed to scratch an itch in the Systemsettings application, where three categories had shown the same icon. Now Systemsettings looks like it is supposed to be, finally :-) The only thing I could maybe complain about was the weather, which was more Scotland/Wales like than Spanish but that didn t stop us at all, not even at the social event outside. So I actually don t complain  We also managed to discuss some new technical stuff, like AppStream for Kubuntu, and plenty of other things that I ll write about in a separate blog post. Generally, I got so many new impressions from this year s Akademy, that I could write a 10-pages long blogpost about it while still having to leave out things.

Kudos to the organizers of this Akademy, you did a fantastic job! I also want to thank the Ubuntu community for funding my flight, and the Kubuntu community for pushing me a little to attend :-). KDE is a great community with people driving the development of Free Software forward. The diversity of people, projects and ideas in KDE is a pleasure, and I am very happy to be part of this community. Akademy2015

Lunar: Reproducible builds: week 15 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Guillem Jover uploaded dpkg/1.18.2 which makes dependency comparisons deep by comparing not only the first dependency alternative, to get them sorted in a reproducible way. Original patch by Chris Lamb. Dhole updated the patch adding support for SOURCE_DATE_EPOCH in gettext. A modified package is in the experimental reproducible repository. Valentin Lorentz submitted a patch adding support for SOURCE_DATE_EPOCH to ocamldoc. Valentin Lorentz also opened a bug about the inability to set an arbitrary RNG seed for ocamlopt which would be a way to fix an issue affecting many OCaml packages. Dhole submitted a patch adding support for SOURCE_DATE_EPOCH in qhelpgenerator. A modified package has been sent to the experimental repository as well. Several packages have been updated for the experimental toolchain: doxygen (akira), and dpkg (h01ger). Also, h01ger has built and uploaded all experimental packages having arch:any packages for armhf: dpkg, gettext, doxygen, fontforge, libxslt and texlive-bin. We are now providing our toolchain for armhf and amd64. Packages fixed As you might have noticed, Debian sid is currently largely uninstallable, due to the GCC 5 transition, which also can be see in our reproducibility test setup. Please help! The following packages became reproducible due to changes in their build dependencies: glosstex, indent, ktikz, liblouis, libmicrohttpd, linkchecker, multiboot, qterm, rrep, trueprint, twittering-mode. The following packages became reproducible after getting fixed: Patches submitted which have not made their way to the archive yet: Lunar reported an issue on an unstable ABI from a generated header in icedove reminding of an issue affecting libical-dev. The bug has since been fixed by Carsten Schoenert. akira identified an unreferenced embeded code copy (causing unreproducibility!) in gperf. reproducible.debian.net The scheduler has temporarily been changed to not schedule any already tested packages for sid and experimental, due to the the GCC 5 transitions, which are well visible in our graphs now. On the plus side this has caused our stretch testing to catch up (and improve stats). (h01ger) depwait packages (packages where the Build-Depends cannot be satisfied) are now listed in the last 24h and last 48h pages (Mattia Rizzolo) Two new amd64 build nodes (with 8 cores and 32 GB RAM each) have been added, kindly sponsored by Profitbricks. (h01ger) The 4 armhf (setup last week by Vagrant Cascadian) and 2 amd64 build nodes have been made available to Jenkins. Remote job scheduling has been implemented and 35 new jobs have been added for pbuilder and schroot creation and maintenance of the nodes. (h01ger) The manual scheduler gained a flag (-a/--architecture) to select which arch to schedule in. (Mattia Rizzolo) armhf will only be testing stretch for now, due to limited hardware ressources. (h01ger) The page listing maintainers of unreproducible packages gained internal anchors. As an example, one can now link to unreproducible orphaned packages. (Mattia Rizzolo) Packages with a bug tagged pending are marked using a new symbol: a brown P (Mattia Rizzolo) diffoscope development debbindiff is now called diffoscope! It also has a website at diffoscope.org. The name was changed to better reflect that it became a general purpose tool, capable of comparing many different archive formats, or directories. Version 29 is the renaming release. Amongst a couple of other cosmetic changes a favicon showing the new logo has been added to the generated HTML reports. Version 30 replaces the file matching algorithm for files listed in .changes to a smarter one that removes only the version number. It also fixes a bug where squashfs directories were being extracted even if their content was being compared at a later stage. It also fixes an issue with the test suite that was detected by debci. Documentation update More rationale have been added for supporting SOURCE_DATE_EPOCH The unfinished Reproducible Builds HOWTO is now visible on the web, feedback and patches most welcome. Package reviews 261 obsolete reviews have been removed, 73 added and 145 updated this week.

Mirco Bauer: Smuxi 1.0 "Finally" Release

And here we go again! We're proud to announce the new version of Smuxi, release 1.0 "Finally". During the development, 20 bug reports and 10 feature requests in 285 commits were worked on.

Finally 1.0 Smuxi is celebrating its 10th anniversary! 10 years ago, Mirco Bauer made the first commit to the Smuxi source code repository and is still very committed to it. He started the Gnosmirc project in 2005 when the only way a 24/7 "always-on" experience with IRC meant you had to use a console based IRC client like bitchx, irssi or epic combined with screen and SSH. This looks very practical at first and is a powerful Unix-ish way of accomplishing that job, but it has the big downside that it doesn't integrate with a desktop environment like GNOME. A bit later the Gnosmirc project was renamed to Smuxi when the new code architecture allowed other frontend implementations besides the GNOME one. The ncurses/STFL based text frontend was later implemented and is considered stable and useful enough for day to day use, but still has some rough edges. WinForms and WPF frontends also exist but need more work to reach a usable state. At this point Smuxi 1.0 contains all features that we could have imagined and even goes beyond with very advanced features like message patterns or language agnostic scripting.

Changes since Smuxi 0.11

Message Persistence One of the biggest drawbacks of the IRC protocol ever was that messages can't be retrieved from the IRC server because the IRC server is simply relaying messages to the connected clients. So, if an IRC client is freshly started and connects it starts to receive new messages, but all message you had received before are no longer available. This always made IRC in a way "volatile" unlike other communication systems like email where messages are relayed and stored on the client side. One common approach for IRC clients is to store log files in a text file. This is a simple feature and gives the user the possibility to read older conversations. Smuxi also supports text file logging like other IRC clients but it has a big user experience drawback as you need to open the file from the disk outside of the IRC client. In Smuxi 1.0 messages sent and received are now stored on the disk in a way they can automatically be retrieved/loaded when you restart Smuxi. It is like you have never closed Smuxi! This feature was already available in Smuxi for some time as a technical preview and it used the Db4o object database, but we were never happy about the performance neither with the stability so it always stayed an optional feature you need to enable. This year we tried a new message buffer backend using the famous SQLite database and it works much faster and stable as a rock. So finally we can enable this feature by default because it just works and enhanced your experience. We hope you enjoy it. Documentation of how you can change Smuxi message buffer backend and behavior can be found here. For instructions how to convert your existing db4o history to SQLite can be found in the "smuxi-message-buffer tool" section.

User Interface Enhancements
  • Synced message markers: the position of of the seen/unseen messages marker is pushed to the smuxi-server and remembered when the frontend reconnects. (Sebastian Poeplau)
  • Persistent message markers: the message marker position is also remembered across Smuxi(-server) restarts.
  • Message Counter: in addition to the highlight counter next to a chat new/unseen messages are also counted. This makes it easy to identify chats with much traffic.
  • Single application instance support. If you start Smuxi again from the menu it will bring the existing instance into the foreground. This makes the Ubuntu Messaging Menu much nicer.
  • The command/message entry is alignment with the messages. (Lex Berezhny)

Text Frontend Enhancements
  • The console background color can now be configured using: /config set STFL/Interface/TerminalBackgroundColor = #000000 (Ond ej Ho ek)
  • The text color contrast if nicks with the background is now ensured (Ond ej Ho ek) #1033
  • Messages containing images will not be skipped but their alternative text is shown instead (Ond ej Ho ek) #1035

New smuxi-message-buffer tool This is a new commandline tool that allows you to convert and export the message history of Smuxi message buffer files. This can be used to convert your existing Db4o history to SQLite like this for example:
for DB_DB4O in $HOME/.local/share/smuxi/buffers/*/*/*/*.db4o; do
    DB_SQLITE=$ DB_DB4O/.db4o/.sqlite3 
    smuxi-message-buffer convert $DB_DB4O $DB_SQLITE
done
Smuxi shouldn't be running when using this tool.

Scripting Enhancements

New Hook Points Smuxi 1.0 supports with the following new hook points:
  • engine/protocol-manager/on-presence-status-changed/ This hook point is raised when the presence status of a protocol manager changes. This happens for example when an IRC connection toggles the away state.
  • engine/session/on-event-message/ This hook point raises event messages that usually begin with "-!-". This can be useful to track state changes that are shown as a message without having a dedicated hook point for it.
  • engine/session/command-$cmd/ This hook point is raised on the engine side for commands, e.g. /some_command that isn't handled by the frontend or engine built-in commands. This is useful for commands that should be available for all frontends and isn't specific to the frontend environment.

New Plugins The following new plugins are supported by Smuxi 1.0:
  • topic-diff: Shows the word differences of the topic after topic changes. (meebey)
  • away-nick: Automatically appends and removes $AWAY_SUFFIX to/from the nick name when you go away using the /away command or by disconnecting all frontends from the smuxi-server. (meebey)
  • system-info: Shows system info. Includes system kernel version, distro name, and CPU vendor information. (AK0)
  • now-playing: This plugin is not new but was rewritten in Python to get rid of the spaghetti code monster which was written in Bash. (jamesaxl)

IRC Enhancements
  • NICKSERV support Notices from Nick/ChanServ are no longer shown on all channels as they like to send greeting messages and other spam which is annoying to see on all channels. #868
  • Automatic rejoin of channels protected with a key works as expected again
  • Connecting to irc.gitter.im is now supported. Gitter's IRCd implementation has a bug in the IRC protocol which is now tolerated.

Twitter Enhancements
  • The /search command shows tweets as live stream
  • Added /delete, /favorite and /unfavorite commands

Behind the Scenes
  • Re-licensed smuxi-common from GPLv2 to MIT/X11

Contributors Contributors to this release are the following people:
  • Mirco Bauer (199 commits)
  • Carlos Mart n Nieto (15 commits)
  • Andr s G. Aragoneses (14 commits)
  • Piotr Dr g (12 commits)
  • Ond ej Ho ek (11 commits)
  • Oliver Schneider (5 commits)
  • Calvin B (4 commits)
  • Victor Seva (3 commits)
  • Will Johansson (2 commits)
  • Sebastian Poeplau (2 commits)
  • Julian Taylor (2 commits)
  • James Axl (2 commits)
  • Daniel Mustieles (2 commits)
  • Christopher James Halse Rogers (2 commits)
  • . Uzun (1 commit)
  • Lex Berezhny (1 commit)
  • Kalle Kaitala (1 commit)
  • Jordi Mas (1 commit)
  • Joe Hansen (1 commit)
  • Jimmie Elvenmark (1 commit)
  • Dimitris Spingos (1 commit)
  • Dean Lee (1 commit)
  • Cl ment Bourgeois (1 commit)
  • Carlos Hernandez (1 commit)
Thank you very much for your contributions to Smuxi! Want this? Go here and grab it right now!

Posted Sun Aug 9 17:48:18 2015

8 August 2015

Matthew Garrett: Difficult social problems are still difficult problems

After less than a week of complaints, the TODO group have decided to pause development of their code of conduct. This seems to have been triggered by the public response to the changes I talked about here, which TODO appear to have been completely unprepared for.

While disappointing in a bunch of ways, this is probably the correct decision. TODO stumbled into this space with a poor understanding of the problems that they were trying to solve. Nikki Murray pointed out that the initial draft lacked several of the key components that help ensure less privileged groups can feel that their concerns are taken seriously. This was mostly rectified last week, but nobody involved appeared to be willing to stand behind those changes in a convincing way. This wasn't helped by almost all of this appearing to land on Github's plate, with the rest of the TODO group largely missing in action[1]. Where were Google in this? Yahoo? Facebook? Left facing an angry mob with nobody willing to make explicit statements of support, it's unsurprising that Github would try to back away from the situation.

But that doesn't remove their blame for being in the situation in the first place. The statement claims
We are consulting with stakeholders, community leaders, and legal professionals, which is great. It's also far too late. If an industry body wrote a new kernel from scratch and deployed it without any external review, then discovered that it didn't work and only then consulted any of the existing experts in the field, we'd never take them seriously again. But when an industry body turns up with a new social policy, fucks up spectacularly and then goes back to consult experts, it's expected that we give them a pass.

Why? Because we don't perceive social problems as difficult problems, and we assume that anybody can solve them by simply sitting down and talking for a few hours. When we find out that we've screwed up we throw our hands in the air and admit that this is all more difficult than we imagined, and we give up. We ignore the lessons that people have learned in the past. We ignore the existing work that's been done in the field. We ignore the people who work full time on helping solve these problems.

We wouldn't let an industry body with no experience of engineering build a bridge. We need to accept that social problems are outside our realm of expertise and defer to the people who are experts.

[1] The repository history shows the majority of substantive changes were from Github, with the initial work appearing to be mostly from Twitter.

comment count unavailable comments

7 August 2015

Norbert Preining: Japanese-English dictionary for Kobo

Since ever the Kobo firmwares also allowed downloading of a bunch of dictionaries, most of which I don t need. As I am fluent in most languages I read and write, the only real dictionary I would like to see is a Japanese-English (I don t dare asking for a Japanese-German). Unfortunately, Kobo never shipped one. OTOH, starting with firmware 3.16.10 they ships two different English-Japanese dictionaries, one excellent Japanese-Japanese dictionary, but not one Japanese-English. So I took the liberty to write a script that allows everyone to enrich the shipped Japanese-Japanese dictionary with English definitions. Kobo Logo Till now I have used Tshering s excellent Japanese dictionary which was enriched with English definitions. It was based on the older Japanese-Japanese dictionary shipped with firmwares before 3.16.10. I didn t want to loose the more complete dictionary from the new firmware, so here is a script that updates the dictionary for you Prerequisites The script is neither fool-proof nor does everything by itself. Furthermore, it requires a set of programs. In details: Supported dictionaries At the moment the program can use the following two dictionaries as sources: Edict2 and Japanese3. Edict2 The Edict dictionary is a free dictionary which is the base of most other dictionaries. Created by the JMdict/EDICT Project it provides a very complete Japanese-English dictionary. To use the dict with the current program, one need to download the edict2.gz file and unpack it with gunzip edict2.gz. If you put it into the same directory as the Perl script, nothing else needs to be done, otherwise one can use the command line option --edict PATH-TO-EDICT to specify the location of the edict2 file. Japanese3 The Japanese3 is a dictionary application for iOS which provides another very complete Japanese-English dictionary. My feeling is that it is 90% and more based on Edict, so adding it will not buy you much, but still a bit (see below for how much!). If you have purchased this dictionary/application, and you manage to get access to your iOS device (via jailbreaking or some other tools), then you need to get the file Japanese3.db from the application folder, and then generate a file via sqlite3 as follows:
$ sqlite3 Japanese3.db
.output japanese3-data
select Entry, Furigana, Summary from entries;
If you save the generated file japanese3-data into the same directory as the Perl script, nothing else needs to be done, otherwise one can use the command line option --japanese3 PATH-TO-J3-DATA to specify the location of the Japanese3 data file. Command line options The program supports the following command line options: Note that in case you pass in the option --unpacked, the files should be properly named (encodings are a problem!). Furthermore, note that the unpacked directory contains lots of .gif and .html files that are actually gzip-compressed files, but lacking the .gz extension. If you want to unpack the dictionary, by advised that the file names in the Zip directory are already encoded in UTF-8, but normal programs (unzip, 7z) assume that they are encoded in some other encoding. Thus, if your are using UTF-8 locale, it is necessary to set LC_CTYPE to C to make sure that the encoding is used, as in
LC_CTYPE=C 7z x ...
Typical run In the following example we use both dict2 and Japanese3 dictionaries, and prefer Edict2:
$ perl enhance-dictionary.pl --dicts edict2 --dicts japanese3
Using the following dictionaries as source for translations: edict2 japanese3
loading edict2 ... done
loading Japanese3 data ... done
unpacking original dictionary ... done
loading dict files ... done
searching for words and updating ... done
total words 805521, matches: 130563 (edict: 123388, japanese3: 7175)
creating output html ... done
creating update dictionary in dicthtml-jaxxdjs-201508072201.zip ... done
$
Installing the dictionary After having created the enhanced dictionary, one can install the generated file as KOBO/.kobo/dicts/dicthtml-jaxxdjs.zip (where KOBO is the mount point of the eReader). The dictionary should be picked up automatically. And the next lookup should give you something like the following:
kobo-glohd-enhanced-ja-dict There is only one caveat: Syncing with Kobo will re-download the original dictionary and overwrite the enhanced one. There is at least one solution here, which I am employing, see this post on MobileRead. Download and development place I am using the github repository kobo-ja-dict-enhance to develop the program. Please report bugs, and feature requests there. The file for the last released version can be downloaded from here: enhance-dictionary.pl Future plans I am planning to get rid of the 7z dependency and use Archive::Zip for all the unpacking and packing. This would also allow to do everything in memory and thus make it faster. There is a branch on github with ongoing work on this matter. I also think about including translations for Hiragana words by providing the translations of all possible Kanji readings. Conclusions Even if Kobo does not provide us with a decent Japanese-English dictionary, adding at least a huge amount of translations to the current dictionary is now easily possible. For serious students of Japanese who are reading or starting to read Japanese eBooks, this will be of great help. Enjoy, and don t forget to give feedback, suggestions, and improvements either here or better via the Github page.

Guido G nther: Debian work in July 2015

July was the third month I contributed to Debian LTS under the Freexian umbrella. In total I spent eight hours working on: Besides that I did CVE triaging of 11 CVEs to check if and how they affect oldoldstable security as part of my LTS front desk work.

5 August 2015

Antonio Terceiro: Elixir in Debian, MiniDebconf at FISL, and Debian CI updates

In June I started keeping track of my Debian activities, and this is my July update. Elixir in Debian Elixir is a functional language built on top of the Erlang virtual machine. If features imutable data structures, interesting concurrency primitives, and everything else that Erlang does, but with a syntax inspired by Ruby what makes it much more aproachable in my opinion. Those interested in Elixir for Debian are encouraged to hang around in #debian-elixir on the OFTC IRC servers. There are still a lot of things to figure out, for example how packaging Elixir libraries and applications is going to work. MiniDebconf at FISL, and beyond I helped organize a MiniDebconf at this year s FISL, in Porto Alegre on the 10th of July. The whole program was targetted at getting more people to participate in Debian, so there were talks about translation, packaging, and a few other more specific topics. I myself gave two talks: one about Debian basics, What is Debian, and how it works , and second one on packaging the free software web , which I will also give at Debconf15 later this month. The recordings are available (all talks in Portuguese) at the Debian video archive thanks to Holger Levsen. We are also organizing a new MiniDebconf in October as part of the Latinoware schedule. Ruby We are in the middle of a transition to switch to Ruby 2.2 as default in Debian unstable, and we are almost there. The Ruby transition is now on hold while GCC 5 one is going on, but will be picked up as soon as were are done with GCC 5. ruby-defaults has been uploaded to experimental for those that want to try having Ruby 2.2 as default before that change hits unstable. I myself have been using Ruby 2.2 as default for several weeks without any problem so far, including using vagrant on a daily basis and doing all my development on sid with it. I started taking notes about Ruby interpreter transitions work to make sure that knowledge is registered. I have uploaded minor security updates of both ruby2.1 and ruby2.2 to unstable. They both reached testing earlier today. I have also fixed another bug in redmine, which I hope to get into stable as well as soon as possible. gem2deb has seen several improvements through versions 0.19, 0.20, 0.20.1 and 0.20.2. I have updated a few packages: Two NEW packages, ruby-rack-contrib and ruby-grape-logging ,were ACCEPTED into the Debian archive. Kudos to the ftp-master team who are doing an awesome job reviewing new packages really fast. Debian Continuous Integration This month I have made good progress with the changes needed to make debci work as a distributed system with one master/scheduler node and as many worker nodes (running tests) as possible. While doing my tests, I have submitted a patch to lxc and updated autodep8 in unstable. At some point I plan to upload both autodep8 and autopkgtest to jessie-backports. Sponsoring I have sponsored a few packages:

4 August 2015

Julien Danjou: Ceilometer, Gnocchi & Aodh: Liberty progress

It's been a while since I talked about Ceilometer and its companions, so I thought I'd go ahead and write a bit about what's going on this side of OpenStack. I'm not going to cover new features and fancy stuff today, but rather a shallow overview of the new project processes we initiated. Ceilometer growing Ceilometer has grown a lot since that time when we started it 3 years ago. It has evolved from a system designed to fetch and store measurements, to a more complex system, with agents, alarms, events, databases, APIs, etc. All those features were needed and asked for by users and operators, but let's be honest, some of them should never have ended up in the Ceilometer code repository, especially not all at the same time. The reality is we picked a pragmatic approach due to the rigidity of the OpenStack Technical Committee in regards to new projects to become OpenStack integrated and, therefore, blessed projects. Ceilometer was actually the first project to be incubated and then integrated. We had to go through the very first issues of that process. Fortunately, now that time has passed, and all those constraints have been relaxed. To me, the OpenStack Foundation is turning into something that looks like the Apache Foundation, and there's, therefore, no need to tie technical solutions to political issues. Indeed, the Big Tent now allows much more flexibility to all of that. Back a year ago, we were afraid to bring Gnocchi into Ceilometer. Was the Technical Committee going to review the project? Was the project going to be in the scope of Ceilometer for the Technical Committee? Now we don't have to ask ourselves those questions, now that we have that freedom, it empowers us to actually do what we think is good in term of technical design without worrying too much about political issues.
Ceilometer development activity
Acknowledging Gnocchi The first step in this new process was to continue working on Gnocchi (a timeserie database and resource indexer designed to overcome historical Ceilometer storage issue) and to decide that it was not the right call to merge it into Ceilometer as some REST API v3, but that it was better to keep it standalone. We managed to get traction to Gnocchi, getting a few contributors and users. We're even seeing talks proposed to the next Tokyo Summit where people leverage Gnocchi, such as "Service of predictive analytics on cost and performance in OpenStack", "Suveil" and "Cutting Edge NFV On OpenStack: Healing and Scaling Distributed Applications". We are also doing some progress on pushing Gnocchi outside of the OpenStack community, as it can be a self-sufficient timeserie and resource database that can be used without any OpenStack interaction. Branching Aodh Rather than continuing to grow Ceilometer, during the last summit we all decided that it was time to reorganize and split Ceilometer into the different components it is made of, leveraging a more service-oriented architecture. The alarm subsystem of Ceilometer being mostly untied to the rest of Ceilometer, we decided it was the first and perfect candidate to do that. I personally engaged into doing the work and created a new repository with only the alarm code from Ceilometer, named Aodh.
Aodh is an Irish word meaning fire. A word picked so it also had some relation to Heat, and because we have some Irish influence around the project .
This made sense for a lot of reason. First because Aodh can now work completely standalone, using either Ceilometer or Gnocchi as a backend or any new plugin you'd write. I love the idea that OpenStack projects can work standalone like Swift does for example without implying any other OpenStack component. I think it's a proof of good design. Secondly, because it allows us to resonate on a smaller chunk of software a reason really under-estimated today in OpenStack. I believe that the size of your software should match a certain ratio to the size of your team. Aodh is, therefore, a new project under the OpenStack Telemetry program (or what remains of OpenStack programs now), alongside Ceilometer and Gnocchi, forked from the original Ceilometer alarm feature. We'll deprecate the latter with the Liberty release, and we'll remove it in the Mitaka release. Lessons learned Actually, moving that code out of Ceilometer (in the case of Aodh), or not merging it in (in the case of Gnocchi) had a few side effects that I admit I think we probably under-estimated back then. Indeed, the code size of Gnocchi or Aodh ended up being much smaller than the entire Ceilometer project Gnocchi is 7 smaller and Aodh 5x smaller than Ceilometer and therefore much more easy to manipulate and to hack on. That allowed us to merge dozens of patches in a few weeks, cleaning-up and enhancing a lot of small things in the code. Those tasks are very much harder in Ceilometer, due to the bigger size of the code base and the small size of our team. By having our small team working on smaller chunks of changes even when it meant actually doing more reviews greatly improved our general velocity and the number of bugs fixed and features implemented. On the more sociological side, I think it gave the team the sensation of finally owning the project. Ceilometer was huge, and it was impossible for people to know every side of it. Now, it's getting possible for people inside a team to cover a much larger portion of those smaller project, which gives them a greater sense of ownership and caring. Which ends up being good for the project quality overall. That also means that we technically decided to have different core teams by project (Ceilometer, Gnocchi, and Aodh) as they all serve different purposes and can all be used standalone or with each others. Meaning we could have contributors completely ignoring other projects. All of that reminds me some discussion I heard about projects such as Glance, trying to fit new features in - some that are really orthogonal to the original purpose. It's now clear to me that having different small components interacting together that can be completely owned and taken care of by a (small) team of contributors is the way to go. People that can therefore trust each others and easily bring new people in, makes a project really incredibly more powerful. Having a project covering a too wide set of features make things more difficult if you don't have enough manpower. This is clearly an issue that big projects inside OpenStack are facing now, such as Neutron or Nova.

3 August 2015

Sven Hoexter: Failing with F5: CMP - Clustered Multiprocessing

The last Whiteboard Wednesday tackled a few of the gotchas of the F5 CMP implementation. I take that as an opportunity to finish a blog post I started to write six month ago on this topic. I've run into two scenarios where CMP got in my way: All of this happened on TMOS 11.4.1+HF run on BigIP 2K systems. Since F5 is sometimes moving very quickly you might not experience those issues on other hardware or with a different software release. applying low and strict connection limits Apparently connection limits are counted on a per tmm level and are not shared between the tmm processes running on different CPUs. So if you've two CPU cores with HT you'll see four tmm processes that will all enforce the connection limit. If you have a max connection limit set to two you might still see a maximum of eight connections. That is a very bad thing if you abuse the F5 to limit your outbound connections because your counter part is not able to enforce connection limits, but forces you via a contract to not open more then two connections. Disable CMP and it will all end on the same tmm process and suddenly your connection limits work as expected. If your connection limit is big enough you can cheat, say it's twelve connections, you can configure a limit of three. With four processes that makes a max of twelve connections as required. Though due to the hashing and distribution among the tmm processes it could happen that you reject connections on one process while others are idle. Another option would be to enter the dirty land of iRules and tables to maintain your own connection count table. There are examples given in the F5 devcentral community, but the best resolution you can get, without starting to scan the whole table on every request, is X requests per minute. active health checks Active monitoring checks were added to detect the operational state of our application servers. Now imagine you have some realtime processing with over a hundred requests per minute. So in case you've issues with one instance you'd like to remove it from the pool rather sooner then later, which requires frequent checking. So here is a simple active http check:
defaults-from http-appmon
destination *:*
interval 9
recv "^HTTP/1\\.1 200 OK"
send "GET /mon/state HTTP/1.1\\r\\nHost: MON\\r\\nConnection: Close\\r\\n\\r\\n"
time-until-up 0
timeout 2
up-interval 1
(Configuration options are described here) With an up-interval of 1 second and a timeout of 2 seconds someone naive like me expected that a fatal issue like a crashed application would be detected within three seconds and no more requests should hit the application after second 3. That turned out to be slightly longer, at least 4s for this case, often longer. Also for a regular maintenance we had to instruct our scripts to flip the application monitor to unavailable and wait for interval X tmm process count + timeout + 1s + max allowed request duration. That is not an instant off. If you use tcpdump (on the F5 where it's patched to show you the tmm that received the request) to look at the health check traffic you'll notice that the tmm threads pass around the handle to do a health check. So my guess is the threads do only sync the health check state of the application when they're due to run the health check. Makes sense performance wise but if you operate with longer timeouts and bigger intervals you will definetly loose a notable amount of requests. Even in our low end case we already loose a notable amount of requests in 4s. While you could resent them you never know exactly if a request was already processed or not. So you have to ensure that somewhere else in your stack. One option is to disable CMP but I guess that's not an option for most people that are required to scale out to handle the load and not only for the sake of redundancy. If you're running the balancer only for the sake of redundancy the choice is easy. You find a hint on this kind of issue if you lookup the documentation for the inband monitor: Note: Systems with multiple tmm processes use a per-process number to calculate failures, depending on the specified load balancing method. That case is a bit different due to the nature of the monitor but points in the same direction. The actual check is executed by bigd according to this document, so I'm pretty sure I'm still missing some pieces of the picture. Just be warned that you've to really test your outage scenarios and you should double check it if you run a transaction based service that is sensitive to lost requests.

Jamie McClelland: Docker: evil spawn or useful tool

There are plenty of criticisms of docker, the system for building a container-based virtual machine running just a single application. I've read many of them have have consistently been either in agreement or at least amused. The most relevant criticism is about the basic approach of building single-application virtual machines. To understand this criticism, let's remember - in traditional application deployment there are at least three distinct jobs: These three groups have lived in happy tension for years. Now we have containers. The problem with containers is that suddenly the system administrators are out entirely and the distribution developers' role may have been dramatically minimized or circumvented altogether. Meanwhile, the application developer is both liberated from any constraints imposed by either the system administrator or the distribution developers (hooray!) but is also sadled with the enormous responsibilities of these two roles - which may not be apparent to the application developer at first. I have practically no distribution developer experience and have considerable experience as a system administrator and an application developer. And, it took me months to sort through how to properly develop and deploy an application using docker in a way that I thought was responsible and secure. I started by learning about how docker wants you to deploy images - by downloading them from their shared regisry. As I mentioned, I have very little experience in the realm of distribution development, but I at least know enough about Debian to know that a lot of time and thought has gone into cryptographically verifying packages that I install, which apparently is not done at all with Docker images. Is that obvious to everyone using Docker?? Fortunately, you can work around this problem by creating your own base image which is trivially easy. So, now I build all images, from scratch, locally. That helps put Debian developers back into the mix. Next, I started looking at Docker Files that I could use to construct my images and discovered something else troubling. Take, for example, the official nginx Docker image. It is based on Debian Jessie (hooray - our distribution developers are in the mix!). However, it then proceeds to install nginx from the nginx repository, not the debian repository. Well, I guess if you are nginx you want to have full control, but still, the Debian developers version of nginx has been vetted to ensure it works with Debian Jessie, so you are really losing something here. So... my next break from Docker convention is to use all Debian packages in my Docker files. Once my images were built and my application was tested and running, I was done, right? Wrong. Well, I would have been done if I didn't care about upgrading my systems, backing up my data or running cron jobs. Remember those things? Those are things that distribution developers and system administrators have been perfecting for decades. And, they are not trivial. I then built a series of scripts to help alert me when a image I am using has updates (you can't just use cron-apt any more since cron isn't running in your container) and help me update the image and deploy it to all my applications (which involves restarting the application). Backing up data is a whole different can of worms - sometimes involving interacting with your container (if it's a database container, you have to launch a database dump) or simply copying files from the host, assuming you got the right Docker volume strategy (which took me days to fully understand). Lastly, I had to run a cron job from the host that then runs whatever commands are needed on each of my containers. This was complicated. In the end, was it worth it? Yes, I think so. However, not because it was simple, which seems to be the Docker mantra. I think it's worth it because I can run 100 instances of my application using significantly less resources than when I was using full virtualization and because I can more easily and flexibly adjust the resource allocation. However, check in with me in a year and I'll probably have a different opinion.

Lunar: Reproducible builds: week 14 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes akira submitted a patch to make cdbs export SOURCE_DATE_EPOCH. She uploded a package with the enhancement to the experimental reproducible repository. Packages fixed The following 15 packages became reproducible due to changes in their build dependencies: dracut, editorconfig-core, elasticsearch, fish, libftdi1, liblouisxml, mk-configure, nanoc, octave-bim, octave-data-smoothing, octave-financial, octave-ga, octave-missing-functions, octave-secs1d, octave-splines, valgrind. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: In contrib, Dmitry Smirnov improved libdvd-pkg with 1.3.99-1-1. Patches submitted which have not made their way to the archive yet: reproducible.debian.net Four armhf build hosts were provided by Vagrant Cascadian and have been configured to be used by jenkins.debian.net. Work on including armhf builds in the reproducible.debian.net webpages has begun. So far the repository comparison page just shows us which armhf binary packages are currently missing in our repo. (h01ger) The scheduler has been changed to re-schedule more packages from stretch than sid, as the gcc5 transition has started This mostly affects build log age. (h01ger) A new depwait status has been introduced for packages which can't be built because of missing build dependencies. (Mattia Rizzolo) debbindiff development Finally, on August 31st, Lunar released debbindiff 27 containing a complete overhaul of the code for the comparison stage. The new architecture is more versatile and extensible while minimizing code duplication. libarchive is now used to handle cpio archives and iso9660 images through the newly packaged python-libarchive-c. This should also help support a couple other archive formats in the future. Symlinks and devices are now properly compared. Text files are compared as Unicode after being decoded, and encoding differences are reported. Support for Sqlite3 and Mono/.NET executables has been added. Thanks to Valentin Lorentz, the test suite should now run on more systems. A small defiency in unquashfs has been identified in the process. A long standing optimization is now performed on Debian package: based on the content of the md5sums control file, we skip comparing files with matching hashes. This makes debbindiff usable on packages with many files. Fuzzy-matching is now performed for files in the same container (like a tarball) to handle renames. Also, for Debian .changes, listed files are now compared without looking the embedded version number. This makes debbindiff a lot more useful when comparing different versions of the same package. Based on the rearchitecturing work has been done to allow parallel processing. The branch now seems to work most of the time. More test needs to be done before it can be merged. The current fuzzy-matching algorithm, ssdeep, has showed disappointing results. One important use case is being able to properly compare debug symbols. Their path is made using the Build ID. As this identifier is made with a checksum of the binary content, finding things like CPP macros is much easier when a diff of the debug symbols is available. Good news is that TLSH, another fuzzy-matching algorithm, has been tested with much better results. A package is waiting in NEW and the code is ready for it to become available. A follow-up release 28 was made on August 2nd fixing content label used for gzip2, bzip2 and xz files and an error on text files only differing in their encoding. It also contains a small code improvement on how comments on Difference object are handled. This is the last release name debbindiff. A new name has been chosen to better reflect that it is not a Debian specific tool. Stay tuned! Documentation update Valentin Lorentz updated the patch submission template to suggest to write the kind of issue in the bug subject. Small progress have been made on the Reproducible Builds HOWTO while preparing the related CCCamp15 talk. Package reviews 235 obsolete reviews have been removed, 47 added and 113 updated this week. 42 reports for packages failing to build from source have been made by Chris West (Faux). New issue added this week: haskell_devscripts_locale_substvars. Misc. Valentin Lorentz wrote a script to report packages tested as unreproducible installed on a system. We encourage everyone to run it on their systems and give feedback!

Next.

Previous.