Search Results: "ema"

19 September 2015

Russ Allbery: Review: Magic's Promise

Review: Magic's Promise, by Mercedes Lackey
Series: Last Herald Mage #2
Publisher: DAW
Copyright: January 1990
ISBN: 0-88677-401-2
Format: Mass market
Pages: 320
Magic's Promise is the second book of the Last Herald Mage trilogy and a sequel to Magic's Pawn. As tempting as it is to skip the first book, which isn't very good, I think you'd miss a lot of the emotional dynamics between Vanyel and his family if you didn't read it. Magic's Promise opens years after the end of Magic's Pawn. Vanyel has become the most powerful Herald-mage of the kingdom, which is lucky for the kingdom because it's beset on all sides by magical attack. Heralds are being killed, countries on the border are looking for opportunities to spring, and Vanyel is being run ragged. Ragged enough that he's worried he's going to hurt someone by mistake when surprised. It's time for a desperately-needed vacation. Unfortunately for his ability to relax, that vacation means going home to visit his parents, who have never understood him and who disapprove strongly of his sexuality. (Not that he's actually had a lover in years.) It also means being in the same household as the armsmaster who broke his wrist as a child, and his father's deeply disapproving priest. And the border near his parents' lands may be heating up as well. Magic's Promise is the book where this trilogy hits its stride. After the events of the previous book, Vanyel is ridiculously overpowered and will be for the rest of the trilogy, but the problems in this book aren't the sort that raw power can fix. (That's a balance Lackey handles well in general.) There are a few places where Vanyel can simply overpower his problems, but the heart of the plot is a mystery that requires analysis, investigation, and a bit of undercover snooping. I enjoyed the defense of the kingdom plot (more than I had remembered, in fact), but the heart of this book is Vanyel and his family coming to terms with each other. Vanyel is a lot older and more confident than he was in the first book, and his service to the kingdom forces some grudging respect. But the way that grudging respect turns into real affection over the course of this book is a delight to read. This degree of reconciliation is wish-fulfillment most of the Valdemar series is heavy on wish-fulfillment but it's the sort of wish-fulfillment that matches the way that you wish the world would work. People figure out that they've been ignorant and cruel and actually change, and Vanyel learns some important lessons about giving people room to change. I particularly liked that the growth of respect wasn't one-sided. Most of the movement comes (and has to come) from Vanyel's family, but Vanyel gains some new-found appreciation and empathy for some strengths that he'd previously been blind to. Lackey's writing is still not the best here. There's something a bit awkward about the dialogue patterns in places, one gets a bit tired of the repetitive narrative focus on Vanyel's exhaustion, and the sentence-level construction of the story is more workable than delightful. You have to be invested in the characters; the beauty of the writing itself isn't going to win you over. As with a lot of Lackey's writing, I think your enjoyment will depend greatly on whether she happens to hit your emotional buttons. But the story she tells here, of people finding the inner goodness in each other and healing over past wounds, is one that I very much enjoy reading. I think this is the third time I read this book, and it's the first of the Valdemar series in this re-read that held up to my memory. So far, it's the best of the series. Followed by Magic's Price. Rating: 8 out of 10

Russ Allbery: Review: Magic's Pawn

Review: Magic's Pawn, by Mercedes Lackey
Series: Last Herald Mage #1
Publisher: DAW
Copyright: June 1989
ISBN: 0-88677-352-0
Format: Mass market
Pages: 349
Continuing my (very slow) re-read of Mercedes Lackey's Valdemar series. I've read all these books before (this series several times, in fact), but long before I started writing reviews. Vanyel is a legendary hero by the time of Talia (the hero of the first Valdemar series, starting with Arrows of the Queen). Talia was reading stories about him at the start of her story: the mage Herald who defended Valdemar against its enemies in the distant past. This is the first book of the trilogy that tells his story, starting (as Lackey's books often do) from rather inauspicious beginnings. Vanyel is the son of a border lord, and about as poorly suited for it as possible. He's small, pretty, entirely uninterested in the hammer-and-tongs sword fighting the arms master wants to teach him, and also gay, not that he has any idea that's even something that exists and has a name. His only ally was his sister, who is now off serving in Valdemar's military. Most of his life is spent hiding, feeling utterly lost and out of place, and wishing he could be a Bard. This is a Valdemar story, so if you're familiar with the series, you know what's coming next. What might come as a surprise is that "next" is quite some time into the story. Vanyel does not get rescued from his situation by a Companion. Instead, his father decides to exile him to the capital under the care of his aunt, who he's only met once and who didn't think highly of him. He expects to be even more miserable, and shuts emotionally down in anticipation. This is one of those books that I remembered as being better than it actually was, and one of the reasons why I enjoyed it less than I expected is that far too much of this book is devoted to describing Vanyel's mental state. This usually involves various elaborate emo analogies (which can be a failing of this series in general), and it's quite hard to maintain sympathy for Vanyel. Lackey gets us to feel for him at the start of the book, when he's really in a bad situation. But once he gets to the capital, he's his own worst enemy, and it's hard to avoid the desire to shake him. I also didn't remember just how much I disliked Tylendel. I'm sure it will come as a surprise to no one that Vanyel eventually meets someone who draws him out of his shell and gives him a reason to want to live. Unfortunately, that person, despite a positive surface impression, is self-obsessed, unstable, and not above manipulating Vanyel into actions that are so obviously catastrophic that it makes one want to yell at the book. I disliked this part of the book so much that even a Companion's choosing was overshadowed. The book does get a bit better after that truly awful middle, but it never hits the emotional stride that other Valdemar books hit. Lackey does introduce the Tayledras, which will be hugely important in later books in this series (and who are some of my favorite characters), but there too I prefer their later appearances. Vanyel's inability to see a good thing when it hits him in the face, punctuated by being occasionally cruel to the people who try to help him, makes it quite hard to enjoy his slow path to becoming a better person. I remember really liking this trilogy, so I think it gets better. But I also remembered liking Vanyel's claiming, and it didn't do much for me on a re-read. I can only recommend this one as a necessary preface to the later books in the trilogy, and expect to need a high tolerance for constant woe-is-me despair. Followed by Magic's Promise. Rating: 5 out of 10

17 September 2015

Julien Danjou: My interview in le Journal du Hacker

A few days ago, the French equivalent of Hacker News, called "Le Journal du Hacker", interviewed me about my work on OpenStack, my job at Red Hat and my self-published book The Hacker's Guide to Python. I've spent some time translating it into English so you can read it if you don't understand French! I hope you'll enjoy it.
Hi Julien, and thanks for participating in this interview for the Journal du Hacker. For our readers who don't know you, can you introduce you briefly?
You're welcome! My name is Julien, I'm 31 years old, and I live in Paris. I now have been developing free software for around fifteen years. I had the pleasure to work (among other things) on Debian, Emacs and awesome these last years, and more recently on OpenStack. Since a few months now, I work at Red Hat, as a Principal Software Engineer on OpenStack. I am in charge of doing upstream development for that cloud-computing platform, mainly around the Ceilometer, Aodh and Gnocchi projects.
Being myself a system architect, I follow your work in OpenStack since a while. It's uncommon to have the point of view of someone as implied as you are. Can you give us a summary of the state of the project, and then detail your activities in this project?
The OpenStack project has grown and changed a lot since I started 4 years ago. It started as a few projects providing the basics, like Nova (compute), Swift (object storage), Cinder (volume), Keystone (identity) or even Neutron (network) who are basis for a cloud-computing platform, and finally became composed of a lot more projects. For a while, the inclusion of projects was the subject of a strict review from the technical committee. But since a few months, the rules have been relaxed, and we see a lot more projects connected to cloud-computing joining us. As far as I'm concerned, I've started with a few others people the Ceilometer project in 2012, devoted to handling metrics of OpenStack platforms. Our goal is to be able to collect all the metrics and record them to analyze them later. We also have a module providing the ability to trigger actions on threshold crossing (alarm). The project grew in a monolithic way, and in a linear way for the number of contributors, during the first two years. I was the PTL (Project Technical Leader) for a year. This leader position asks for a lot of time for bureaucratic things and people management, so I decided to leave my spot in order to be able to spend more time solving the technical challenges that Ceilometer offered. I've started the Gnocchi project in 2014. The first stable version (1.0.0) was released a few months ago. It's a timeseries database offering a REST API and a strong ability to scale. It was a necessary development to solve the problems tied to the large amount of metrics created by a cloud-computing platform, where tens of thousands of virtual machines have to be metered as often as possible. This project works as a standalone deployment or with the rest of OpenStack. More recently, I've started Aodh, the result of moving out the code and features of Ceilometer related to threshold action triggering (alarming). That's the logical suite to what we started with Gnocchi. It means Ceilometer is to be split into independent modules that can work together with or without OpenStack. It seems to me that the features provided by Ceilometer, Aodh and Gnocchi can also be interesting for operators running more classical infrastructures. That's why I've pushed the projects into that direction, and also to have a more service-oriented architecture (SOA)
I'd like to stop for a moment on Ceilometer. I think that this solution was very expected, especially by the cloud-computing providers using OpenStack for billing resources sold to their customers. I remember reading a blog post where you were talking about the high-speed construction of this brick, and features that were not supposed to be there. Nowadays, with Gnocchi and Aodh, what is the quality of the brick Ceilometer and the programs it relies on?
Indeed, one of the first use-case for Ceilometer was tied to the ability to get metrics to feed a billing tool. That's now a reached goal since we have billing tools for OpenStack using Ceilometer, such as CloudKitty. However, other use-cases appeared rapidly, such as the ability to trigger alarms. This feature was necessary, for example, to implement the auto-scaling feature that Heat needed. At the time, for technical and political reasons, it was not possible to implement this feature in a new project, and the functionality ended up in Ceilometer, since it was using the metrics collected and stored by Ceilometer itself. Though, like I said, this feature is now in its own project, Aodh. The alarm feature is used since a few cycles in production, and the Aodh project brings new features on the table. It allows to trigger threshold actions and is one of the few solutions able to work at high scale with several thousands of alarms. It's impossible to make Nagios run with millions of instances to fetch metrics and triggers alarms. Ceilometer and Aodh can do that easily on a few tens of nodes automatically. On the other side, Ceilometer has been for a long time painted as slow and complicated to use, because its metrics storage system was by default using MongoDB. Clearly, the data structure model picked was not optimal for what the users were doing with the data. That's why I started Gnocchi last year, which is perfectly designed for this use case. It allows linear access time to metrics (O(1) complexity) and fast access time to the resources data via an index. Today, with 3 projects having their own perimeter of features defined and which can work together Ceilometer, Aodh and Gnocchi finally erased the biggest problems and defects of the initial project.
To end with OpenStack, one last question. You're a Python developer for a long time and a fervent user of software testing and test-driven development. Several of your blogs posts point how important their usage are. Can you tell us more about the usage of tests in OpenStack, and the test prerequisites to contribute to OpenStack?
I don't know any project that is as tested on every layer as OpenStack is. At the start of the project, there was a vague test coverage, made of a few unit tests. For each release, a bunch of new features were provided, and you had to keep your fingers crossed to have them working. That's already almost unacceptable. But the big issue was that there was also a lot of regressions, et things that were working were not anymore. It was often corner cases that developers forgot about that stopped working. Then the project decided to change its policy and started to refuse all patches new features or bug fix that would not implement a minimal set of unit tests, proving the patch would work. Quickly, regressions were history, and the number of bugs largely reduced months after months. Then came the functional tests, with the Tempest project, which runs a test battery on a complete OpenStack deployment. OpenStack now possesses a complete test infrastructure, with operators hired full-time to maintain them. The developers have to write the test, and the operators maintain an architecture based on Gerrit, Zuul, and Jenkins, which runs the test battery of each project for each patch sent. Indeed, for each version of a patch sent, a full OpenStack is deployed into a virtual machine, and a battery of thousands of unit and functional tests is run to check that no regressions are possible. To contribute to OpenStack, you need to know how to write a unit test the policy on functional tests is laxer. The tools used are standard Python tools, unittest for the framework and tox to run a virtual environment (venv) and run them. It's also possible to use DevStack to deploy an OpenStack platform on a virtual machine and run functional tests. However, since the project infrastructure also do that when a patch is submitted, it's not mandatory to do that yourself locally.
The tools and tests you write for OpenStack are written in Python, a language which is very popular today. You seem to like it more than you have to, since you wrote a book about it, The Hacker's Guide to Python, that I really enjoyed. Can you explain what brought you to Python, the main strong points you attribute to this language (quickly) and how you went from developer to author?
I stumbled upon Python by chance, around 2005. I don't remember how I hear about it, but I bought a first book to discover it and started toying with that language. At that time, I didn't find any project to contribute to or to start. My first project with Python was rebuildd for Debian in 2007, a bit later. I like Python for its simplicity, its object orientation rather clean, its easiness to be deployed and its rich open source ecosystem. Once you get the basics, it's very easy to evolve and to use it for anything, because the ecosystem makes it easy to find libraries to solve any kind of problem. I became an author by chance, writing blog posts from time to time about Python. I finally realized that after a few years studying Python internals (CPython), I learned a lot of things. While writing a post about the differences between method types in Python which is still one of the most read post on my blog I realized that a lot of things that seemed obvious to me where not for other developers. I wrote that initial post after thousands of hours spent doing code reviews on OpenStack. I, therefore, decided to note all the developers pain points and to write a book about that. A compilation of what years of experience taught me and taught to the other developers I decided to interview in the book.
I've been very interested by the publication of your book, for the subject itself, but also the process you chose. You self-published the book, which seems very relevant nowadays. Is that a choice from the start? Did you look for an editor? Can you tell use more about that?
I've been lucky to find out about others self-published authors, such as Nathan Barry who even wrote a book on that subject, called Authority. That's what convinced me it was possible and gave me hints for that project. I've started to write in August 2013, and I ran the firs interviews with other developers at that time. I started to write the table of contents and then filled the pages with what I knew and what I wanted to share. I manage to finish the book around January 2014. The proof-reading took more time than I expected, so the book was only released in March 2014. I wrote a complete report on that on my blog, where I explain the full process in detail, from writing to launching. I did not look for editors though I've been proposed some. The idea of self-publishing really convince me, so I decided to go on my own, and I have no regret. It's true that you have to wear two hats at the same time and handle a lot more things, but with a minimal audience and some help from the Internet, anything's possible! I've been reached by two editors since then, a Chinese and Korean one. I gave them rights to translate and publish the books in their countries, so you can buy the Chinese and Korean version of the first edition of the book out there. Seeing how successful it was, I decided to launch a second edition in Mai 2015, and it's likely that a third edition will be released in 2016.
Nowadays, you work for Red Hat, a company that represents the success of using Free Software as a commercial business model. This company fascinates a lot in our community. What can you say about your employer from your point of view?
It only has been a year since I joined Red Hat (when they bought eNovance), so my experience is quite recent. Though, Red Hat is really a special company on every level. It's hard to see from the outside how open it is, and how it works. It's really close to and it really looks like an open source project. For more details, you should read The Open Organization, a book wrote by Jim Whitehurst (CEO of Red Hat), which he just published. It describes perfectly how Red Hat works. To summarize, meritocracy and the lack of organization in silos is what makes Red Hat a strong organization and puts them as one of the most innovative company. In the end, I'm lucky enough to be autonomous for the project I work on with my team around OpenStack, and I can spend 100% working upstream and enhance the Python ecosystem.

15 September 2015

Norbert Preining: In Memoriam Adrian Frutiger

Just a few days ago another great font designer has passed away, Adrian Frutiger, 24 May 1928 10 September 2015. Famous in particular for his fonts Frutiger, Avenir, and the Univers family. His fonts are used throughout the worlds for visual communication like road signs. Much has been written about him: Linotype s excellent obituary, Adam Twardoch, David Airey, New York Times article. For an interview conducted with Frutiger in 1999, see this Eye Magazine article. An interesting article on Frutiger s typefaces by Charles Bigelow.
frutiger1 Here I want to recall a different quality of Adrian Frutiger his interest in the visual language of symbols, their development and interaction. His book Der Mensch und seine Zeichen (Signs and Symbols) is a profound study on the development, history, and use of all kind of symbols. In this book, translated into many languages, Frutiger explores the depth and breath of symbols, but the most important part for him, easily to be seen from the German title, is the human part ( Der Mensch ). Symbols are created, changed, and used by and for humans. His studies exhibit connections between various cultures when it comes to sign usage and design.
frutiger5 He also explores signets as modern sign language, and their importance for visual communcation and identity building.
frutiger4 The book also discusses the development of the Roman alphabet, and of course passes the Univers font family.
frutiger2 My favorite piece is this overlay of a s from different fonts, to show that there is a common pattern in these fonts.
frutiger3 He concludes this book with a very wise statement:
Zum Festhalten des Gedankens, zur Vermittlung der Aussage gen gen seit langem schon die Alphabet-Zeichen nicht mehr allein. Orientierung und Kommunikation sind heute unm glich ohne Schemata, Zeichen und Signale. Der geschriebene Ausdruck wird durch die Bildvermittlung notwendig erg nzt.
SInce long the alphabet letters alone do not suffice to take hold of thoughts and to transmit statements. Nowadays, orientation and communication are impossible without schemata, signs and signals. The written expression is necessarily enhanced through image transmission.
(my translation) After Hermann Zapf, this is now the second loss of the typographic world in this year. We will miss both of them.

14 September 2015

Lunar: Reproducible builds: week 20 in Stretch cycle

What happened in the reproducible builds effort this week: Media coverage Motherboard published an article on the project inspired by the talk at the Chaos Communication 15. Journalists sadly rarely pick their headlines. The sensationalist How Debian Is Trying to Shut Down the CIA got started a few rants here and there. One from OpenBSD developper Ted Unangst lead to a good email contact and some thorough comments. Toolchain fixes The modified version of gettext has been removed from the experimental toolchain. Fixing individual package seems a better approach for now. Chris Lamb sent two patches for abi-compliance-checker: one to drop the timestamp from generated HTML reports and another to make umask and timestamps deterministic in the abi tarball. Bugs submitted by Dhole lead to a discussion on the best way to adapt pod2man now that we have SOURCE_DATE_EPOCH specified. There is really a whole class of issues that are currently undiscovered waiting for tests running on a different date. This is likely to should happen soon. Chris Lamb uploaded a new version of debhelper in the reproducible repository, cherry-picking a fix for interactions between ddebs and udebs. Packages fixed The following packages became reproducible due to changes in their build dependencies: aspic, django-guardian, erlang-sqlite3, etcd, libnative-platform-java, mingw-ocaml, nose2, oar, obexftp, py3cairo, python-dugong, python-secretstorage, python-setuptools, qct, qdox, recutils, s3ql, wine. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net The configuration of all remote armhf and amd64 nodes in now finished. The remaining reproducibility tests running on the Jenkins host has been removed. armhf results and graphs are now visible in dashboard. We can now test the whole archive in 2-3 weeks using the current 12 amd64 jobs and 3 months using the current 6 armhf builders. We will be looking at improving the armhf sitation, maybe using more native systems or via arm64. (h01ger) The Jenkins UI is now more responsive since all jobs building packages have been moved to remote hosts. (h01ger) A new job has been added to collect information about build nodes to be included in the variation table. (h01ger) The currently scheduled page has been split for amd64 and armhf. They now give an overview (refreshed every minute, thanks to Chris Lamb) of the packages currently being tested. (h01ger) Several cleanup and bugfixes have been made, especially in the remote building and maintenance scripts. They should now be more robust against network problems. The automatic scheduler is now also run closer to when schroots and pbuilders are updated. (h01ger, mapreri) Package reviews 16 reviews have been removed, 54 added and 55 updated this week. Santiago Vila renamed lc_messages_randomness with the more descriptive different_pot_creation_date_in_gettext_mo_files. New issues added this week: timestamps_in_reports_generated_by_abi_compliance_checker, umask_and_timestamp_variation_in_tgz_generated_by_abi_compliance_checker, and timestamps_added_by_blast2. 23 new FTBFS bugs have been filled by Chris Lamb, and Niko Tyni. Misc. Red Hat developper Mike McLean had a talk at Flock 2015 about reproducible builds in Koji. Slides and video recording are available. Koji is the build infrastructure used by Fedora, Red Hat and other distributions. It already keeps track of the environment used for a given build, so the required changes for handling the environment are smaller than the ones in Debian. Fedora is still missing a team effort to fix non-determinism in the package builds, but it is great to see Fedora moving forward.

13 September 2015

Russ Allbery: Review: My Real Children

Review: My Real Children, by Jo Walton
Publisher: Tor
Copyright: May 2014
ISBN: 0-7653-3265-5
Format: Hardcover
Pages: 317
Patricia, the one form of her name that she's never used, is what the staff in the nursing home call her. "VC" is the frequent annotation on her daily chart: very confused. She's old, and forgetful, and trying very hard not to be her mother, and she remembers two lives. Not only remembers but lives: the nursing home isn't the same from day to day. They keep misplacing the lift, and the staff keep changing. But the memories are the strongest. She has two lives, two sets of children, all of whom she recognizes. When someone from one life visits, the other feels like a dream, but both of them persist. Two lives that divided at the most significant decision of her life. My Real Children is one of those books that fits in the broader speculative fiction genre as alternate history, but also has many of the virtues of mimetic fiction. It's the story of one woman's life, told, except for the opening frame, chronologically from start to finish. At the start, it's simple biography, telling Patty's story from childhood, through World War II, until the moment of her fateful decision in 1949. From there the narrative splits, and she lives two lives. And the world splits as well: in one, Kennedy was assassinated by a bomb; in another, the Cuban Missile Crisis escalated into something much worse. Much is still unchanged, including the general lives of people Patty knew well in one life and not in another, but the worlds keep diverging from each other and from ours. The exact nature of the divergence is not, however, a significant part of the book. This is not the type of alternate history that is obsessed with the point of change, or even the implications for the world. Rather, this novel is about one woman, one decision, and two very different paths through life. Different friends, different challenges, different careers... different hopes and dreams, and different children. But always the same person, the same sense of practicality, the same ethics. It's beautifully told, striking just the right balance of showing important moments in detail and passing quickly, but smoothly, over some years. And, as with Among Others, Walton strikes the perfect emotional balance, leaving some reactions understated, letting the reader bring their own reactions to the book, and filling it with small bits of telling, factual detail rather than internal monologues. This is also one of the most heart-wrenching books I've ever read. My Real Children is a whole life, and not an easy one, complete with the heartaches and betrayals, the good and bad decisions, care for elderly relatives, dementia, tragedy, and even abuse. And moments of shining joy or profound satisfaction, but made all the more poignant because they're fleeting. The reader knows, in a sense, how the story ends, which casts a shadow over the rest of the book. But, even without that shadow, life can be hard, or cruel, alongside the wonder and joy. And it ends, and the ending is not neat and clean and uplifting, although one can choose how one meets it. This is the story of a life, all of it, and how choices can divide that life, and how they can cast a long shadow over how one later defines oneself. This book hurts. A lot. Enough that I want to warn you in advance I was in tears through the last third of the book. But it hurts because it's so very real and deep, not via a blow-by-blow description of daily actions, but by striding unafraid into the deep complications of choice and consequence. It's also a profoundly feminist book, in a quiet and descriptive way. It shared the Tiptree award for 2015 and fully deserved it. Both of the protagonist's lives are struggles against expectations, around gender, sexuality, and the role of women in society. Those struggles take very different forms, but they share some deep similarities. Walton shows the impact from the bottom up: the effects on specific lives, specific people, specific situations. It's brilliantly done. One can't help but think of The Female Man given the telling of parallel lives, but this is a case study, whereas Russ's novel was a battle cry. It gets under your skin instead of in your face. (Use of subtlety on this topic is easier now, forty years of occasional progress past when Russ wrote her classic.) This is a much harder book to read than Among Others. It's less forgiving, less reassuring, less willing to provide a happy ending. Lives rarely have clean, happy endings; one has to construct them, by choosing the material to build the ending on. But it's a brilliant, profound book, in that way that stories about people in all their complexity can be. Be cautious about your mood when you read this, particularly if you've had the experience of caring for a dying relative, but it's a book I will remember for a very long time. Rating: 9 out of 10

Steve Kemp: All about sharing files easily

Although I've been writing a bit recently about file-storage, this post is about something much more simple: Just making a random file or two available on an ad-hoc basis. In the past I used to have my email and website(s) hosted on the same machine, and that machine was well connected. Making a file visible just involved running ~/bin/publish, which used scp to write a file beneath an apache document-root. These days I use "my computer", "my work computer", and "my work laptop", amongst other hosts. The SSH-keys required to access my personal boxes are not necessarily available on all of these hosts. Add in firewall constraints and suddenly there isn't an obvious way for me to say "Publish this file online, and show me the root". I asked on twitter but nothing useful jumped out. So I ended up writing a simple server, via sinatra which would allow: This worked, and was even secure-enough, given that I run SSL if you import my CA file. But using basic auth felt like cheating, and I've been learning more Go recently, and I figured I should start taking it more seriously, so I created a small repository of learning-programs. The learning programs started out simply, but I did wire up a simple TOTP authenticator. Having TOTP available made me rethink things - suddenly even if you're not using SSL having an eavesdropper doesn't compromise future uploads. I'd also spent a few hours working out how to make extensible commands in go, the kind of thing that lets you run: 
cmd sub-command1 arg1 arg2
cmd sub-command2 arg1 .. argN
The solution I came up with wasn't perfect, but did work, and allow the seperation of different sub-command logic. So suddenly I have the ability to run "subcommands", and the ability to authenticate against a time-based secret. What is next? Well the hard part with golang is that there are so many things to choose from - I went with gorilla/mux as my HTTP-router, then I spend several hours filling in the blanks. The upshot is now that I have a TOTP-protected file upload site: 
publishr init    - Generates the secret
publishr secret  - Shows you the secret for import to your authenticator
publishr serve   - Starts the HTTP daemon
Other than a lack of comments, and test-cases, it is complete. And stand-alone. Uploads get dropped into ./public, and short-links are generated for free. If you want to take a peak the code is here: The only annoyance is the handling of dependencies - which need to be "go got ..". I guess I need to look at godep or similar, for my next learning project. I guess there's a minor gain in making this service available via golang. I've gained protection against replay attacks, assuming non-SSL environment, and I've simplified deployment. The downside is I can no longer login over the web, and I must use curl, or similar, to upload. Acceptible tradeoff.

12 September 2015

Ritesh Raj Sarraf: Google Take Action - Encryption

So I got an email yesterday from Google's Jess Hemerly, talking about how they care about encryption To quote a small snippet of the email: 
We use locks to keep our homes and our possessions safe offline. But how can we protect our digital things, like photos on a smartphone or email traveling across the web? The answer is encryption.
To change, it has to start from home. That means, add encryption support to Gmail. Add encryption to Google Photos. To every service you provide. So that, if tomorrow, your service is compromised, you don't public our data. What all the service providers need to realize, is that it is not the tool that needs encryption, but the data. You could claim that newer Androids are safer because encryption is enabled by default. But the data that you are backing up on your cloud, is not encrypted. It only makes sense to have an end-to-end encryption, or rather, encryption at source. Had you just stacked the interface to have write() transparently call encrypt() for any and all data, there'd be no worry about privacy. But wait, then how'd you munch.

Categories:

Keywords:

Like:

11 September 2015

Daniel Pocock: Payment for shadow work

A significant court ruling yesterday means that companies in the EU have to pay employees for time spent traveling to and from work. If something seems too good to be true, it usually is. Once you get past the headline, you find that it only applies to workers who are going to appointments outside their workplace. Some companies have been careless about scheduling such appointments, sometimes leaving an employee more than 2 hours from his home at the end of the day and this new ruling may just force them to either try a little harder to schedule appointments with consideration for the employee or pay for the extra time if excessive travel sometimes occurs. The ruling may be good news for IT workers, road congestion and the environment. With companies now becoming more efficient in managing travel time, they may well need to improve the software they use for scheduling the employee's time. With certain types of employee potentially having a shorter journey after the last appointment, this means less road congestion and less pollution. Professionals who have to fly may also be taking note: if you work in Paris but have to fly to London one day for meetings, then your working time starts when you leave your home at 05:30 and finishes at 22:00 when you get home. Shadow work comes in many forms It is good to see society taking note of the hidden cost of shadow work. What about all those other nasty and inefficient practices that take time but you don't get paid for them? Have you ever wondered why some companies send bills and statements to you by email or post, but others expect you to waste 5 minutes of your time logging in to their web site to download the bill or statement? Should people be paid for this time too and would that force these companies to stop being so lazy? Some companies have been trying to avoid doing the right thing by arguing that email is not secure, but this just doesn't add up: the original PGP MIME specs for email encryption were published through the IETF process in 1996. Barclaycard recently began offering to send credit card statements using encrypted PDF, proving that many of the excuses given by other companies are just that: excuses. For developers For people who are interested in improving the way their organization sends communications to customers, a popular choice is the Apache Camel integration framework. I've deployed Camel for customer trade notification and other purposes at several large and well known firms in financial services. It has a rich set of features for transforming messages, including a PGP data format for encryption and it is able to send messages out using a vast number of connectivity options, including many of the basic ones like SMTP, SMPP (for SMS), XMPP/Jabber and message queues.

10 September 2015

Gunnar Wolf: So you want to get our book?

OK, I already bragged that our book on Operating Systems is finally printed and has, thus, been formally published. What I had not yet mentioned is how we planned its physical distribution. Yes, it is available for sale at some UNAM libraries... But coming to UNAM is sadly an option only for people who are in Mexico City. I have been quite busy, and was unable to come up with anything earlier, but I have finally finished setting up a decent although minimal web page for the book. In it, I mention the possible ways you can get your own printed copy of Fundamentos de sistemas operativos:
  • If you are in Mexico, the advised way is to call or mail the library at Instituto de Investigaciones Econ micas (+52-55)5623-0080 or ventiiec@unam.mx.
    They will ship the book (they would ship it overseas, but it'd be too expensive!) and are able to process electronic payment opetions.
    The book printed at UNAM has substantive part of its pages printed in color, and let me tell you... It's worth it.
  • If you are not in Mexico or you prefer not to deal with a human, you can buy the book from the on-demand printing service lulu.com.
    For cost reasons, it is printed in black and white, but it is the same content (minus two typos ;-) ). Lulu.com is an international company, so they will get it shipped to you cheaper and faster And I have requested the book to be made available to libraries such as Amazon and Barnes and Noble (and was told it should take a couple of weeks to have it ready there).
Of course, the book is and will always be free for downloading, and its sources are online so you can easily derive from it as well. Enjoy! (and please report me any bugs you see!)

Norbert Preining: Getting CafeOBJ via MacPorts

I have finally finished the Portfile for CafeOBJ, which allows users to install CafeOBJ via the usual port command of MacPorts. For one needs to add our local source (which only ships CafeOBJ port), but I hope to get the port into the standard distribution soon. cafeobj-logo Necessary steps:
  • Add the following line to your MacPorts sources.conf, which usually resides in /opt/local/etc/macports/sources.conf: 
    http://cafeobj.org/macport/ports.tar.gz
  • After that, call the following two commands (as root or with sudo), the first syncronizing the ports, the second one installing CafeOBJ: 
    port sync
port install cafeobj
This should give you a working installation of CafeOBJ 1.5.3. The port will be updated as soon as 1.5.4 is released (which should be rather soon). Enjoy, and please leave comments and suggestions how to improve the port file here, per email, or at our bug tracker.

8 September 2015

Dirk Eddelbuettel: It was twenty years ago today...

sgt pepper Hm, wasn't there a catchy tune that started that way? Just kidding. But about twenty years ago today I sent this email about a new Debian package upload -- and that makes it (as best as I can tell) the earliest trace of me doing Debian work. It so happened that I did upload two packages in July of 1995 as well, but it was so early in the project that we did not have a mailing list archive for such things yet (!!). And I have a vague recollection that the two in July were an adoption---whereas the post referenced above announced efax as my first new package added to the distribution. And there were more to come ... Anyway, thanks for all the fish! Twenty years with Debian has been a great, great ride. I learned a lot from a lot of fantastic people, and I hope I helped a few people along the way with a package or two I still maintain. Debian remains a truly fabulous project which I hope will go strongly for another 20 (or even 22).

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Russ Allbery: Review: Dark Currents

Review: Dark Currents, by Jacqueline Carey
Series: Agent of Hel #1
Publisher: Roc
Copyright: 2012
Printing: October 2013
ISBN: 0-451-41483-7
Format: Mass market
Pages: 399
This is another much-belated review. Hopefully I can remember the book well enough to do it justice. Daisy Johanssen has normal urban fantasy problems: her father was an incubus who (somewhat unsurprisingly) didn't stick around, she has to keep careful control of her own demonic side, her father occasionally tries to tempt her to invoke demonic power, and it's possible that she could breach the walls protecting the world from demons and bring about Armageddon. The typical sort of thing. Also, there's an unexplained murder in her home town, a mid-western US resort town whose image does not go well with eldritch murders. Daisy's only official role with the town police department is as a part-time file clerk, but this is urban fantasy, so you know the protagonist is going to be hip-deep in some sort of crime investigation. In this case, that's because the Norse goddess Hel (as distinct from the origin of Daisy's father entirely different thing) also lives here, in a manner of speaking, and Daisy is her liaison. In this case, Hel decides she's the one who should dispense justice. Carey previously has primarily been an epic fantasy writer, and this is her first foray into urban fantasy. (Well, arguably Santa Olivia and its sequel count, but those felt closer to superhero fiction to me.) The language and tone of her epic fantasy is far different than the conventions of either urban fantasy or superhero fiction, and I thought Santa Olivia was somewhat awkward in places. (None of the profanity quite worked, for instance.) Dark Currents felt smoother and more comfortable, aided by a charming mid-western feel. A few of Daisy's first-person narrative quirks wore poorly, such as every reference to "the Seven Deadlies," but Carey did a solid job with a typical out-of-her-depth urban fantasy protagonist. I'm pretty tired of the standard mythology of urban fantasy, and that is a bit of a drawback here: vampires and werewolves both feature, and I think I could happily go ten years before reading another story about vampires or hunky werewolves. Carey adds her own take on ghouls, but it's still pretty similar to lots (and lots) of other series. Thankfully, there's a bit of Norse mythology, a bit of fairy, and one extremely memorable lamia (of the serpent-tail-below-the-waist variety, at least some of the time). Lurine, a reclusive, retired starlet, a friend of Daisy's mother, and Daisy's childhood babysitter, is the lamia in question. She absolutely steals the show. The dynamic between her and Daisy is perfect: Carey resists the urge to ruin it with dark hints or falling-outs to increase tension. Instead, she lets it be a bit quirky, a bit complicated, but full of the reliability of a deep friendship. And Lurine is a wonderful character, the kind that only works as a supporting character because she's best seen through other people's eyes. Daisy's mother just makes the dynamic better. The reader can believe she's the sort of person who would have an ill-advised fling with an incubus, but also the sort of person who could raise a daughter like Daisy without many resources other than sheer grit. She never loses her optimistic outlook on the world, but has enough backbone behind it to make Lurine's friendship with her entirely believable. The plot here is nothing particularly special. Daisy unravels various clues, gets deeper into complicated relationships between various factions, occasionally finds herself in serious trouble, and relies on the help of friends and allies. I would have enjoyed the book more with a lot less Cody and more of the less typical mythology. But the relationships between the female characters, and the sheer persistence of mid-western normality and ordinary life challenges in the face of so much supernatural meddling, help this book stand out from the crowd. Dark Currents is different from most of what Carey has written, and you may be a bit disappointed if you were hoping for the epic sweep of the Kushiel series. But I found myself charmed by it, and will be reading the next book of the series. Followed by Autumn Bones. Rating: 7 out of 10

7 September 2015

Ben Armstrong: Hike at Blomidon Park: Late Summer, 2015

I had the wonderful privilege to go camping and hiking with my kids scouting group, the Pathfinders of Tantallon SDA church. The day started with a quick trip to Pugwash with one of the leaders to bring back some chairs to their school, and then we headed back out to Blomidon to meet up with the group. Click the photo below to start the slideshow.
The road trip started early to fetch some chairsThe road trip started early to fetch some chairs click to start
The smudgy truck windows make an interesting filter Still an hour or more away from our first stop More funky filtering, this time with trees participating Wentworth valley taken over the cluttered dash We disturbed a great blue heron s breakfast at Wallace The beach at Pugwash SDA Camp where we loaded the chairs Trucking along past Truro After dropping off chairs, finally approaching Blomidon Getting very close to Blomidon Some bikers out to enjoy the views Interesting white berries Interesting red berries My first up close look at the point with my hiking buddy, Dave, on the first day An experimental panorama. Not sure I have the knack for keeping the horizon straight. We must bring the group out here tomorrow! Fast ringneck snake! Hard to get a clear shot Pre-dawn over the campground The first blush of coming dawn The Moon and Venus just before dawn Evergreens surrounding our camp site, pre-dawn Seems I m still the only one up Half of the tents on the spacious group site Half of the tents on the spacious group site My daughter, the artist My two youngest and their best friend Some relaxing down time after breakfast Not sure who said what, but apparently they were hilarious. :) The smoke was a bit much for my eldest Geoff entertaining the troops Dave making breakfast Dave making breakfast Breakfast just wrapping up Relaxing while we finish breakfast Looks like that needs some tweaking A bit too smoky The whole group The whole group Just goofing around Who s winning? Enjoying the last embers of the breakfast fire before heading to Jodrey Trail I admire this young lady s great eye for photography She has some sweet gear A lot of old hardwoods out here Dave did this hike with me yesterday Excellent hiking buddy! Words can t describe how much more stunning these views are in person All the cameras came out Got to get that perfect shot! A tree clinging to the eroding ground above the sheer cliff A lookoff on Jodrey Trail A lookoff on Jodrey Trail Lining up her shot /a> A lookoff on Jodrey Trail A lookoff on Jodrey Trail A lookoff on Jodrey Trail A fern with sharply serrated sturdy leaves I m not familiar with Breaking camp at group site 404 One final chance to enjoy the view from the park entrance before heading home
facebooktwittergoogle_plusredditpinterestlinkedinmail

Thorsten Alteholz: My Debian Activities in August 2015

FTP assistant Another month passed and another statistic arrives: This month I marked 408 packages for accept and rejected only 32 of them. Almost like last month I had to send 14 emails to maintainers. Squeeze LTS This was my fourteenth month that I did some work for the Squeeze LTS initiative, started by Raphael Hertzog at Freexian. As anybody else visited Debconf15, I got assigned a workload of 17h this month. I spent some time to prepare an upload of php5, which I did to people.d.o to let others do some tests with the package. I also uploaded some DLAs
  • [DLA 290-1] xmltooling security update Thorsten Alteholz
  • [DLA 290-2] opensaml2 security update Thorsten Alteholz
  • [DLA 292-1] libstruts1.2-java security update Thorsten Alteholz
  • [DLA 296-1] extplorer security update Thorsten Alteholz
  • [DLA 297-1] wesnoth-1.8 security update Thorsten Alteholz
  • [DLA 298-1] roundup security update Thorsten Alteholz
The patch for [DLA 290-1] was prepared by Ferenc Wagner. As opensaml2 needed a rebuild with the new version of xmltooling, that upload got [DLA-290-2] instead of an own DLA-number. This month I also had another term of doing frontdesk work. So I answered questions on the IRC channel and looked for CVEs that are important for Squeeze LTS or could be ignored. Other stuff As $WORK needed some time this month, my other activities had been almost nil. But expect more to come in September :-) .

5 September 2015

Russell Coker: A Long Term Review of Android Devices

Xperia X10 My first Android device was The Sony Ericsson Xperia X10i [1]. One of the reasons I chose it was for the large 4 screen, nowadays the desirable phones (the ones that are marketed as premium products) are all bigger than that (the Galaxy S6 is 5.1 ) and even the slightly less expensive phones are bigger. At the moment Aldi is advertising an Android phone with a 4.5 screen for $129. But at the time there was nothing better in the price range that I was willing to pay. I devoted a lot of my first review to the default apps for SMS and Email. Shortly after that I realised that the default email app is never going to be adequate (I now use K9 mail) and the SMS app is barely adequate (but I mostly use instant messaging). I ve got used to the fact that most apps that ship with an Android device are worthless, the camera app and the app to make calls are the only built in apps I regularly use nowadays. In the bug list from my first review the major issue was lack of Wifi tethering which was fixed by an update to Android 2.3. Unfortunately Android 2.3 ran significantly more slowly which decreased the utility of the phone. The construction of the phone is very good. Over the last 2 years the 2 Xperia X10 phones I own have been on loan to various relatives, many of whom aren t really into technology and can t be expected to take good care of things. But they have not failed in any way. Apart from buying new batteries there has been no hardware failure in either phone. While 2 is a small sample size I haven t see any other Android device last nearly as long without problems. Unfortunately I have no reason to believe that Sony has continued to design devices as well. The Xperia X10 phones crash more often than most Android phones with spontaneous reboots being a daily occurrence. While that is worse than any other Android device I ve used it s not much worse. My second review of the Xperia X10 had a section about ways of reducing battery use [2]. Wow, I d forgotten how much that sucked! When I was last using the Xperia X10 the Life360 app that my wife and I use to track each other was taking 15% of the battery, on more recent phones the same app takes about 2%. The design of modern phones seems to be significantly more energy efficient for background tasks and the larger brighter displays use more energy instead. My father is using one of the Xperia phones now, when I give him a better phone to replace it I will have both as emergency Wifi access points. They aren t useful for much else nowadays. Samsung Galaxy S In my first review of the Galaxy S I criticised it for being thin, oddly shaped, and slippery [3]. After using it for a while I found the shape convenient as I could easily determine the bottom of the phone in my pocket and hold it the right way up before looking at it. This is a good feature for a phone that s small enough to rotate in my pocket the Samsung Galaxy Note series of phones is large enough to not rotate in a pocket. In retrospect I think that being slippery isn t a big deal as almost everyone buys a phone case anyway. But it would still be better for use on a desk if the bulge was at the top. I wrote about my Galaxy S failing [4]. Two of my relatives had problems with those phones too. Including a warranty replacement I ve seen 4 of those phones in use and only one worked reliably. The one that worked reliably is now being used by my mother, it s considerably faster than the Xperia X10 because it has more RAM and will probably remain in regular use until it breaks. CyanogenMod I tried using CyanogenMod [5]. The phone became defective 9 months later so even though CyanogenMod is great I don t think I got good value for the amount of time spent installing it. I haven t tried replacing the OS of an Android phone since then. I really wish that they would start manufacturing phones that can have the OS replaced as easily as a PC. Samsung Galaxy S3 and Wireless Charging The Galaxy S3 was the first phone I owned which competes with phones that are currently on sale [6]. A relative bought one at the same time as me and her phone is running well with no problems. But my S3 had some damage to it s USB port which means that the vast majority of USB cables don t charge it (only Samsung cables can be expected to work). After I bought the S3 I bought a Qi wireless phone charging device [7]. One of the reasons for buying that is so if a phone gets a broken USB port then I can still use it. It s ironic that the one phone that had a damaged USB port also failed to work correctly with the Qi card installed. The Qi charger is gathering dust. One significant benefit of the S3 (and most Samsung phones) is that it has a SD socket. I installed a 32G SD card in the S3 and now one of my relatives is happily using it as a media player. Nexus 4 I bought a Nexus 4 [8] for my wife as she needed a better phone but didn t feel like paying for a Galaxy S3. The Nexus 4 is a nice phone in many ways but the lack of storage is a serious problem. At the moment I m only keeping it to use with Google Cardboard, I will lend it to my parents soon. In retrospect I made a mistake buying the Nexus 4. If I had spent a little more money on another Galaxy S3 then I would have had a phone with a longer usage life as well as being able to swap accessories with my wife. The Nexus 4 seems reasonably solid, the back of the case (which is glass) broke on mine after a significant impact but the phone continues to work well. That s a tribute to the construction of the phone and also the Ringke Fusion case [9]. Generally the Nexus 4 is a good phone so I don t regret buying it. I just think that the Galaxy S3 was a better choice. Galaxy Note 2 I got a Samsung Galaxy Note 2 in mid 2013 [10]. In retrospect it was a mistake to buy the Galaxy S3, the Note series is better suited to my use. If I had known how good it is to have a larger phone I d have bought the original Galaxy Note when it was first released. Generally everything is good about the Note 2. While it only has 16G of storage (which isn t much by today s standards) it has an SD socket to allow expansion. It s currently being used by a relative as a small tablet. With a 32G SD card it can fit a lot of movies. Bluetooth Speakers I received Bluetooth speakers in late 2013 [11]. I was very impressed by them but ended up not using them for a while. After they gathered dust for about a year I started using them again recently. While nothing has changed regarding my review of the Hive speakers (which I still like a lot) it seems that my need for such things isn t as great as I thought. One thing that made me start using the Bluetooth speakers again is that my phone case blocks the sound from my latest phone and makes it worse than phone sound usually is. I bought Bluetooth speakers for some relatives as presents, the relatives seemed to appreciate them but I wonder how much they actually use them. Nexus 5 The Nexus 5 [12] is a nice phone. When I first reviewed it there were serious problems with overheating when playing Ingress. I haven t noticed such problems recently so I think that an update to Android might have made it more energy efficient. In that review I was very impressed by the FullHD screen and it made me want a Note 3, at the time I planned to get a Note 3 in the second half of 2014 (which I did). Galaxy Note 3 Almost a year ago I bought the Samsung Galaxy Note 3 [13]. I m quite happy with it at the moment but I don t have enough data for a long term review of it. The only thing to note so far is that in my first review I was unhappy with the USB 3 socket as that made it more difficult to connect a USB cable in the dark. I ve got used to the socket and I can now reliably plug it in at night with ease. I wrote about Rivers jeans being the only brand that can fit a Samsung Galaxy Note series phone in the pocket [14]. The pockets of my jeans have just started wearing out and I think that it s partly due to the fact that I bought a Armourdillo Hybrid case [15] for my Note 3. I ve had the jeans for over 3 years with no noticable wear apart from the pockets starting to wear out after 10 months of using the Armourdillo case. I don t think that the Armourdillo case is bad, but the fact that it has deep grooves and hard plastic causes it to rub more on material when I take the phone out of my pocket. As I check my phone very frequently this causes some serious wear. This isn t necessarily a problem given that a phone costs 20* more than a pair of jeans, if the case was actually needed to save the phone then it would be worth having some jeans wear out. But I don t think I need more protection than a gel case offers. Another problem is that the Armourdillo case is very difficult to remove. This isn t a problem if you don t need access to your phone, IE if you use a phone like the Nexus 5 that doesn t permit changing batteries or SD cards. But if you need to change batteries, SD cards, etc then it s really annoying. My wife seems quite happy with her Armoudillo case but I don t think it was a good choice for me. I m considering abandoning it and getting one of the cheap gel cases. The sound on the Note 3 is awful. I don t know how much of that is due to a limitation in the speaker and how much is due to the case. It s quite OK for phone calls but not much good for music. Tablets I m currently on my third tablet. One was too cheap and nasty so I returned it. Another was still cheap and I hardly ever used it. The third is a Galaxy Note 10 which works really well. I guess the lesson is to buy something worthwhile so you can use it. A tablet that s slower and has less storage than a phone probably isn t going to get used much. Phone Longevity I owned the Xperia X10 for 22 months before getting the Galaxy S3. As that included 9 months of using a Galaxy S I only had 13 months of use out of that phone before lending it to other people. The Galaxy S3 turned out to be a mistake as I replaced it in only 7 months. I had the Note 2 for 15 months before getting the Note 3. I have now had the Note 3 for 11 months and have no plans for a replacement any time soon this is the longest I ve owned an Android phone and been totally satisfied with it. Also I only need to use it for another 4 months to set a record for using an Android phone. The Xperia was free as part of a telco contract. The other phones were somewhere between $500 and $600 each when counting the accessories (case, battery, etc) that I bought with them. So in 4 years and 7 months I ve spent somewhere between $1500 and $1800 on phones plus the cost of the Xperia that was built in to the contract. The Xperia probably cost about the same so I ll assume that I spent $2000 on phones and accessories. This seems like a lot. However that averages out to about $1.20 per day (and hopefully a lot less if my Note 3 lasts another couple of years). I could justify $1.20 per day for either the amount of paid work I do on Android phones or the amount of recreational activities that I perform (the Galaxy S3 was largely purchased for Ingress). Conclusion I think that phone companies will be struggling to maintain sales of high end phones in the future. When I chose the Xperia X10 I knew I was making a compromise, the screen resolution was an obvious limitation on the use of the device (even though it was one of the best devices available). The storage in the Xperia was also a limitation. Now FullHD is the minimum resolution for any sort of high-end device and 32G of storage is small. I think that most people would struggle to observe any improvement over a Nexus 5 or Note 3 at this time. I think that this explains the massive advertising campaign for the Galaxy S6 that is going on at the moment. Samsung can t sell the S6 based on it being better than previous phones because there s not much that they can do to make it obviously better. So they try and sell it for the image.
Related posts:
  1. Long Term Adverts I ve just seen a mailing list post from someone who...
  2. Samsung Galaxy S3 First Review with Power Case My new Samsung Galaxy S3 arrived a couple of days...
  3. Back to the Xperia X10 10 months ago I was given a Samsung Galaxy S...

4 September 2015

Riku Voipio: Migration to Scaleway ARM server

The C1 ServerScaleway started selling ARM based hosted server in April. I've intended to blog about this for a while, since it was time to upgrade from wheezy to jessie was timely, why not switch provider from an X86 based to ARM one at the same time? In many ways scaleway node is opposite to what "Enterprise ARM" people are working on. Each server is based on an oldish ARMv7 Quad-Core Marvell Armada XP, instead of a brand new 64-bit ARMv8 cpu. There is no UEFI, ACPI or any other "industry standards" involved, just a smooth web interface and a command line tool to manage your node(s). And the node is yours, it's not shared with others with virtualization. The picture above is a single node, which is stacked with 911 other nodes into a single rack. This week, the C1 price was dropped to a very reasonable 2.99 per month, or 0.006 per hour. Software runs on hardware, news at 11The performance is more than enough for my needs - shell, email and light web serving. dovecot, postfix, irssi and apache2 are just an apt-get away. Anyone who says you need x86 for Linux servers is forgetting that Linux software is open source, and if not already available, can be compiled to any architecture with little effort. Thus the migration pains were only because I chose to modernize configuration of dovecot and friends. Details of the new setup shall be left for another post.

Julien Danjou: Data validation in Python with voluptuous

Continuing my post series on the tools I use these days in Python, this time I would like to talk about a library I really like, named voluptuous. It's no secret that most of the time, when a program receives data from the outside, it's a big deal to handle it. Indeed, most of the time your program has no guarantee that the stream is valid and that it contains what is expected. The robustness principle says you should be liberal in what you accept, though that is not always a good idea neither. Whatever policy you chose, you need to process those data and implement a policy that will work lax or not. That means that the program need to look into the data received, check that it finds everything it needs, complete what might be missing (e.g. set some default), transform some data, and maybe reject those data in the end.
Data validation The first step is to validate the data, which means checking all the fields are there and all the types are right or understandable (parseable). Voluptuous provides a single interface for all that called a Schema. 
>>> from voluptuous import Schema
>>> s = Schema( 
...   'q': str,
...   'per_page': int,
...   'page': int,
...  )
>>> s( "q": "hello" )
 'q': 'hello' 
>>> s( "q": "hello", "page": "world" )
voluptuous.MultipleInvalid: expected int for dictionary value @ data['page']
>>> s( "q": "hello", "unknown": "key" )
voluptuous.MultipleInvalid: extra keys not allowed @ data['unknown']

The argument to voluptuous.Schema should be the data structure that you expect. Voluptuous accepts any kind of data structure, so it could also be a simple string or an array of dict of array of integer. You get it. Here it's a dict with a few keys that if present should be validated as certain types. By default, Voluptuous does not raise an error if some keys are missing. However, it is invalid to have extra keys in a dict by default. If you want to allow extra keys, it is possible to specify it. 
>>> from voluptuous import Schema
>>> s = Schema( "foo": str , extra=True)
>>> s( "bar": 2 )
 "bar": 2

It is also possible to make some keys mandatory. 
>>> from voluptuous import Schema, Required
>>> s = Schema( Required("foo"): str )
>>> s( )
voluptuous.MultipleInvalid: required key not provided @ data['foo']

You can create custom data type very easily. Voluptuous data types are actually just functions that are called with one argument, the value, and that should either return the value or raise an Invalid or ValueError exception. 
>>> from voluptuous import Schema, Invalid
>>> def StringWithLength5(value):
...     if isinstance(value, str) and len(value) == 5:
...             return value
...     raise Invalid("Not a string with 5 chars")
...
>>> s = Schema(StringWithLength5)
>>> s("hello")
'hello'
>>> s("hello world")
voluptuous.MultipleInvalid: Not a string with 5 chars

Most of the time though, there is no need to create your own data types. Voluptuous provides logical operators that can, combined with a few others provided primitives such as voluptuous.Length or voluptuous.Range, create a large range of validation scheme. 
>>> from voluptuous import Schema, Length, All
>>> s = Schema(All(str, Length(min=3, max=5)))
>>> s("hello")
"hello"
>>> s("hello world")
voluptuous.MultipleInvalid: length of value must be at most 5

The voluptuous documentation has a good set of examples that you can check to have a good overview of what you can do. Data transformation
What's important to remember, is that each data type that you use is a function that is called and returns a value, if the value is considered valid. That value returned is what is actually used and returned after the schema validation: 
>>> import uuid
>>> from voluptuous import Schema
>>> def UUID(value):
...     return uuid.UUID(value)
...
>>> s = Schema( "foo": UUID )
>>> data_converted = s( "foo": "uuid?" )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']
>>> data_converted = s( "foo": "8B7BA51C-DFF5-45DD-B28C-6911A2317D1D" )
>>> data_converted
 'foo': UUID('8b7ba51c-dff5-45dd-b28c-6911a2317d1d')

By defining a custom UUID function that converts a value to a UUID, the schema converts the string passed in the data to a Python UUID object validating the format at the same time. Note a little trick here: it's not possible to use directly uuid.UUID in the schema, otherwise Voluptuous would check that the data is actually an instance of uuid.UUID: 
>>> from voluptuous import Schema
>>> s = Schema( "foo": uuid.UUID )
>>> s( "foo": "8B7BA51C-DFF5-45DD-B28C-6911A2317D1D" )
voluptuous.MultipleInvalid: expected UUID for dictionary value @ data['foo']
>>> s( "foo": uuid.uuid4() )
 'foo': UUID('60b6d6c4-e719-47a7-8e2e-b4a4a30631ed')

And that's not what is wanted here. That mechanism is really neat to transform, for example, strings to timestamps. 
>>> import datetime
>>> from voluptuous import Schema
>>> def Timestamp(value):
...     return datetime.datetime.strptime(value, "%Y-%m-%dT%H:%M:%S")
...
>>> s = Schema( "foo": Timestamp )
>>> s( "foo": '2015-03-03T12:12:12' )
 'foo': datetime.datetime(2015, 3, 3, 12, 12, 12) 
>>> s( "foo": '2015-03-03T12:12' )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']

Recursive schemas So far, Voluptuous has one limitation so far: the ability to have recursive schemas. The simplest way to circumvent it is by using another function as an indirection. 
>>> from voluptuous import Schema, Any
>>> def _MySchema(value):
...     return MySchema(value)
...
>>> from voluptuous import Any
>>> MySchema = Schema( "foo": Any("bar", _MySchema) )
>>> MySchema( "foo":  "foo": "bar" )
 'foo':  'foo': 'bar' 
>>> MySchema( "foo":  "foo": "baz" )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']['foo']

Usage in REST API
I started to use Voluptuous to validate data in a the REST API provided by Gnocchi. So far it has been a really good tool, and we've been able to create a complete REST API that is very easy to validate on the server side. I would definitely recommend it for that. It blends with any Web framework easily. One of the upside compared to solution like JSON Schema, is the ability to create or re-use your own custom data types while converting values at validation time. It is also very Pythonic, and extensible it's pretty great to use for all of that. It's also not tied to any serialization format. On the other hand, JSON Schema is language agnostic and is serializable itself as JSON. That makes it easy to be exported and provided to a consumer so it can understand the API and validate the data potentially on its side.

Norbert Preining: How to rename a local OfflineIMAP managed folder

Since quite some time I am using OfflineIMAP to keep my mailbox status from various servers locally available on my machine. I did more or less follow the excellent guide A unix style mail setup, but didn t use folder name translation right from the beginning. So I got stuck with four folders named INBOX: acc1/INBOX, acc2/INBOX, etc. This wouldn t be a big problem, but when using Mutt with the folder sidebar patch (as available in the mutt-patched package in Debian), there is the problem that only the INBOX part is shown. offlineimap So I decided to rename all the local folders to the scheme described in the above guide. At the end I want to have a folder layout as follows: 
  Maildir/acc1/acc1
  Maildir/acc1/acc1.folder1
  Maildir/acc1/acc1.folder2
where the acc1 related to the INBOX on the remote account acc1, and the others are folders on the remote end. Unfortunately, renaming is not supported in offlineimap, it would simply download everything again at best! So I checked what needs to be done, and it turned out it is not as complicated as I thought. The necessary steps are:
  • rename the actual maildir folders
  • rename the status files in ~/.offlineimap/Account-acc1/LocalStatus/
  • rename the status files in ~/.offlineimap/Repository-acc1local/FolderValidity/
  • add the correct nametrans functions to your configuration file
  • do a dry-run test
To make this more specific, let us assume I want to reorganize the account acc1 in this way, and there are two folders synced. That is, before we start the Maildir folder contains: 
Maildir/acc1/INBOX
Maildir/acc1/folder1
Maildir/acc1/folder2
and we want to achieve that the layout is 
Maildir/acc1/acc1
Maildir/acc1/acc1.folder1
Maildir/acc1/acc1.folder2
Due to the fact that both the LocalStatus files and the FolderValidity files are naned in the same way as the folders, the following trivial script does all the necessary renaming for this simple case: 
for d in ~/Maildir/acc1 ~/.offlineimap/Account-acc1/LocalStatus ~/.offlineimap/Repository-acc1local/FolderValidity ; do
  mv $d/INBOX $d/acc1
  mv $d/folder1 $d/acc1.folder1
  mv $d/folder2 $d/acc1.folder2
done
Finally, the correct renaming has to be done by adding nametrans rules to the helper script. The functions oimaptransfolder_acc1 and localtransfolder_acc1 as defined in the above tutorial can be used as is, and need only be added to the respective Repository sections in the configuration file. After having done the necessary renaming, what remains is to give it a good test. Best way to do it is with 
offlineimap --dry-run -o -a acc1
which runs the syncronization in dry-run mode, so nothing is actually copied, and only once and only for acc1. If during this test run nothing strange appears, like many many emails are deleted on either end, then you are fine. Enjoy.

31 August 2015

Matthew Garrett: Working with the kernel keyring

The Linux kernel keyring is effectively a mechanism to allow shoving blobs of data into the kernel and then setting access controls on them. It's convenient for a couple of reasons: the first is that these blobs are available to the kernel itself (so it can use them for things like NFSv4 authentication or module signing keys), and the second is that once they're locked down there's no way for even root to modify them.

But there's a corner case that can be somewhat confusing here, and it's one that I managed to crash into multiple times when I was implementing some code that works with this. Keys can be "possessed" by a process, and have permissions that are granted to the possessor orthogonally to any permissions granted to the user or group that owns the key. This is important because it allows for the creation of keyrings that are only visible to specific processes - if my userspace keyring manager is using the kernel keyring as a backing store for decrypted material, I don't want any arbitrary process running as me to be able to obtain those keys[1]. As described in keyrings(7), keyrings exist at the session, process and thread levels of granularity.

This is absolutely fine in the normal case, but gets confusing when you start using sudo. sudo by default doesn't create a new login session - when you're working with sudo, you're still working with key posession that's tied to the original user. This makes sense when you consider that you often want applications you run with sudo to have access to the keys that you own, but it becomes a pain when you're trying to work with keys that need to be accessible to a user no matter whether that user owns the login session or not.

I spent a while talking to David Howells about this and he explained the easiest way to handle this. If you do something like the following:
$ sudo keyctl add user testkey testdata @u
a new key will be created and added to UID 0's user keyring (indicated by @u). This is possible because the keyring defaults to 0x3f3f0000 permissions, giving both the possessor and the user read/write access to the keyring. But if you then try to do something like:
$ sudo keyctl setperm 678913344 0x3f3f0000
where 678913344 is the ID of the key we created in the previous command, you'll get permission denied. This is because the default permissions on a key are 0x3f010000, meaning that the possessor has permission to do anything to the key but the user only has permission to view its attributes. The cause of this confusion is that although we have permission to write to UID 0's keyring (because the permissions are 0x3f3f0000), we don't possess it - the only permissions we have for this key are the user ones, and the default state for user permissions on new keys only gives us permission to view the attributes, not change them.

But! There's a way around this. If we instead do:
$ sudo keyctl add user testkey testdata @s
then the key is added to the current session keyring (@s). Because the session keyring belongs to us, we possess any keys within it and so we have permission to modify the permissions further. We can then do:
$ sudo keyctl setperm 678913344 0x3f3f0000
and it works. Hurrah! Except that if we log in as root, we'll be part of another session and won't be able to see that key. Boo. So, after setting the permissions, we should:
$ sudo keyctl link 678913344 @u
which ties it to UID 0's user keyring. Someone who logs in as root will then be able to see the key, as will any processes running as root via sudo. But we probably also want to remove it from the unprivileged user's session keyring, because that's readable/writable by the unprivileged user - they'd be able to revoke the key from underneath us!
$ sudo keyctl unlink 678913344 @s
will achieve this, and now the key is configured appropriately - UID 0 can read, modify and delete the key, other users can't.

This is part of our ongoing work at CoreOS to make rkt more secure. Moving the signing keys into the kernel is the first step towards rkt no longer having to trust the local writable filesystem[2]. Once keys have been enrolled the keyring can be locked down - rkt will then refuse to run any images unless they're signed with one of these keys, and even root will be unable to alter them.

[1] (obviously it should also be impossible to ptrace() my userspace keyring manager)
[2] Part of our Secure Boot work has been the integration of dm-verity into CoreOS. Once deployed this will mean that the /usr partition is cryptographically verified by the kernel at runtime, making it impossible for anybody to modify it underneath the kernel. / remains writable in order to permit local configuration and to act as a data store, and right now rkt stores its trusted keys there.

comment count unavailable comments

Next.

Previous.