Search Results: "beh"

6 March 2013

Raphael Geissert: A bashism a week: returning

Inspired by Thorsten Glaser's comment about where you can break from, this "bashism a week" is about a behaviour not implemented by bash.

return is a special built-in utility, and it should only be used on functions and scripts executed by the dot utility. That's what the POSIX:2001 specification requires.

If you return from any other scope, for example by accidentally calling it from a script that was not sourced but executed directly, the bash shell won't forgive you: it does not abort the execution of commands. This can lead to undesired behaviour.

A wide variety of shell interpreters silently handle such calls to return as if exit had been called.

An easy way to avoid such undesired behaviours is to follow the best practice of setting the e option, i.e.
set -e
. With that option set at the moment of calling return outside of the allowed scopes, bash will abort the execution, as desired.

The POSIX specification does not guarantee the above behaviour either as the result in such cases is "unspecified", however.

5 March 2013

Lisandro Damián Nicanor Pérez Meyer: My Debian freeze experience (so far)

This is the first freeze in which I'm involved with upload rights. And it turned out to be a quite interesting ride so far, so I thought it would be nice to write about it.

As some of you may know, I'm a part of the Qt/KDE team. Before the freeze I was mostly involved in leaf packages, with some patch here or there, nothing fancy. And then the freeze came...

Bugs in Qt

...and bugs appeared in Qt. But they didn't get solved, even if the patches were there. Due to personal reasons, the manpower in Qt/KDE land decreased below normal levels (which were already low).

I took the time to review them, apply them in a local branch, build and test the fixes. I did a Qt upload before, but it was a team-consented one. This time there was not much reaction in our IRC channel as it used to be, so I was doubting if going ahead or not. I asked Ana, my great friend and former sponsor, for an opinion on the subject, and she gave me a really important advice: the patches were looking good and there is one really big true: if something get's broken, it can be fixed with a later upload.

You might be asking yourself why I was that afraid of doing the upload. Well, when one maintains such a medular package for many users one has to be careful And I also got used that those "big ones" like Qt where normally handled by hand skilled people. Do not take me wrong here, it's not that those people where keeping them for themselves, it's knowing that one does not has the same skills nor experience as them.

But again, no one was able to upload and I had the chance and will to do another upload if needed, so off it went. That was Qt 4:4.8.2-2.

Then new experiences followed: asking a buildd maintainer for a giveback, asking the Release Team for an unblock (more on this later), etc. While sponsoring me, Ana gave me another excellent advice which I always keep in my mind:

You can't know **everything** about Debian.

And that also includes a not so technical skill: communicating with other teams. But finally we got this new version of Qt in testing. Cool :-)

Of course, new bugs appeared, and my lack of skills (and sometimes, time) where replaced by team work: Pino looking at patches and Sune contacting upstream. The eleven uploads that followed are a nice example of team work, even if I was the one who signed and did the uploads. Whoever uses Qt must know that these wonderful people (including those who are not so active nowadays like Modestas or Fathi) have done lots to bring the better to their users.Thank you guys!

Be careful, they might bite you back!

Coming back to the non-technical skills, sometimes you have to communicate with other teams in Debian. And each team is (naturally) a separate world: possibly different people, different goals, etc. Of course, we share the goal to make Debian the best experience we can, but we do not necessarily agree on the paths to achieve so.

During the freeze, there is a team that gets lots of pressure, and not by chance: the Release Team. They handle a very important task, which is to ride the freeze to get to a release. OK, that's what everyone knows. Now, one thing is knowing that and another is really understanding what does that means.

Of course I was in the first group. From the outside, communicating with the RT was a kind of "special art", and not an easy one. I have even been advised to not ask for more than one or two unblocks per weekend, as they might "bite me back". So I put on my flamesuit on and... launched reportbug release.debian.org.

Now I'm really happy to say that my experience was far from what I described above. And yes, I had the chance to even disagree on some stuff. But remember: non-technical skills, a.k.a. social skills. Once I started to know what was going on inside the RT (joining #debian-release was a big help for that) I learnt some nice tips to approach them. Please allow me to list some of them:

  • Remember: you are the maintainer of the package, they are like gatekeepers that are there to help us coordinate to do a release. But they don't maintain the code, you do that. So try to be verbose when needed, explain the changes and don't forget a nice diff. They need to understand what is going on: they can't read your mind.
  • They are human beings too: not everyday might be their best day (the same goes for you too!). And they are under the pressure of a release. Be patient, that finally pays off.
  • Does your changes seem not so clear? try to improve them.
  • The package has a lot of changes but you really feel they are needed? Try to explain that as good as you can.
  • Try to put yourself in their position: do we really want this? If in doubt, there is a nice way to know what they think: a pre-approval bug.
I want to make a stop in this last point. A pre-approval bug it's an unblock bug in which you edit the subject to add "pre-approval" in it. Easy, isn't it? It gives you the opportunity to know what the RT thinks before doing the upload. In other words: it gives you the chance to communicate and do things in the best possible way for all the parts involved.

I've have also seen pre-approval bugs that were really not needed. But to learn where the threshold of what can be directly uploaded and what deserves a pre-approval bug is you need to know the guidelines the RT gives you. Do you still have doubts? fire a pre-approval bug and try to be clear.

Of course, this are all fruits of my experience with the RT during this time. If the RT thinks different from what I'm writing here, please stand up: we are hear to listen to you and learn :-)

As a side note, I think I should file a wishlist bug to include the pre-approval bug option in reportbug. Yes, I'm lazy :-)

Summing up

Overall this was a very nice and positive experience. We are not done yet. Are we really done at some point? Let's hope not, because this is where the fun comes from :-)

2 March 2013

Bastian Blank: LDAP, Insignificant Space and Postfix

For some, LDAP, just like X.500, is black magic. I won't argue against. Sometimes it really shows surprising behavior. It always makes sense, if you think about what LDAP is built for. One surprising behavior is the handling of the "Insignificant Space"
LDAP supports syntax and comparator methods. The syntax specifies how the entry should look. Usually this is some form of text, but numbers or stuff like telephone numbers are supported as well. The comparators specifies how the values are compared. Most of the text comparators are defined to apply the handling of insignificant spaces. Insignificant space handling normalizes the use of spaces. First all leading and trailing spaces are removed. All the internal spaces are normalized to at most two spaces. At the end all strings starts and ends with one space to allow proper sub-string matches. The resulting strings are used for comparisons. This behavior makes sense most of the time. If the user want to find something in the directory, he usually don't cares about spaces, but about content. But I found one occurrence where this produces some grieve. Postfix supports LDAP since some time. And lets say, it does not care about spaces in its queries. This is no problem as e-mail addresses do not contain spaces. Or do they?
Yes, e-mail addresses can contain spaces. This is not widely known, but still allowed. This addresses are quoted and the command looks like RCPT TO:<" test"@example.com>. The local part is quoted and contains a space at the beginning. And this is where the problem starts. Postfix sanitizes the address. It uses a simplified internal representation of the address for all lookups. So the address gets test@example.com. This form is used in all table lookups. The LDAP table uses the internal form of the address. This address is copied verbatim into the query. This query may look this way (mail= test@example.com). It is sent to the server this way. The LDAP server applies the insignificant space modifications. The query is now interpreted and the comparator specified modifications are applied. The query gets effectively changed to (mail=test@example.com). And this is where the fun starts. Postfix accepts undeliverable mails. Depending on the setup, such LDAP queries may be used to check for valid addresses. Because of the space handling, the sender can add spaces to the beginning of the address and it will still be considered valid. In later steps this addresses are not longer valid. Addresses starting with spaces are considered invalid in some locations of Postfix. What surprised me a bit is that virtual alias handling did not map them. The unmodified addressed showed up on the LMTP backend server. That's why they showed up my radar. I would say Postfix is wrong in this case. The LDAP server applies the spec correctly and defines spaces in e-mail addresses as insignificant. Postfix on the other side considers them significant. The easiest fix would be to not allow any spaces in the LDAP table.

Russ Allbery: Review: Asimov's, June 2011

Review: Asimov's Science Fiction, June 2011
Editor: Sheila Williams
Issue: Volume 35, No. 6
ISSN: 1065-2698
Pages: 112
Sheila Williams's editorial is about a new ebook-only anthology of Asimov's stories, with some discussion of electronic sales. This sort of business editorial is what she's best at writing, although in this case I'm far enough in my reading that I'm reading it through a bit of a time machine. Silverberg's essay is about popular bubbles and is mildly interesting, although I've read quite a lot about this at greater length. I think he misses the plot at the end, though, where he brings in the 2008 collapse and works on focusing the blame on speculators rather than those selling the fraudulent investments. James Patrick Kelly has a surprisingly graceful introduction to fanfic that does a better job at conveying the underlying enthusiasm than most mainstream coverage. The review column this month is Peter Heck's workman-like survey of recently released books of interest. "The Cold Step Beyond" by Ian R. MacLeod: Most of MacLeod's writing I've read has focused on lush world-building, usually with a mix of magic and industrial revolution. This felt like a departure: a character study of a heavily-modified warrior and monster hunter against a clearly science-fictional background. The protagonist has no memory of her past beyond her training in a religious institution, and is as much machine as woman. She is on one of her first missions to hunt dangerous monsters. But she's given almost no direction, and rather than encountering a monster, she encounters a feral woman with an ancient ray gun. MacLeod has a very SFnal twist waiting in the wings, but what I liked best were the character dynamics and the interplay of duty and self-determination. (8) "All the News That's Fit" by Carol Emshwiller: This story follows Emshwiller's normal pattern of telling the story of an outsider and representative of outsider culture trying to understand a society, but her protagonist here is both more alone and more proactive than her average protagonist. She lives in a distant mountain village only rarely visited by a traveling reporter who brings news from outside. When he doesn't come, she goes out looking for him, at first to rescue him if he ran into trouble and later to find the news herself. Much of the story has the reader worrying nervously for her as she observes the large city while clearly having no idea how to live there. The ending provides an interesting and (at least for me) unexpected twist on the rural person coming to the big city, but it was marred by a level of anger from the protagonist that I never thought was well-motivated. (6) "Walking Stick Fires" by Alan DeNiro: There is a science fiction subgenre that, for lack of a better term, I'll call post-apocalyptic surrealism. This is an entry in that strange subgenre, featuring two alien bikers having mostly unmotivated adventures with strange objects and even stranger creatures on an Earth that's apparently been invaded and mostly devastated by multiple parties. Like a lot of post-apocalyptic surrealism, it never makes much sense. It reminded me of Rucker but with a bit more characterization and less hand-waving math. (4) "Apocalypse Daily" by Felicity Shoulders: I really liked the premise for this story. The protagonist is a concept designer for a social-oriented MMORPG, but one where only scores and resources carry over rather than the whole world. The goal of the game is survival: the world ends regularly, globally or locally, in a huge variety of ways, and one scores points and earns resources for further iterations by finding ways to survive that catastrophe. The game is structured to encourage cooperation, at least at first. But Katrina gets into a mildly paranoid workplace conflict with someone else on her team, leading to some significant changes in the underlying expectations of the game. Unfortunately, while the background and idea are quite solid, the direction the story goes is predictable pessimism, the conclusion doesn't say much that I found interesting, and I ended up detesting the protagonist. I would have preferred a more unique psychological twist. (6) "The Fighter" by Colin P. Davies: This is a rather short story, essentially an idea study in the context of a police stop of a futuristic entertainment fighter. This is another one of those depressing takes on a common SF trope. I didn't entirely follow the world-building, but it was all dark and pointless enough that I didn't really want to dig further. (4) "Kiss Me Twice" by Mary Robinette Kowal: I've read a few police procedurals featuring AIs before, but this is by far the best one I've read. Huang is a detective, and like all the detectives he makes extensive use of the police department AI. Unlike most of the detectives, though, he treats her as a full partner and close to human, and she reciprocates with a Mae West persona and a closer relationship with him than most of the department. They're in the middle of investigating a murder when there is a raid on the police department and the AI herself is stolen. The rest of the story entangles their investigation with the theft of the AI. The police AI is, of course, restored from backup, but that's the first time that's ever happened to her, and she's shaken by it, and more disturbed by the knowledge that there are two copies of her running. The mystery angle of the story is reasonably satisfying, but the AI characterization and Huang's careful feeling-out of their relationship and appropriate behavior on his part is the best part of the story. This is a great story: long, satisfying, and thoroughly excellent. I'd happily read more fiction in this universe. (8) Rating: 7 out of 10

1 March 2013

Joerg Jaspert: tmux - like screen, just nicer. Replacing clusterssh too.

By now I am a long time happy user of tmux instead of screen. I started using it somewhere in 2011 and by now I only have one usage of screen left over (direct ttyS0 access for one serial). As usual I have a custom config for it, and if you are interested there is a link for you: tmux.conf. But the more interesting part and why I blog is that I also wrote a little helper around tmux. Ingeniously called tm. And a colleague said I should let other people use it too and place it somewhere accessible. So, err, here it is. Or better, the usage output of it, the actual script has a link somewhere below.
tmux helper by Joerg Jaspert <joerg@ganneff.de>
Call as: tm CMD [host]...
CMD is one of
 ls          List running sessions
 s           Open ssh session to host
 ms          Open multi ssh sessions to hosts, synchronizing input
 $anything   Either plain tmux session with name of $anything or
             session according to TMDIR file
TMDIR file:
Each file in $TMDIR defines a tmux session.
Filename is the commandline option $anything
Content is defined as:
  First line: Session name
  Second line: extra tmux options
  Any following line: A hostname to open a shell with in the normal
  ssh syntax. (ie [user@]hostname)
Environment variables recognized by this script:
TMPDIR - Where tmux stores its session information
         DEFAULT: If unset: /tmp
TMSORT - Should ms sort the hostnames, so it always opens the same
         session, no matter in which order hostnames are presented
         DEFAULT: true
TMOPTS - Extra options to give to the tmux call
         Note that this ONLY affects the final tmux call to attach
         to the session, not to the earlier ones creating it
         DEFAULT: -2
TMDIR  - Where are session information files stored
         DEFAULT: $ HOME /.tmux.d
TMWIN  - Where does your tmux starts numbering its windows?
         This script tries to find the information in your config,
         but as it only checks $HOME/.tmux.conf it might fail.
         So if your window numbers start at anything different to 0,
         like mine do at 1, then you can set TMWIN to 1

It is mainly useful for using tmux as a ssh-multiplexer and replacement for clusterssh though its happy to do any other tmux session too. As the help text may already have told you, you can use it to list existing tmux sessions (oh, wow!). Honestly, that one is there because tm ls is shorter than tmux ls More interestingly, tm s HOSTNAME opens a tmux session with the first window being a ssh shell to HOSTNAME. Should you later type tm s HOSTNAME again, it will attach to the existing session. But the real kicker for me (and colleagues) actually is the ms subcommand of it. ms being short for multi-session, it is my way of no-longer using clusterssh. Basically you type tm ms HOST1 user@HOST2 and it opens a session with one tmux window consisting of two panes. Showing you host1 and host2 at the same time. And the input to both is synchronized, that is - all your input is send to both of them. You aren t limited to just two hosts, do as many as you wish. Actually, I have no idea if tmux has an upper limit, but I did already have more than 100 hosts open at once. The major limitation there is not tmux - but simply the available screenspace, as all hosts are shown in ONE window! (The one advantage of clusterssh, it uses one xterm per host) This is quite helpful in all kinds of administration work involving clusters - imaging installing a package on 5 hosts of a cluster, you usually want it on all of them - or any kind of situations where one wants to run one command on many systems. As I m already said, I m lazy. And so there is a way to easily define multisessions that one uses regulary, by creating files inside the directory behind $TMDIR, usually $ HOME /.tmux.d. They are simple text files and let you easily define sessions you can then (re)open with a simple tm whateveritsname. And if you want the whole script by now: tm

28 February 2013

Keith Packard: x-on-resize

x-on-resize: a simple display configuration daemon I like things to be automated as much as possible, and having abandoned Gnome to their own fate and switched to xfce, I missed the automatic display reconfiguration stuff. I decided to write something as simple as possible that did just what I needed. I did this a few months ago, and when Carl Worth asked what I was using, I decided to pack it up and make it available. Automatic configuration with a shell script I ve had a shell script around that I used to bind to a key press which I d hit when I plugged or unplugged a monitor. So, all I really need to do is get this script run when something happens. The missing tool here was something to wait for a change to happen and automatically invoke the script I d already written. Resize vs Configure The first version of x-on-resize just listened for ConfigureNotify events on the root window. These get sent every time anything happens with the screen configuration, from hot-plug to notification when someone runs xrandr. That was as simple as possible; the application was a few lines of code to select for ConfigureNotify events, and invoke a program provided on the command line. However, it was a bit too simple as it would also respond to manual invocations of xrandr and call the script then as well. So, as long as I was content to accept whatever the script did, things were fine. And, with a laptop that had a DisplayPort connector for my external desktop monitor, and a separate VGA connector for projectors at conferences, the script always did something useful. Then I got this silly laptop that has only DisplayPort, and for which a dongle is required to get to VGA for projectors. I probably could write something fancy to figure out the difference between a desktop DisplayPort monitor and DisplayPort to VGA dongle, but I decided that solving the simpler problem of only invoking the script on actual hotplug events would be better. So, I left the current invoke-on-resize behavior intact and added new code that watched the list of available outputs and invoked a new config script when that set changed. The final program, x-on-resize, is available via git at
git://people.freedesktop.org/~keithp/x-on-resize
I even wrote a manual page. Enjoy!

27 February 2013

Neil McGovern: Let s get physical

Two recent announcements have been made about how it's viewed that people should interact with each other. Both, in my opinion, are misguided. Firstly, Yahoo's chief executive, Marissa Mayer has announced that she's banning staff from remote working. The idea behind this announcement is simple - that "some of the best decisions and insights come from hallway and cafeteria discussions, meeting new people, and impromptu team meetings". This is absolutely spot on, but the next sentence in the leaked internal memo is more problematic: "Speed and quality are often sacrificed when we work from home".
Don't get me wrong, working remotely has some special challenges, but they are by no means insurmountable. There's lots of tips for people working remotely which will turn into a future blog post at some point (I'm running an internal training about how to do it in a couple of weeks), but three simple rules are stay connected, set a routine and take care of yourself. The second decision is one by Ubuntu to move Ubuntu Developer Summits to a purely online meeting, ditching the physical meeting. This misses the point of conferences. If we simply wanted to listen to talks and presentations, why meet up at all? Webcasts have been around for the last 20 years, and yet conferences still exist. The most important part of a conference isn't the talks, it's the "hallway track" - it's the ability for people to meet up, chat and socialise. Be this an impromptu meeting in the corridor, or over a few nice beers. Without this component, why schedule a time at all? Simply publish a list of talks over the coming 3 months, and anyone can pick the best time for them to attend. At Collabora, many of our engineers work remotely. One of the perks we offer is the ability to attend conferences, and to "touch base" and visit and work from one of the offices. It is important to recognise the importance of collaboration physically and it shouldn't be discounted the way Ubuntu has done. But it should not be seen as a silver bullet to an organisation, like Yahoo seem to be implying. Both extremes are wrong, and a balance must be struck to ensure the best outcome for productivity and innovation. (Title from an iconic 80s song)

25 February 2013

Russ Allbery: Review: The Hare with Amber Eyes

Review: The Hare with Amber Eyes, by Edmund de Waal
Publisher: Farrar, Straus and Giroux
Copyright: 2010
ISBN: 0-374-10597-9
Format: Kindle
Pages: 351
A netsuke ( ) is a miniature sculpture, often made from ivory or boxwood, generally rounded and no larger than a matchbox. They were a part of Japanese clothing starting from the 17th century: traditional garments had no pockets, but people needed a way to carry small objects around with them, so they wore pouches or small boxes attached to a cord. The cord could then be passed under the sash of the garment and be held in place there by a netsuke attached to the end of the cord, since the netsuke was too large to slip under the sash. The hare with amber eyes is one of 264 netsuke. It, and all the rest, belonged to Edmund de Waal's uncle Iggie. They sat in a vitrine in his home in Japan, where de Waal went for a two-year scholarship and was working on a book on japonisme: the passionate and creative Western misunderstanding of Japan. When Iggie later died in 1991, de Waal inherited the collection. This book is his attempt to tell their history. The Hare with Amber Eyes falls into the genre of family memoir or biography, but it's one told from an unusual angle. Edmund de Waal is a potter by profession: an artist who creates physical objects that then pass from his studio out into the world. The physicality of the netsuke their occupation of time and space and their passsage through the world fascinates him, and he tries to express that fascination to the reader and uses it as a focal point for the memoir. He tells, here, the story of his own family, but he does so via the netsuke, following them from their purchase in Paris in 1871 through one branch of the family and then another until they reach Iggie and, finally, him. Along the way, he tries to put them into context. At first, the context is primarily artistic, and the early parts of this book are something of a tour of late 19th century artistic culture in Paris. But, as the story continues, the connection becomes more emotional and the context becomes more complicated. Anyone who is a fan of the TV program Antiques Roadshow or any similar effort to trace the history of specific objects will see the immediate appeal in this presentation. Antiques are fascinating in their mute path through lives and history; attempting to reconstruct the rooms they've sat in and the conversations they've witnessed is something I've always found intriguing. But it's also very difficult to do. Most of our objects, and most owners of objects, simply don't have that sort of documentation. I was wondering, at the start of this book, how de Waal was going to manage to construct his history. The answer is that his family is, or at least was, famous enough to have letters, papers, and history in libraries throughout Europe. That's what makes the parts of the book before Iggie's living memory possible. It's a bit disappointing, if unsurprising; de Waal doesn't have any special technique to find hidden history, and most objects could never have this sort of history written about them. The first owner of the netsuke in his family is Charles Ephrussi, a critic, historian, and art historian who was a patron of the Impressionists and who was important enough in Parisian art culture that he is the inspiration of a character in Marcel Proust's Remembrance of Things Past. This means that quite a lot of this book is about people who moved through social circles and lived in ways that I find quite foreign, which made it hard to invest too much in the first part of the book. The Ephrussi family was originally from Odessa and made a fortune as grain merchants some time prior to the start of the history, and by the time de Waal opens their story, they have vast inherited wealth. There is no sign in the book that Charles needs to work in any way. The first part of the book paints a picture of the culture of art patronage partly via descriptions of Charles's rooms and personal collection, to which the netsuke were added during the surge in interest in Japanese art in Europe produced by the opening of Japan during the Meiji period. It's interesting, but for me in a distant and intellectual way. (People with a deeper interest in Impressionist art or other late 19th century art may have a different reaction.) De Waal writes at length at the start of this book about avoiding sentimentality and trying to take an honest and immersive look at the history of the netsuke. The emotional intensity that he brings to this goal seemed a bit odd, since little in the early part of the book seems to tempt towards sentimentality. But then the Dreyfus Affair enters Charles's story, followed by his gift of the vitrine and the netsuke to a cousin in Austria as a marriage gift, and I started to realize the implications of following a possession of a prominent Jewish family in Europe during the first half of the twentieth century. This is the sort of non-fiction book that you can spoil, so I won't give a detailed description of the rest of the book. But there is a specific event in the history of the netsuke that de Waal tells that would be the story most people would start with, and I now understand the sentimentality that de Waal was trying to avoid. It's a great story, but one of the things I like about this book is that de Waal puts it into a broader context and complicates the story. He includes the long periods of time when the netsuke were mostly ignored, he ties them into the history of western obsession with (and exploitation of) Japanese art, and he shows how the sturdy nature of the netsuke, meant for physical handling rather than display, led to them creating the emotional connection that made them important to the family. It's a book that deals with anti-Semitism with accuracy and specifics without being either lurid or overwhelming. It's a book that traces a line through one of the most overwritten and well-trod areas of history and manages to stay on its own path. It's a book that made me angry, and made me cry, and then made me step back and think about those emotions in a broader context and think about the benefits of leaving those emotions behind. I've never read a book quite like this before. It's slow in places (particularly if one is not very interested in art history), utterly compelling in others, and carefully avoids any simple themes or grand stories. It's entangled with some of the darkest events of recent history, but de Waal stays tightly focused on the family through the lens of the netsuke, which is valuable for keeping the story from drifting into well-worn and less unique material. I was not as completely taken by this book as the review that got me to buy it (it's an excellent review, better than this one). But I'm glad that I read it, and I feel like I now know more about the world. Rating: 7 out of 10

24 February 2013

Sylvain Le Gall: Configuration management: Puppet is worth it.

Replying to an old blog post of Martin F. Krafft: Configuration management, I want to give my point of view. The problems listed by madduck are quite common with Puppet, but I think Puppet is still worth, mostly because you can solve all these problems. Let give you my opinion on the list: True. I think the approach of puppet is not really UNIXish. It is probably on purpose. The biggest issue is probably the PKI. It breaks frequently for unknown reason. The "non intuitive configuration language" is probably a matter of taste. I think the language is not very well designed and strange, but I can cope with that. The attempt to versioning -- if I understand correcly what it means -- refers to the fact that when Puppet replace a file it moves the old file to a bucket. This is not a good thing, but you can say "backup => '.puppet-bak'" and you get almost the same behavior as ".dpkg-old". False debate. We can discuss for hours on Ruby, PHP, Java or whatever pet language people has invented. I am not a fan of Ruby but it is still nice as a general purpose language. To my mind, Ruby is still better to write daemon than bash. False debate.
   info: Caching catalog for centi.....
   info: Applying configuration version '1350597216'
   notice: Finished catalog run in 3.08 seconds
The config of this node is not complex, but 3s is not that bad for something that runs every 30min. If you need sub-second speed for this kind of thing, maybe you are not looking for this kind of tool. Does 144s of server time per day is a big deal ? With a lot more complex setup, I can reach 30s for a run, although this is the point where I manage a lot of thing with it. False and True. Augeas allows you to replace a single value (even more precise than a line). Just have a look at the augeas type. This is pretty nice and allow to do thing like replacing "Defaults env_reset" by "Defaults env_reset, !tty_tickets" in 4 lines of code. So this i not precisely "a single line of text", but there is other way to do it. False. Well if you organize your code with manifests/site.pp and manifests/classes/*.pp, it seems like there is a separation between the two. Next you can try inheritance and define to create specific high-level features. False-ish. Hey at least there are error message ;-) Now, most of the error that are related to the programming language are useless (at least as cryptic as a C++ error message). But as usual with error message in programming language True. Multi versions installation is horrible and you have to fix a lot of stuff to manage a sane overall configuration. Not sure to understand this point, I use puppet over IPv6... To whoever is considering using puppet, this is worth a try. It is a nice system that really helps to maintain a decent configuration across nodes.

23 February 2013

Russ Allbery: Surfacing

(Wasn't that the subject of the last journal post I made? Oh, no, it was just similar.) Hi. Sorry that I've been a bad correspondant! I owe lots of people mail and I'm rather behind in various things. Over the past couple of weeks, I've dealt with a couple of ant invasions, a compromised credit card number, and then a respiratory infection that completely knocked me on my ass. I'm still only back up to about 80%, although I did manage to do a complete day of work today for the first time in over a week. More normal posting and communication should hopefully resume shortly. The System of the World is a very long book, and is not, so far, all that horribly interesting, but I'm going to finally finish this series if it kills me, and illness thankfully let me get a solid start on it. Finally finishing Stephenson's Baroque Cycle so that I can stop thinking about it was one of my reading goals for the year. (I would have considered just not finishing the series, but it won an award, and I have a thing about eventually reading all the winners of certain awards so that I can judge them fairly.)

22 February 2013

Richard Hartmann: Finland I

Finland. Helsinki, Lahti, streets Arriving at Helsinki airport, we filed a claim with Lufthansa as a hard shell suitcase had a splintered corner. We were surprised that so many Finns arrived from Munich with skis, more on that later. We picked up our car and started on our way towards Koli; driving with a top speed of 100 km/h and often being limited to 80 km/h or even 60 km/h is... unusual... Finnish police/authorities seem to be obsessed with enforcing those speed limits as there are a lot of speed cameras along the way. Finnish people seem to be similarly obsessed with slot machines; there is an incredible amount of them at gas stations and a constant stream of people playing them. From an outsider's perspective, it's weird that a country as strict about one form of addiction, alcohol, and working against it vigorously, by means of taxes, would allow another form of addiction, gambling, run as freely and this allow so many slot machines. Speaking of taxes on alcohol: a single 0.33 l bottle of beer is more expensive in a Finnish supermarket than 0.5 l of beer in a German restaurant. Which also explains why supermarkets tend to have a rather large section with relatively cheap alcohol free beer. Anyway, coming back to streets: Highway intersections don't have continuous on/off ramps from which you change from one highway to another; you drive off of the highway, stop at a traffic light, and then continue onto the other highway. Weird system, but given the amount of traffic we witnessed, it's probably Good Enough (tm). Stopping for a short time in Lohti simply because it's apparently famous for winter sports competitions, we arrived at Future Freetime in Koli national park after about five to six gruelling hours of net driving through somewhat bad weather and behind slow drivers. Koli Hiking up to Ukko-Koli and its sister peaks proved to be rather exhausting as we kept on breaking through the snow cover to our knees and sometimes even our hips. Once we were up there, we realized that even though you couldn't see it in between the trees, there was fog all over the plains so we couldn't see anything. Still, it was a nice hike even if somewhat short. Note to self: Even when a trail is marked locally, if OpenStreetMap does not know about it... don't walk along it. Especially not when the going's rough already. And if there's a sign suggesting you wear snow shoes... wear snow shoes. Returning to Koli Hotel and the museum next to it, we walked over to the ski slope. The highest peak within Koli,Ukko-Koli, is 347 meters high, the local ski slope starts a good way below that. This would explain why a lot of Finns came back from Munich with their skis... Afterwards, we rented a snow mobile, without guide or supervision, and drove from Loma-Koli over lake Pielien towards Purnuniemi and in a large circle down towards lake Ryyn skyl where we turned around and went back the same way. If we thought Finnish streets don't have a lot of signs we soon realized that snow mobile tracks have even less. There are at most two or three signs pointing you in the right direction, but on the plus side, there are no posted speed limits for snow mobiles, either. In somewhat related news, snow mobiles can go at least 95 km/h. At that point, the scratched and dirty visor of your rental helmet will keep slamming down, forcing you to take one hand off the handle and thus stop accelerating to maintain stability. To round off the day, we heated up the sauna built into our little wooden hut. Running outside three times to rub myself off with snow from head to toes, I almost slipped and fell while standing still. When your feet are too hot for the snowy ground, you'll start to melt your own little pools of slippery water/snow mush within seconds. File that one under "I would never have guessed unless I had experienced it myself". Generic The MarkDown source of this blog post is not even 5 kiB in size; even in a worst case scenario, pushing this to my ikiwiki instance via git will eat up less 10 kiB of mobile data. Which is good because I have 78 MiB of international data left on this plan. This is also the reason why there are no links in this blog post: I am writing everything off line and don't want to search for the correct URLs to link to. I really wish EU regulators would start to tackle data roaming now that SMS and voice calls are being forced down into somewhat sane pricing regions by regulations. PS:
-rw-r--r-- 1 richih richih 4.6K Feb 11 22:55 11-Finland-I.mdwn
[...]
Writing objects: 100% (7/7), 2.79 KiB, done.

Richard Hartmann: Finland II

Second part; bullet points to save time

20 February 2013

Daniel Pocock: Automatically mirroring SVN repositories to git and github

Updated 21 February 2013: the scripts below have now been adapted to run conveniently from cron and mirror multiple repositories. You can find full scripts and documentation on github There are a number of scripts around, including an earlier blog post of my own, for making a once-off conversion of an SVN repository to a Git repository. However, what if the project team is not yet ready for a full move to git? You may want to start benefiting from git (e.g. using the project as a submodule of another git project, or using Travis-CI.org for automated builds) while development officially continues to take place in SVN. If you don't need to make commits into git just now, you can set up a mirror and refresh it periodically from SVN. This process can be used with various SVN repositories, including Sourceforge and Google Code SVN repositories. Process overview
  • An authors.txt file is created manually before the first run.
  • A git-svn clone of the SVN repository is created. This clone needs to be retained between replication runs.
  • On each run, a temporary git repository is created. The svn-clone is rebased and pushed to the fresh git repository.
  • Fixes (e.g. branch renames and tagging) are made in the temporary git repository.
  • The final results are then pushed from the temporary repository to the real git mirror that is being maintained.
  • Finally, the temporary git repository is deleted, but the svn-clone is kept.
Here is a diagram of the workflow: Config file Here is a sample config file for the script, it mirrors the rather complicated reSIProcate repository with 10,000 commits:
PROJECT_ROOT=/home/daniel/svn-mirror/resiprocate
AUTHORS_FILE=$ PROJECT_ROOT /authors.txt
SVN_REPO=https://svn.resiprocate.org/rep/resiprocate
SVN_LAYOUT="--trunk=main --branches=branches --tags=tags"
GIT_REPO=git@github.com:resiprocate/resiprocate.git
Of particular note here:
  • The $PROJECT_ROOT is where the svn-clone and the temporary repositories will be created
  • $GIT_REPO is the final destination repository. For testing, it could just be a local directory initialised with git init --bare .
  • In this repository, our trunk is actually called main, hence we have the layout argument --trunk=main. Many projects would just use --trunk=trunk
The script The script itself is below. I'd suggest running it manually a few times first and observing how it behaves, disk space requirements, etc.
#!/bin/bash
set -e
if [ ! $# -eq 1 ];
then
  echo "Usage:"
  echo "$0 <config-file>"
  exit 1
fi
CONFIG_FILE="$1"
. "$ CONFIG_FILE "   exit 1
SVN_CLONE="$ PROJECT_ROOT /svn-clone"
GIT_BARE="$ PROJECT_ROOT /git-bare-tmp"
if [ ! -d "$ PROJECT_ROOT " ];
then
  mkdir -p "$ PROJECT_ROOT "
fi
cd "$ PROJECT_ROOT "
if [ ! -d "$ SVN_CLONE " ];
then
  git svn clone \
    "$ SVN_REPO " \
    -A "$ AUTHORS_FILE " \
    $ SVN_LAYOUT  \
    "$ SVN_CLONE "
  cd "$ SVN_CLONE "
else
  cd "$ SVN_CLONE "
  git remote rm bare   echo "failed to delete remote:bare, proceeding anyway"
  git svn rebase \
    --fetch-all \
    -A "$ AUTHORS_FILE "
fi
git remote add bare "$ GIT_BARE "
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
if [ -d "$ GIT_BARE " ];
then
  rm -rf "$ GIT_BARE "
fi
mkdir -p "$ GIT_BARE "
cd "$ GIT_BARE "
git init --bare .
git symbolic-ref HEAD refs/heads/trunk
cd "$ SVN_CLONE "
git push bare
cd "$ GIT_BARE "
git branch -m trunk master
git for-each-ref --format='%(refname)' refs/heads/tags   \
   cut -d / -f 4   \
   while read ref;
   do
      git tag "$ref" "refs/heads/tags/$ref"
      git branch -D "tags/$ref"
   done
git remote add origin "$ GIT_REPO "
git config branch.master.remote origin
git config branch.master.merge refs/heads/master
git push --tags origin master
git push --all
rm -rf "$ GIT_BARE "
</config-file>
AttachmentSize
svn-mirror.png32.65 KB

Daniel Pocock: Automatically mirroring SVN repositories to git and github

Updated 21 February 2013: the scripts below have now been adapted to run conveniently from cron and mirror multiple repositories. You can find full scripts and documentation on github There are a number of scripts around, including an earlier blog post of my own, for making a once-off conversion of an SVN repository to a Git repository. However, what if the project team is not yet ready for a full move to git? You may want to start benefiting from git (e.g. using the project as a submodule of another git project, or using Travis-CI.org for automated builds) while development officially continues to take place in SVN. If you don't need to make commits into git just now, you can set up a mirror and refresh it periodically from SVN. This process can be used with various SVN repositories, including Sourceforge and Google Code SVN repositories. Process overview
  • An authors.txt file is created manually before the first run - see my other blog entry for convenient scripts to help generate authors.txt
  • A git-svn clone of the SVN repository is created. This clone needs to be retained between replication runs.
  • On each run, a temporary git repository is created. The svn-clone is rebased and pushed to the fresh git repository.
  • Fixes (e.g. branch renames and tagging) are made in the temporary git repository.
  • The final results are then pushed from the temporary repository to the real git mirror that is being maintained.
  • Finally, the temporary git repository is deleted, but the svn-clone is kept.
Here is a diagram of the workflow: Config file Here is a sample config file for the script, it mirrors the rather complicated reSIProcate repository with 10,000 commits:
PROJECT_ROOT=/home/daniel/svn-mirror/resiprocate
AUTHORS_FILE=$ PROJECT_ROOT /authors.txt
SVN_REPO=https://svn.resiprocate.org/rep/resiprocate
SVN_LAYOUT="--trunk=main --branches=branches --tags=tags"
GIT_REPO=git@github.com:resiprocate/resiprocate.git
Of particular note here:
  • The $PROJECT_ROOT is where the svn-clone and the temporary repositories will be created
  • $GIT_REPO is the final destination repository. For testing, it could just be a local directory initialised with git init --bare .
  • In this repository, our trunk is actually called main, hence we have the layout argument --trunk=main. Many projects would just use --trunk=trunk
The script The script itself is below. I'd suggest running it manually a few times first and observing how it behaves, disk space requirements, etc.
#!/bin/bash
set -e
if [ ! $# -eq 1 ];
then
  echo "Usage:"
  echo "$0 <config-file>"
  exit 1
fi
CONFIG_FILE="$1"
. "$ CONFIG_FILE "   exit 1
SVN_CLONE="$ PROJECT_ROOT /svn-clone"
GIT_BARE="$ PROJECT_ROOT /git-bare-tmp"
if [ ! -d "$ PROJECT_ROOT " ];
then
  mkdir -p "$ PROJECT_ROOT "
fi
cd "$ PROJECT_ROOT "
if [ ! -d "$ SVN_CLONE " ];
then
  git svn clone \
    "$ SVN_REPO " \
    -A "$ AUTHORS_FILE " \
    $ SVN_LAYOUT  \
    "$ SVN_CLONE "
  cd "$ SVN_CLONE "
else
  cd "$ SVN_CLONE "
  git remote rm bare   echo "failed to delete remote:bare, proceeding anyway"
  git svn rebase \
    --fetch-all \
    -A "$ AUTHORS_FILE "
fi
git remote add bare "$ GIT_BARE "
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
if [ -d "$ GIT_BARE " ];
then
  rm -rf "$ GIT_BARE "
fi
mkdir -p "$ GIT_BARE "
cd "$ GIT_BARE "
git init --bare .
git symbolic-ref HEAD refs/heads/trunk
cd "$ SVN_CLONE "
git push bare
cd "$ GIT_BARE "
git branch -m trunk master
git for-each-ref --format='%(refname)' refs/heads/tags   \
   cut -d / -f 4   \
   while read ref;
   do
      git tag "$ref" "refs/heads/tags/$ref"
      git branch -D "tags/$ref"
   done
git remote add origin "$ GIT_REPO "
git config branch.master.remote origin
git config branch.master.merge refs/heads/master
git push --tags origin master
git push --all
rm -rf "$ GIT_BARE "
</config-file>
AttachmentSize
svn-mirror.png32.65 KB

Vincent Bernat: lldpd 0.7.1

A few weeks ago, a new version of lldpd, a 802.1AB (aka LLDP) implementation for various Unices, has been released. LLDP is an industry standard protocol designed to supplant proprietary Link-Layer protocols such as EDP or CDP. The goal of LLDP is to provide an inter-vendor compatible mechanism to deliver Link-Layer notifications to adjacent network devices. In short, LLDP allows you to know exactly on which port is a server (and reciprocally). To illustrate its use, I have made a xkcd-like strip: xkcd-like strip for the use of LLDP If you would like more information about lldpd, please have a look at its new dedicated website. This blog post is an insight of various technical changes that have affected lldpd since its latest major release one year ago. Lots of C stuff ahead!

Version & changelog UPDATED: Guillem Jover told me how he met the same goals for libbsd :
  1. Save the version from git into .dist-version and use this file if it exists. This allows one to rebuild ./configure from the published tarball without losing the version. This also handles Thorsten Glaser s critic.
  2. Include CHANGELOG in DISTCLEANFILES variable.
Since this is a better solution, I have adopted the appropriate line of codes from libbsd. The two following sections are partly technically outdated.

Automated version In configure.ac, I was previously using a static version number that I had to increase when releasing:
AC_INIT([lldpd], [0.5.7], [bernat@luffy.cx])
Since the information is present in the git tree, this seems a bit redundant (and easy to forget). Taking the version from the git tree is easy:
AC_INIT([lldpd],
        [m4_esyscmd_s([git describe --tags --always --match [0-9]* 2> /dev/null   date +%F])],
        [bernat@luffy.cx])
If the head of the git tree is tagged, you get the exact tag (0.7.1 for example). If it is not, you get the nearest one, the number of commits since it and part of the current hash (0.7.1-29-g2909519 for example). The drawback of this approach is that if you rebuild configure from the released tarball, you don t have the git tree and the version will be a date. Just don t do that.

Automated changelog Generating the changelog from git is a common practice. I had some difficulties to make it right. Here is my attempt (I am using automake):
dist_doc_DATA = README.md NEWS ChangeLog
.PHONY: $(distdir)/ChangeLog
dist-hook: $(distdir)/ChangeLog
$(distdir)/ChangeLog:
        $(AM_V_GEN)if test -d $(top_srcdir)/.git; then \
          prev=$$(git describe --tags --always --match [0-9]* 2> /dev/null) ; \
          for tag in $$(git tag   grep -E '^[0-9]+(\.[0-9]+) 1, $$'   sort -rn); do \
            if [ x"$$prev" = x ]; then prev=$$tag ; fi ; \
            if [ x"$$prev" = x"$$tag" ]; then continue; fi ; \
            echo "$$prev [$$(git log $$prev -1 --pretty=format:'%ai')]:" ; \
            echo "" ; \
            git log --pretty=' - [%h] %s (%an)' $$tag..$$prev ; \
            echo "" ; \
            prev=$$tag ; \
          done > $@ ; \
        else \
          touch $@ ; \
        fi
ChangeLog:
        touch $@
Changelog entries are grouped by version. Since it is a bit verbose, I still maintain a NEWS file with important changes.

Core

C99 I have recently read 21st Century C which has some good bits and also handles the ecosystem around C. I have definitively adopted designated initializers in my coding style. Being a GCC extension since a long time, this is not a major compatibility problem. Without designated initializers:
struct netlink_req req;
struct iovec iov;
struct sockaddr_nl peer;
struct msghdr rtnl_msg;
memset(&req, 0, sizeof(req));
memset(&iov, 0, sizeof(iov));
memset(&peer, 0, sizeof(peer));
memset(&rtnl_msg, 0, sizeof(rtnl_msg));
req.hdr.nlmsg_len = NLMSG_LENGTH(sizeof(struct rtgenmsg));
req.hdr.nlmsg_type = RTM_GETLINK;
req.hdr.nlmsg_flags = NLM_F_REQUEST   NLM_F_DUMP;
req.hdr.nlmsg_seq = 1;
req.hdr.nlmsg_pid = getpid();
req.gen.rtgen_family = AF_PACKET;
iov.iov_base = &req;
iov.iov_len = req.hdr.nlmsg_len;
peer.nl_family = AF_NETLINK;
rtnl_msg.msg_iov = &iov;
rtnl_msg.msg_iovlen = 1;
rtnl_msg.msg_name = &peer;
rtnl_msg.msg_namelen = sizeof(struct sockaddr_nl);
With designated initializers:
struct netlink_req req =  
    .hdr =  
        .nlmsg_len = NLMSG_LENGTH(sizeof(struct rtgenmsg)),
        .nlmsg_type = RTM_GETLINK,
        .nlmsg_flags = NLM_F_REQUEST   NLM_F_DUMP,
        .nlmsg_seq = 1,
        .nlmsg_pid = getpid()  ,
    .gen =   .rtgen_family = AF_PACKET  
 ;
struct iovec iov =  
    .iov_base = &req,
    .iov_len = req.hdr.nlmsg_len
 ;
struct sockaddr_nl peer =   .nl_family = AF_NETLINK  ;
struct msghdr rtnl_msg =  
    .msg_iov = &iov,
    .msg_iovlen = 1,
    .msg_name = &peer,
    .msg_namelen = sizeof(struct sockaddr_nl)
 ;

Logging Logging in lldpd was not extensive. Usually, when receiving a bug report, I asked the reporter to add some additional printf() calls to determine where the problem was. This was clearly suboptimal. Therefore, I have added many log_debug() calls with the ability to filter out some of them. For example, to debug interface discovery, one can run lldpd with lldpd -ddd -D interface. Moreover, I have added colors when logging to a terminal. This may seem pointless but it is now far easier to spot warning messages from debug ones. logging output of lldpd

libevent In lldpd 0.5.7, I was using my own select()-based event loop. It worked but I didn t want to grow a full-featured event loop inside lldpd. Therefore, I switched to libevent. The minimal required version of libevent is 2.0.5. A convenient way to check the changes in API is to use Upstream Tracker, a website tracking API and ABI changes for various libraries. This version of libevent is not available in many stable distributions. For example, Debian Squeeze or Ubuntu Lucid only have 1.4.13. I am also trying to keep compatibility with very old distributions, like RHEL 2, which does not have a packaged libevent at all. For some users, it may be a burden to compile additional libraries. Therefore, I have included libevent source code in lldpd source tree (as a git submodule) and I am only using it if no suitable system libevent is available. Have a look at m4/libevent.m4 and src/daemon/Makefile.am to see how this is done.

Client

Serialization lldpctl is a client querying lldpd to display discovered neighbors. The communication is done through an Unix socket. Each structure to be serialized over this socket should be described with a string. For example:
#define STRUCT_LLDPD_DOT3_MACPHY "(bbww)"
struct lldpd_dot3_macphy  
        u_int8_t                 autoneg_support;
        u_int8_t                 autoneg_enabled;
        u_int16_t                autoneg_advertised;
        u_int16_t                mau_type;
 ;
I did not want to use stuff like Protocol Buffers because I didn t want to copy the existing structures to other structures before serialization (and the other way after deserialization). However, the serializer in lldpd did not allow to handle reference to other structures, lists or circular references. I have written another one which works by annotating a structure with some macros:
struct lldpd_chassis  
    TAILQ_ENTRY(lldpd_chassis) c_entries;
    u_int16_t        c_index;
    u_int8_t         c_protocol;
    u_int8_t         c_id_subtype;
    char            *c_id;
    int              c_id_len;
    char            *c_name;
    char            *c_descr;
    u_int16_t        c_cap_available;
    u_int16_t        c_cap_enabled;
    u_int16_t        c_ttl;
    TAILQ_HEAD(, lldpd_mgmt) c_mgmt;
 ;
MARSHAL_BEGIN(lldpd_chassis)
MARSHAL_TQE  (lldpd_chassis, c_entries)
MARSHAL_FSTR (lldpd_chassis, c_id, c_id_len)
MARSHAL_STR  (lldpd_chassis, c_name)
MARSHAL_STR  (lldpd_chassis, c_descr)
MARSHAL_SUBTQ(lldpd_chassis, lldpd_mgmt, c_mgmt)
MARSHAL_END;
Only pointers need to be annotated. The remaining of the structure can be serialized with just memcpy()1. I think there is still room for improvement. It should be possible to add annotations inside the structure and avoid some duplication. Or maybe, using a C parser? Or using the AST output from LLVM?

Library In lldpd 0.5.7, there are two possible entry points to interact with the daemon:
  1. Through SNMP support. Only information available in LLDP-MIB are exported. Therefore, implementation-specific values are not available. Moreover, SNMP support is currently read-only.
  2. Through lldpctl. Thanks to a contribution from Andreas Hofmeister, the output can be requested to be formatted as an XML document.
Integration of lldpd into a network stack was therefore limited to one of those two channels. As an exemple, you can have a look at how Vyatta made the integration using the second solution. To provide a more robust solution, I have added a shared library, liblldpctl, with a stable and well-defined API. lldpctl is now using it. I have followed those directions2:
  • Consistent naming (all exported symbols are prefixed by lldpctl_). No pollution of the global namespace.
  • Consistent return codes (on errors, all functions returning pointers are returning NULL, all functions returning integers are returning -1).
  • Reentrant and thread-safe. No global variables.
  • One well-documented include file.
  • Reduce the use of boilerplate code. Don t segfault on NULL, accept integer input as string, provide easy iterators,
  • Asynchronous API for input/output. The library delegates reading and writing by calling user-provided functions. Those functions can yield their effects. In this case, the user has to callback the library when data is available for reading or writing. It is therefore possible to integrate the library with any existing event-loop. A thin synchronous layer is provided on top of this API.
  • Opaque types with accessor functions.
Accessing bits of information is done through atoms which are opaque containers of type lldpctl_atom_t. From an atom, you can extract some properties as integers, strings, buffers or other atoms. The list of ports is an atom. A port in this list is also an atom. The list of VLAN present on this port is an atom, as well as each VLAN in this list. The VLAN name is a NULL-terminated string living in the scope of an atom. Accessing a property is done by a handful of functions, like lldpctl_atom_get_str(), using a specific key. For example, here is how to display the list of VLAN assuming you have one port as an atom:
vlans = lldpctl_atom_get(port, lldpctl_k_port_vlans);
lldpctl_atom_foreach(vlans, vlan)  
    vid = lldpctl_atom_get_int(vlan,
                               lldpctl_k_vlan_id));
    name = lldpctl_atom_get_str(vlan,
                                lldpctl_k_vlan_name));
    if (vid && name)
        printf("VLAN %d: %s\n", vid, name);
 
lldpctl_atom_dec_ref(vlans);
Internally, an atom is typed and reference counted. The size of the API is greatly limited thanks to this concept. There are currently more than one hundred pieces of information that can be retrieved from lldpd. Ultimately, the library will also enable the full configuration of lldpd. Currently, many aspects can only be configured through command-line flags. The use of the library does not replace lldpctl which will still be available and be the primary client of the library.

CLI Having a configuration file was requested since a long time. I didn t want to include a parser in lldpd: I am trying to keep it small. It was already possible to configure lldpd through lldpctl. Locations, network policies and power policies were the three items that could be configured this way. So, the next step was to enable lldpctl to read a configuration file, parse it and send the result to lldpd. As a bonus, why not provide a full CLI accepting the same statements with inline help and completion?

Parsing & completion Because of completion, it is difficult to use a YACC generated parser. Instead, I define a tree where each node accepts a word. A node is defined with this function:
struct cmd_node *commands_new(
    struct cmd_node *,
    const char *,
    const char *,
    int(*validate)(struct cmd_env*, void *),
    int(*execute)(struct lldpctl_conn_t*, struct writer*,
        struct cmd_env*, void *),
    void *);
A node is defined by:
  • its parent,
  • an optional accepted static token,
  • an help string,
  • an optional validation function and
  • an optional function to execute if the current token is accepted.
When walking the tree, we maintain an environment which is both a key-value store and a stack of positions in the tree. The validation function can check the environment to see if we are in the right context (we want to accept the keyword foo only once, for example). The execution function can add the current token as a value in the environment but it can also pop the current position in the tree to resume walk from a previous node. As an example, see how nodes for configuration of a coordinate-based location are registered:
/* Our root node */
struct cmd_node *configure_medloc_coord = commands_new(
    configure_medlocation,
    "coordinate", "MED location coordinate configuration",
    NULL, NULL, NULL);
/* The exit node.
   The validate function will check if we have both
   latitude and longitude. */
commands_new(configure_medloc_coord,
    NEWLINE, "Configure MED location coordinates",
    cmd_check_env, cmd_medlocation_coordinate,
    "latitude,longitude");
/* Store latitude. Once stored, we pop two positions
   to go back to the "root" node. The user can only
   enter latitude once. */
commands_new(
    commands_new(
        configure_medloc_coord,
        "latitude", "Specify latitude",
        cmd_check_no_env, NULL, "latitude"),
    NULL, "Latitude as xx.yyyyN or xx.yyyyS",
    NULL, cmd_store_env_value_and_pop2, "latitude");
/* Same thing for longitude */
commands_new(
    commands_new(
        configure_medloc_coord,
        "longitude", "Specify longitude",
        cmd_check_no_env, NULL, "longitude"),
    NULL, "Longitude as xx.yyyyE or xx.yyyyW",
    NULL, cmd_store_env_value_and_pop2, "longitude");
The definition of all commands is still a bit verbose but the system is simple enough yet powerful enough to cover all needed cases.

Readline When faced with a CLI, we usually expect some perks like completion, history handling and help. The most used library to provide such features is the GNU Readline Library. Because this is a GPL library, I have first searched an alternative. There are several of them: From an API point of view, the first three libraries support the GNU Readline API. They also have a common native API. Moreover, this native API also handles tokenization. Therefore, I have developed the first version of the CLI with this API3. Unfortunately, I noticed later this library is not very common in the Linux world and is not available in RHEL. Since I have used the native API, it was not possible to fallback to the GNU Readline library. So, let s switch! Thanks to the appropriate macro from the Autoconf Archive (with small modifications), the compilation and linking differences between the libraries are taken care of. Because GNU Readline library does not come with a tokenizer, I had to write one myself. The API is also badly documented and it is difficult to know which symbol is available in which version. I have limited myself to:
  • readline(), addhistory(),
  • rl_insert_text(),
  • rl_forced_update_display(),
  • rl_bind_key()
  • rl_line_buffer and rl_point.
Unfortunately, the various libedit libraries have a noop for rl_bind_key(). Therefore, completion and online help is not available with them. I have noticed that most BSD come with GNU Readline library preinstalled, so it could be considered as a system library. Nonetheless, linking with libedit to avoid licensing issues is possible and help can be obtained by prefixing the command with help.

OS specific support

BSD support Until version 0.7, lldpd was Linux-only. The rewrite to use Netlink was the occasion to abstract interfaces and to port to other OS. The first port was for Debian GNU/kFreeBSD, then for FreeBSD, OpenBSD and NetBSD. They all share the same source code:
  • getifaddrs() to get the list of interfaces,
  • bpf(4) to attach to an interface to receive and send packets,
  • PF_ROUTE socket to be notified when a change happens.
Each BSD has its own ioctl() to retrieve VLAN, bridging and bonding bits but they are quite similar. The code was usually adapted from ifconfig.c. The BSD ports have the same functionalities than the Linux port, except for NetBSD which lacks support for LLDP-MED inventory since I didn t find a simple way to retrieve DMI related information. They also offer greater security by filtering packets sent. Moreover, OpenBSD allows to lock the filters set on the socket:
/* Install write filter (optional) */
if (ioctl(fd, BIOCSETWF, (caddr_t)&fprog) < 0)  
    rc = errno;
    log_info("privsep", "unable to setup write BPF filter for %s",
        name);
    goto end;
 
/* Lock interface */
if (ioctl(fd, BIOCLOCK, (caddr_t)&enable) < 0)  
    rc = errno;
    log_info("privsep", "unable to lock BPF interface %s",
        name);
    goto end;
 
This is a very nice feature. lldpd is using a privileged process to open the raw socket. The socket is then transmitted to an unprivileged process. Without this feature, the unprivileged process can remove the BPF filters. I have ported the ability to lock a socket filter program to Linux. However, I still have to add a write filter.

OS X support Once FreeBSD was supported, supporting OS X seemed easy. I got sponsored by xcloud.me which provided a virtual Mac server. Making lldpd work with OS X took only two days, including a full hour to guess how to get Apple Xcode without providing a credit card. To help people installing lldpd on OS X, I have also written a lldpd formula for Homebrew which seems to be the most popular package manager for OS X.

Upstart and systemd support Many distributions propose upstart and systemd as a replacement or an alternative for the classic SysV init. Like most daemons, lldpd detaches itself from the terminal and run in the background, by forking twice, once it is ready (for lldpd, this just means we have setup the control socket). While both upstart and systemd can accommodate daemons that behave like this, it is recommended to not fork. How to advertise readiness in this case? With upstart, lldpd will send itself the SIGSTOP signal. upstart will detect this, resume lldpd with SIGCONT and assume it is ready. The code to support upstart is therefore quite simple. Instead of calling daemon(), do this:
const char *upstartjob = getenv("UPSTART_JOB");
if (!(upstartjob && !strcmp(upstartjob, "lldpd")))
    return 0;
log_debug("main", "running with upstart, don't fork but stop");
raise(SIGSTOP);
The job configuration file looks like this:
# lldpd - LLDP daemon
description "LLDP daemon"
start on net-device-up IFACE=lo
stop on runlevel [06]
expect stop
respawn
script
  . /etc/default/lldpd
  exec lldpd $DAEMON_ARGS
end script
systemd provides a socket to achieve the same goal. An application is expected to write READY=1 to the socket when it is ready. With the provided library, this is just a matter of calling sd_notify("READY=1\n"). Since sd_notify() has less than 30 lines of code, I have rewritten it to avoid an external dependency. The appropriate unit file is:
[Unit]
Description=LLDP daemon
Documentation=man:lldpd(8)
[Service]
Type=notify
NotifyAccess=main
EnvironmentFile=-/etc/default/lldpd
ExecStart=/usr/sbin/lldpd $DAEMON_ARGS
Restart=on-failure
[Install]
WantedBy=multi-user.target

OS include files Linux-specific include files were a major pain in previous versions of lldpd. The problems range from missing header files (like linux/if_bonding.h) to the use of kernel-only types. Those headers have a difficult history. They were first shipped with the C library but were rarely synced and almost always outdated. They were then extracted from kernel version with almost no change and lagged behind the kernel version used by the released distribution4. Today, the problem is acknowledged and is being solved by both the distributions which extract the headers from the packaged kernel and by kernel developers with a separation of kernel-only headers from user-space API headers. However, we still need to handle legacy. A good case is linux/ethtool.h:
  • It can just be absent.
  • It can use u8, u16 types which are kernel-only types. To work around this issue, type munging can be setup.
  • It can miss some definition, like SPEED_10000. In this case, you either define the missing bits and find yourself with a long copy of the original header interleaved with #ifdef or conditionally use each symbol. The latest solution is a burden by itself but it also hinders some functionalities that can be available in the running kernel.
The easy solution to all this mess is to just include the appropriate kernel headers into the source tree of the project. Thanks to Google ripping them for its Bionic C library, we know that copying kernel headers into a program does not create a derivative work.

  1. Therefore, the use of u_int16_t and u_int8_t types is a left-over of the previous serializer where the size of all members was important.
  2. For more comprehensive guidelines, be sure to check Writing a C library.
  3. Tokenization is not the only advantage of libedit native API. The API is also cleaner, does not have a global state and has a better documentation. All the implementations are also BSD licensed.
  4. For example, in Debian Sarge, the Linux kernel was a 2.6.8 (2004) while the kernel headers were extracted from some pre-2.6 kernel.

19 February 2013

Ingo Juergensmann: LVM on RAID5 broken - how to fix?

Some time ago one of my disks in my Software RAID5 failed. No big problem as I had two spare A08U-C2412 available to replace that single 1 TB SATA disk. I can't remember any details but somewhat went wrong and I ended up with a non-booting system. I think I tried to add the HW-RAID as a physical volume to the LVM and thought of migrating the SW-RAID to the HW-RAID or do mirroring or such. Anyway: I booted into my rescue system that was on a RAID1 partition on those disks, but LVM didn't came up anymore, because the SW-RAID5 wasn't recognized during boot. So I re-created the md device and discovered that my PV was gone as well. =:-0 No, big deal, I thought, because I have a backup of that machine on another host. I restored /etc/lvm and tried to do a vgcfgrestore after I re-created the PV with pvcreate. First I didn't use the old UUID, so vgcfgrestore complained. After creating the proper PV with UUID LVM did recognize the PV, VG and LVs. Unfortunately I can't mount any of the LVs. Something seems to be broken:
hahn-rescue:~# mount /dev/vg/sys /mnt
mount: you must specify the filesystem type
Feb 19 07:50:02 hahn-rescue kernel: [748288.740949] XFS (dm-0): bad magic number
Feb 19 07:50:02 hahn-rescue kernel: [748288.741009] XFS (dm-0): SB validate failed
Running a gpart scan on my SW-RAID5 did gave me some results:
hahn-rescue:~# gpart /dev/md3
Begin scan...
Possible partition(SGI XFS filesystem), size(20470mb), offset(5120mb)
Possible partition(SGI XFS filesystem), size(51175mb), offset(28160mb)
Possible partition(SGI XFS filesystem), size(1048476mb), offset(117760mb)
Possible partition(SGI XFS filesystem), size(204787mb), offset(1168640mb)
Possible partition(SGI XFS filesystem), size(204787mb), offset(1418240mb)
Possible partition(SGI XFS filesystem), size(1048476mb), offset(1626112mb)
*** Fatal error: dev(/dev/md3): seek failure.
These is not the complete lists of VM as a comparison to the output of lvs shows:
hahn-rescue:~# lvs
  LV                 VG   Attr     LSize   Pool Origin Data%  Move Log Copy%  Convert
  storage1           lv   -wi-ao--   1.00t                                          
  AmigaSeagateElite3 vg   -wi-a---   3.00g                                          
  audio              vg   -wi-a---  70.00g                                          
  backup             vg   -wi-a---   1.00t                                          
  data               vg   -wi-a---  50.00g                                          
  hochzeit           vg   -wi-a---  40.00g                                          
  home               vg   -wi-a---   5.00g                                          
  pics               vg   -wi-a--- 200.00g                                          
  sys                vg   -wi-a---  20.00g                                          
  video              vg   -wi-a--- 100.00g                                          
  windata            vg   -wi-a--- 100.00g   
Please notice that /dev/lv/storage1 is my HW-RAID where I stored the images of /dev/vg/-LVs to run xfs_repair and such on. Anyway, the sizes of the recognized XFS partitions by gpart is mostly correct, but some a missing and xfs_repair can not do anything good on the backup images on storage1. Everything ends up in /lost+found, because the blocks seem to be mixed up somehow. What I figured out is, that my old RAID5 device was in metaformat=1.2 whereas the new one is in format 0.9. My best guess is now to recreate the RAID5 device with format 1.2, do a vgcfgrestore on that and have (hopefully!) a working LVM with working LVs back that I can then mount again. If there's anything else I might be able to try, dear Lazyweb, please tell me. Please see the attached config files/tarballs for a complete overview. Side note: except for AmigaSeagateElite3, which is a dd image of an old Amiga SCSI disk, I should have a fairly complete backup on my second backup location, so there's not much lost, but I would be a real timesaver when I would be able to recover the lost LVs. Both systems are behind DSL/cable and have a limit of 10 Mbps upstream. It would take weeks to transfer the data and sending an USB disk would be faster.
AttachmentSize
mdadm.conf_.txt954 bytes
hahn-lvm-restore.tar.gz14.7 KB
Kategorie:

18 February 2013

John Sullivan: SCALE

I will be speaking at the Southern California Linux Expo (and yes, given the topics covered, it's missing a GNU). My talk, "Four Freedoms for Freedom," is on Sunday, February 24, 2013 from 16:30 to 17:30.
The most obvious people affected by all four of the freedoms that define free software are the programmers. They are the ones who will likely want to -- and are able to -- modify software running on their computers. But free software is a movement to advance and defend freedom for anyone and everyone using any computing device, not just programmers. In many countries now, given the ubiquity of tablets, phones, laptops and desktops, "anyone and everyone using any computing device" means nearly all citizens. But new technological innovations in these areas keep coming with new restrictions, frustrating and controlling users even while creating a perception of empowerment. The Free Software Foundation wants to gain the support and protect the interests of everyone, not just programmers. How do we reach people who have no intention of ever modifying a program, and how do we help them?
Other presentations on my list to check out (in chronological order, some conflicting): If you will be there and want to meet up, drop me a line.

16 February 2013

Daniel Pocock: Adapting the Swisscom Piccolo (Motorola 7640-47) Router for bridging and other ISPs

Background The Piccolo is actually quite a powerful router. Unfortunately, Swisscom has crippled it, removing fundamental configuration options from the web management interface, and even worse, breaking some of the (undocumented) command line configuration options. At a bare minimum, the router can be turned into a bridge/modem, enabling more powerful functionality to be implemented using a wifi router with OpenWRT or a similar solution. Documentation for the Motorola generic version of this device provides some helpful clues about the configuration and may be a useful resource for anybody adapting it. The undocumented telnet interface The username and password for the web interface also allow telnet access to administer the router. The telnet interface actually has quite a lot of features, many more than the web interface. The normal username is admin. If the admin password is not known it can be reset to 1234 using the factory reset button on the back of the router.
$ telnet 192.168.1.1
Trying 192.168.1.1...
Connected to 192.168.1.1.
Escape character is '^]'.
login: admin
Password: (default 1234)
Terminal shell v1.0
Copyright (C) 2011 Motorola, Inc. All rights reserved.
Motorola Netopia Model 7640-47 Annex A VDSL2 IAD
Running Netopia SOC OS version 9.0.10 (build h2d8)
ADSL/VDSL capable
(admin completed login: Admin account with read/write access.)
Removing the Swisscom backdoor The first step for any Piccolo owner is to remove the backdoor from Swisscom. Swisscom links the routers to a management platform that allows them to change configuration remotely, or even push out firmware updates. Allowing an external party such as your ISP to push updates to your router is a major risk. Just imagine if Swisscom decides to update firmware while you are making an online purchase or checking in to a flight - your router reboots itself and your transaction is hosed. Let's have a look at the default configuration of the router, notice these cwmp settings:
Centro_piccolo> show conf
.
.
.
set management cwmp enable on
set management cwmp acs-url "https://rms.bluewin.ch/cwmpWeb/WGCPEMgt"
set management cwmp acs-username "bluewincustomer"
set management cwmp acs-password "***************"
.
.
.
If you didn't already know that was there, you might be quite unpleasantly surprised right now. Removing it is easy:
Centro_piccolo> conf
Config Mode v1.3
Centro_piccolo (top)>> management
Centro_piccolo (management)>> cwmp enable
Centro_piccolo (management cwmp enable)>> set
      enable (on) [ off   on ]: off
Centro_piccolo (management cwmp enable)>> up
Centro_piccolo (management cwmp)>> up
Centro_piccolo (management)>> up
Centro_piccolo (top)>> save
Configuration data saved.
Centro_piccolo (top)>> exit
Notice that after you make a change, you must type the save command to persist the change. Disable the Piccolo's DHCP server If using a more powerful router (e.g. a TP-Link or Buffalo running OpenWRT) then the DHCP server in the Piccolo is not needed. Furthermore, if bridging from a provider that uses the DHCP (as opposed to PPPoE) mechanism to give you your real-world IP address, then the Piccolo's DHCP server will answer the DHCP requests from your other router, and it will never get the proper IP address from the ISP. This scenario must be avoided by disabling the DHCP server in the Piccolo.
Centro_piccolo (top)>> conn name LAN
Centro_piccolo (conn name "LAN")>> dhcp-server-enable
Centro_piccolo (conn name "LAN" dhcp-server-enable)>> set off
Centro_piccolo (conn name "LAN" dhcp-server-enable)>> save
Now, when you want to administer the Piccolo, you need to manually configure an IP address (e.g. 192.168.1.2/24) on the PC that is connecting to the Piccolo. Putting the router into bridge mode When I purchased the router, I was using it with Swisscom. Their own network operates ethernet over ATM and the router obtains an IP address using DHCP. I subsequently changed my ISP to Init7, an independent ISP. Init7, like most third-party ISPs in Switzerland, uses PPPoE rather than the bridged mode. For either mode of operation, the router can bridge the DHCP or PPPoE packets to a more flexible router, such as a TP Link WR-1043ND running OpenWRT or even a Linux server. In the normal configuration, I discovered two bridges in the router:
Centro_piccolo> show bridge interfaces
Bridge interfaces:
bridge name     bridge id               STP enabled     interfaces
br1             8000.0024c94dfcf0       no              eth0.16
                                                        eth1.16
br2             8000.0024c94dfcf1       no              ptm0
                                                        atm1
Clearly, bridging is possible. The Motorola/Netopia manuals refer to various bridge configuration options that just don't seem to exist on this unit. Nonetheless, it is possible to join those two bridges into one:
Centro_piccolo> conf
Config Mode v1.3
Centro_piccolo (top)>> link name LAN
Centro_piccolo (link name "LAN")>> port-vlan
Centro_piccolo (link name "LAN" port-vlan)>> ports
Centro_piccolo (link name "LAN" port-vlan ports)>> set
        ports ("lan-1 lan-2") [ lan-1, lan-2, ptm, vc-1 ]: "lan-1 lan-2 ptm vc-1"
Centro_piccolo (link name "LAN" port-vlan ports)>> up
Centro_piccolo (link name "LAN" port-vlan)>> up
Centro_piccolo (link name "LAN")>> up
Centro_piccolo (link)>> name WAN
Centro_piccolo (link name "WAN")>> port
Centro_piccolo (link name "WAN" port-vlan)>> view
      port-vlan
        ports "ptm vc-1"
        priority 0
Centro_piccolo (link name "WAN" port-vlan)>> ports
Centro_piccolo (link name "WAN" port-vlan ports)>> set
        ports ("ptm vc-1") [ lan-1, lan-2, ptm, vc-1 ]: ""
Centro_piccolo (link name "WAN" port-vlan ports)>> view
        ports ""
Centro_piccolo (link name "WAN" port-vlan ports)>> up
Centro_piccolo (link name "WAN" port-vlan)>> up
Centro_piccolo (link name "WAN")>> up
Centro_piccolo (link)>> save
Configuration data saved.
Centro_piccolo (conn name "WAN" link-oid)>> exit
and now let's check the results:
Centro_piccolo> show bridge interfaces
 Bridge interfaces:
bridge name     bridge id               STP enabled     interfaces
br1             8000.0024c94dfcf0       no              eth0.16
                                                        eth1.16
                                                        ptm0
                                                        atm1
Any WAN connection parameters (PPPoE or DHCP) should now be ignored by the router, and it should just be bridging PPPoE or DHCP traffic from the ISP to the other router. The Piccolo is now behaving like a modem/bridge.

15 February 2013

Iustin Pop: Garmin Montana

Garmin Montana notes Sometimes, I get waaay too excited about gadgets, and I have to share it. So, sorry, here it goes :) For some reason that is not very important, I was looking for a better solution for GPSes with maps. Side-note: if you ask why Android is not good enough , here's the story: last year, knowing that Google Maps supports caching (finally), I upgraded Maps, preloaded London, and went on a trip. Only to exit the Victoria underground station and to stare ~20 minutes in light rain at my phone, wondering why it doesn't get a fix. I gave up in the end, and used the cached maps for getting around. Later I learned the reason: Maps was not broken, but without data, Android can't get a lock easily; and more-over, the recent Maps update without GPS active even once afterwards, to do whatever it needed, meant I couldn't get a lock until back in Switzerland. Or, alternatively: while on a trip to Japan, it turns out that Google Maps caching is not supported there, probably because of rights over mapping data. So I had location, but only temporary cached maps (from the wifi in the hotels). Due to this kind of problems makes I consider Android maps a toy, so I wanted a GPS device with proper maps (on the device, not in the cloud), and with good GPS capabilities (no fancy A-GPS or anything, just plain good standalone GPS). Having had good experiences with the sport line of Garmin (but which doesn't have maps, so it's out of the question), I looked at Garmin. Many people complained about some things (same all over the place): The latter is something that jumped at me from many of the blogs/fora that I read; I like customisable power tools , not one-button mice. The other thing that I didn't really like was that most devices are very focused on a niche: the n vi line on automotive, the Zumo line on motorbikes, and the hand-held lines with some blurring of hiking/geocaching/etc. So no one-size fits all device Features! Customisation! That is, until I found out the Montana series. My my, I was blown. Both by the price (I couldn't believe how expensive it was, without maps!) and by the features. As it turns out, the Montana has different profiles , and it can act: That is for flexibility. As for performance/features: Now for the downsides. Being a multi-functional device, it's not as good as a dedicated one: But the small (for me) draw-backs aside, it's definitely not a dumbed-down device. It's the first device that I have which I feel it a bit too customisable . From completely changeable menus, to profiles that remember their maps selection, to many dashboards, to almost-programmable shortcuts (e.g. a shortcut to change profile, set destination to a given way point, reset the trip log, and turn track logging on) a bit overwhelming at first. Add the 3-axis compass, barometric altimeter, ANT+ support and you get a very powerful device. Software As to the software, yes it has some bugs - mainly that the device crashes sometimes (abruptly) if you play too much with the routes (didn't happen to me yet on the road , only when testing it at home). On the PC-side, Garmin's BaseCamp is nice, but doesn't hold a candle to Google Maps, for example, in terms of UI; I don't mean shininess, but simply how well the UI works. The most annoying thing is how it doesn't scale detail appropriately with the zoom level. But it works, and I can plan hikes offline, and upload both tracks and routes to it (this is important: it seems other recent Garmins can't replay tracks, only routes). And I can load many maps, and toggle them at will, etc. The many maps feature is not to be overlooked. I could have, for example, standard Garmin maps loaded, but just in case they're insuficiently detailed or out of date (hey, Navteq is slow ), I can also have Open Street Maps from two days ago. The routing is then done on the currently active maps, so you are not restricted to a single mapping provider. And of course you can also have custom POIs (Point of interest) on it. Bycicle use The bicycle mount is solid enough, so that on a rough roads it's held firmly. From my ~20 minute test this morning, a few things to note: Voice output By default, the unit can only beep, but if coupled via a 3.5" jack to headphones or via the automotive mount, it can properly speak; quality varies with voice, and whether it is TTS or not. I managed to find one voice (non-TTS) that sounds really well, but of course with less information, and one TTS voice that sounds OK-ish but gives full info. By full info , I mean the following: the behaviour of the various TTS voices is not consistent. For example, on a simulated route, various voices were saying the following: I don't know why these differences, or if they are consistent across routes. But I'll keep with voice E for now :) Summary So, sorry again for sounding like an ad (I don't have any relation with Garmin the company), but I find this device awesome - powerful, rugged, and working (so far) very well. I'm glad that there still exist companies who don't believe they know better than their customers, and instead offer you with a tool that you can make it work just like you want. Or almost ;-) Garmin's tag line: Big and Tough Goes anywhere, everywhere . I tend to agree :)

14 February 2013

Jon Dowland: Rpm, Yum, Puppet and GPG

I wrote that I would double-check how secure the module selection and downloading is in Puppet. Well, puppet module resolves, fetches and downloads unsigned tarballs from a HTTP source and unpacks them without any verification whatsoever. Related: I've been looking at rpm/yum GPG behaviour. rpm supports checking the signature of RPMs as a separate operation from installing them. You can't ask it to not install a package if the signature is absent or not correct. yum is better when dealing with repositories. It can be told to check the GPG signature on all RPMs both globally (the [main] section of yum.conf) and on a per-repository basis. GPG signature checking can be disabled on the command line with --nogpgcheck. It cannot be selectively enabled on the command line. However, yum install can install local RPMs and RPMs on web servers as well as from repositories. In both of these cases, it will not check the GPG signature at all, no matter what you've put in your yum.conf. Finally, even if all the above worked properly, the GPG keys published by Fedora have almost no public signatures (none at all for EPEL), so neither you nor I could establish a trust path to them. Luckily I can establish a trust path to the RH security key.

Next.

Previous.