Search Results: "spa"

3 June 2012

Jaldhar Vyas: Managua (doo dooo do doo-doo) Managua (doo do-do doo)

Mahna Mahna The writings been on the wall for a while but now it's official. I will not be able to attend DebConf this year. Or YAPC::NA either. I am still holding out hope of being able to go to Disneyland however.

Sandro Tosi: How to migrate your entire Google account to a new one

I've been using a Google account since years, but having a "more serious" one along. The time has come to make the switch and elect the other as main one.

Google has a lot of services, and I'm using several of them, so changing the main account requires to migrate data of those services to the new account, given it's not possible to merge two accounts. To know the full list of services your account is signed in, go to your Account Product page. Some migrations are easy, others hell no: so I'm writing this post to keep track of the migration as it's going on.

First of all, there's a really good post about this migration at LifeHacker: it contains a lot of info I'm using here, but not all the services I need. Additionally, Google has a list of services and methods to migrate from one account to another.

Just to be generic, I'll call AccountSRC the original account and AccountDST the one I want to migrate to.

Migrate GMail

GMail is probably the most important service I have, and also the most difficult one to migrate. I have a lot of filters, so the first step is to migrate them:
  1. Login into GMail for AccountSRC
  2. Gear icon > Settings > Labs
  3. Enable the "Filter import/export" plugin
  4. Reload GMail to enable the plugin (if not done automatically)
  5. Gear icon > Settings > Filters
  6. At the bottom of the page, Select all
  7. Export: this will download an XML file with the filters in it
  8. Login into GMail for AccountDST
  9. Enable "Filter import/export" plugin
  10. Load the file containing the saved filters
  11. Import all (or select which one) filters; this will also create automatically the labels defined in the filters.
I currently have a forward rule from AccountDST to AccountSRC: so let's revert the forward direction: from AccountSRC to AccountDST. This will allow the new account to receive the mails sent to the old one.
  1. Login into GMail for AccountDST
  2. Gear icon > Forwarding and POP/IMAP > Disable forwarding: this will stop redirecting mails from AccountDST to AccountSRC
  3. Login into GMail for AccountSRC
  4. Gear icon > Forwarding and POP/IMAP > add a forwarding address
  5. Enter AccountDST as the address to forward mails to
  6. A verification code is sent to AccountDST and you'll need to enter that code in AccountSRC to verify you have access to both mailboxes
  7. Once verified, select "Forward a copy of incoming mail to" to AccountDST and to delete the GMail copy on AccountSRC
There's no automatic way to migrate all your settings from the old GMail account to the new one, so you'll have to comparing settings pages to set what you had before to AccountDST. I suggest to first enable the labs you have on AccountSRC, and then go thru every page and report the configurations.

Did I forget something? yes, the hardest part: migrate mails! There are a lot of guides about migrating mails, the way I prefer is thru IMAP and Thunderbird (as described here): you'll move IMAP folders, and since they are equivalent to GMail labels, you'll automatically get your email with the correct labels (yes, the same mail appears in several folders, one for each label, and moving an email from one folder will not remove it from all the others).
  1. Enable IMAP access on both AccountSRC and AccountDST
  2. Open Thunderbird (or Icedove if you're on Debian like me) and register both accounts
  3. You can now start moving folders/emails from AccountSRC to AccountDST
    1. Copying a (missing) folder, will automatically create a label with the same name
    2. Moving a mail from the same folders on the two accounts will add the label called with the name of the folder to that email
  4. Be ready, it's a LOOOOONG process, but it's safe and it guarantees a perfect result
With the method above you have all the labels you had before, but not the label colors (and I have a lot :( ).

Another tool that's would be interesting to evaluate for this task is Gmvault.


Migrate Contacts

Contacts are available inside GMail:
  1. Login into GMail for AccountSRC
  2. On the combobox on top-left, select Contacts
  3. More > Export
  4. Export all the contacts in Google CSV, since it's specific to be re-imported
  5. Login into GMail for AccountDST
  6. Contacts > More > Import
  7. Select the previously saved file and import it

Migrate Reader

Migrating Reader subscriptions is easy:
  1. Login into Reader for AccountSRC
  2. Reader Settings -> Import/export
  3. Export your subscriptions in OPML format
  4. Now login into Reader for AccountDST
  5. Reader Settings -> Import/export
  6. And import the saved OPML file
Sadly, that doesn't migrate your starred items, but there's a (long & manual) solution here.


Migrate Blogger

Login to Blogger for AccountSRC; then, for each blog you have there do this dance:
  1. Select the blog
  2. Go into Settings > Permissions
  3. Add AccountDST as a new author: this will send an email to AccountDST GMail account
  4. Accept the invitation (this will ask you to create a new Blogger account for AccountDST if you don't already have one)
  5. Go back to to Blogger with AccountSRC
  6. Blog > Settings > Permissions
  7. Grand admin rights to AccountDST
At this point, both AccountSRC and AccountDST have admin rights; you can leave it as it is, or remove AccountSRC and so only AccountDST will be the admin (other shares should be kept unchanged). The same procedure is described in this Google support answer.

What can't be migrated as easily are all the profile settings you've done on AccountSRC. So profile information, blogs followed and so on have to be migrated by hand: boring but they are just few things, doable in a bunch of minutes.


Migrate Android phone

Android phones require a primary Google account to work, and of course my phone uses AccountSRC. Several sources say the only way to change primary account is to factory reset the phone. But since new market app supports multiple accounts and Android 4.0 (Ice Cream Sandwich) not longer needs a primary account, I just waited for the update to ICS to come.

What you really need to do is to add a new account, AccountDST, to Settings > Account & Sync, and all the Google apps on the phone will be able to use the new account (you might need to select what to sync, and the new account from inside each app).


Migrate Calendar

Even with Calendar the settings will have to be migrated by hand: they are just in one page, so just keep two windows opened and keep switching to sync them. The calendars are easy to move over: for the main calendar:
  1. Login into Calendars for AccountSRC
  2. Go into Settings > Calendars tab
  3. Select export calendars: this will download an archive of your calendars in a zip file; you will need to extract all the ics files from that zip archive so you'll have the main calendar readily available
  4. Login into Calendars for AccountDST
  5. Settings > Calendars tab
  6. Import the main calendar
for the other calendars you can do this way:
  1. Login into Calendars for AccountSRC
  2. Settings > Calendars tab
  3. Share the calendar with AccountDST and give it "Make changes AND manage sharing" right - the new account will have full control over that calendar

Migrate Docs

The procedure to migrate documents in Google Docs (now Google Drive) is described here, and it boils down to:
  1. Login into Docs for AccountSRC
  2. Select all the documents
  3. Share them with AccountDST
  4. In the same window select AccountDST as "is owner"
and you've got all the docs as owned by the new account.


Migrate Groups

There's no way to migrate all your Groups subscriptions from one account to another (if not playing with alternate email address, but it was too risky to try it), so you'll have to migrate all of them by hand.

On the other hand, if you've created some groups, that's possible to move them to the new account:
  1. Login into Groups for AccountSRC
  2. Enter in the "manage" area for the groups you've created
  3. Manage members > Invite members
  4. Invite AccountDST
  5. Accept the invitation, clicking on the email sent to AccountDST account
  6. Go back into AccountSRC manage area
  7. Select the AccountDST account
  8. Edit the subscription and set "Membership type" to "owner": now AccountDST is the owner of the list

Migrate Analytics

The process to migrate the account on Google Analytics is similar to Groups:
  1. Login into Analytics for AccountSRC
  2. Select Admin on top-right, then for each account you have
  3. Select the account, select "Users" tab, click on "+ New User"
  4. Enter AccountDST into the "Email Address" field and give that account the "Administrator" role
  5. Save
  6. Now AccountDST is able to access the same data on Analytics as AccountSRC

Migrate Picasa

To transfer your whole Picasa account follow these steps (as also described in this support post):
  1. Login into Picasa for AccountSRC
  2. Click on the gear on top-right > Photo Settings > Privacy and permissions
  3. On the "Migrate account" line click on "Migrate my photos"
  4. Insert AccountDST into the box for destination address
  5. In case you're using picasa to store blogger photos, also select the tick below
  6. At this point, AccountSRC will receive an email: copy the link in a session where you've logged in as AccountDST and then visit the page: that will start the process
  7. Wait until the transfer is completed: you'll receive an email to AccountDST when it's done

Migrate Google+

I'm not exactly a heavy user of G+ but I have some circles I'd like to move over (also because Reader sharing now works with G+ and it's a feature I user a lot), this is a process to move circles and contacts; as stated there, there's no way to move shared items and stuff (sad!).


Migrate AdSense

It's not possible to migrate the current AdSense account: you have to register a new one. When you do, it's possible that Google notice the new account is for the same person of the first one, and will ask to close the old account on AdSense in favor of the new one.


Migrate Alert

Alerts allows to export the list of alerts you've created but it's not possible to import that list in another Google account, so you'll have to recreate them from scratch (I had just a few so it was a matter of 1 minute or so).


Migrate Zoho Notebook

If you still miss Google Notebook (why big-G you let it die? couldn't you open-sourced it? it was so awesome! anyway...), you probably know that there are two main alternatives to it: Evernote and Zoho Notebook. I decided to go with Zoho (I'm not entirely happy with it, but it's running for now), and given it allows to login with Google account, it was currently using AccountSRC, so I need to migrate that too:
  1. Login into Zoho Notebook for AccountSRC
  2. My Account > Profile > Email address
  3. "Add new email" and add AccountDST, it will send a confirmation email
  4. Then "Make Primary" on AccountDST, so now the new account can login into Zoho notebook with the Google info and have all the notes there.

Migrate Bookmarks

Google provides a guide to migrate Bookmarks from one account to another that basically is:
  1. Login into Bookmarks for AccountSRC
  2. Click on Export bookmarks and it will save an HTML file with the bookmarks in it
  3. Import the file in a browser
  4. Use the Google toolbar to import in into AccountDST
That's kinda ugly way, but it seems to work.


Re-enable Web History

If you used History, then you need to re-enable it on the new account:
  1. Login into History for AccountDST
  2. Click on "Turn Web History on" to enable Web History on the new account
but it seems to be impossible to export/import or migrate the history from one account to another, that's a shame: years and years of stats lost.


Migrate Books

Books doesn't allow to migrate the contents between accounts, so you'll have to do it by hand:
  1. Select one book from AccountSRC, copying its link
  2. Go to AccountDST and navigate to the copied link
  3. Add the book to the right category, as it was before.

Migrate Webmaster Tools

Google released a document that describes how to migrate to another account; it's similar to Analytics and Groups migration.

In addition to that, you might also want to become an owner, that will give full control on the sites.


Did I forget some services? Let me know!

Everyone has a different experience with Google products, so in case you want me to let know how you migrated from one account to another for a product I didn't mentioned, just leave a comment and I'll edit the post (to be as complete as possible).

2 June 2012

Petter Reinholdtsen: Debian Edu interview: Mike Gabriel

Back in 2010, Mike Gabriel showed up on the Debian Edu and Skolelinux mailing list. He quickly proved to be a valuable developer, and thanks to his tireless effort we now have Kerberos integrated into the Debian Edu Squeeze version. Who are you, and how do you spend your days? My name is Mike Gabriel, I am 38 years old and live near Kiel, Schleswig-Holstein, Germany. I live together with a wonderful partner (Angela Fu ) and two own children and two bonus children (contributed by Angela). During the day I am part-time employed as a system administrator and part-time working as an IT consultant. The consultancy work touches free software topics wherever and whenever possible. During the nights I am a free software developer. In the gaps I also train in becoming an osteopath. Starting in 2010 we (Andreas Buchholz, Angela Fu , Mike Gabriel) have set up a free software project in the area of Kiel that aims at introducing free software into schools. The project's name is "IT-Zukunft Schule" (IT future for schools). The project links IT skills with communication skills. How did you get in contact with the Skolelinux/Debian Edu project? While preparing our own customised Linux distribution for "IT-Zukunft Schule" we were repeatedly asked if we really wanted to reinvent the wheel. What schools really need is already available, people said. From this impulse we started evaluating other Linux distributions that target being used for school networks. At the end we short-listed two approaches and compared them: a commercial Linux distribution developed by a company in Bremen, Germany, and Skolelinux / Debian Edu. Between 12/2010 and 03/2011 we went to several events and met people being responsible for marketing and development of either of the distributions. Skolelinux / Debian Edu was by far much more convincing compared to the other product that got short-listed beforehand--across the full spectrum. What was most attractive for me personally: the perspective of collaboration within the developmental branch of the Debian Edu project itself. In parallel with this, we talked to many local and not-so-local people. People teaching at schools, headmasters, politicians, data protection experts, other IT professionals. We came to two conclusions: First, a technical conclusion: What schools need is available in bits and pieces here and there, and none of the solutions really fit by 100%. Any school we have seen has a very individual IT setup whereas most of each school's requirements could mapped by a standard IT solution. The requirement to this IT solution is flexibility and customisability, so that individual adaptations here and there are possible. In terms of re-distributing and rolling out such a standardised IT system for schools (a system that is still to some degree customisable) there is still a lot of work to do here locally. Debian Edu / Skolelinux has been our choice as the starting point. Second, a holistic conclusion: What schools need does not exist at all (or we missed it so far). There are several technical solutions for handling IT at schools that tend to make a good impression. What has been missing completely here in Germany, though, is the enrolment of people into using IT and teaching with IT. "IT-Zukunft Schule" tries to provide an approach for this. Only some schools have some sort of a media concept which explains, defines and gives guidance on how to use IT in class. Most schools in Northern Germany do not have an IT service provider, the school's IT equipment is managed by one or (if the school is lucky) two (admin) teachers, most of the workload these admin teachers get done in there spare time. We were surprised that only a very few admin teachers were networked with colleagues from other schools. Basically, every school here around has its individual approach of providing IT equipment to teachers and students and the exchange of ideas has been quasi non-existent until 2010/2011. Quite some (non-admin) teachers try to avoid using IT technology in class as a learning medium completely. Several reasons for this avoidance do exist. We discovered that no-one has ever taken a closer look at this social part of IT management in schools, so far. On our quest journey for a technical IT solution for schools, we discussed this issue with several teachers, headmasters, politicians, other IT professionals and they all confirmed: a holistic approach of considering IT management at schools, an approach that includes the people in place, will be new and probably a gain for all. What do you see as the advantages of Skolelinux/Debian Edu? There is a list of advantages: international context, openness to any kind of contributions, do-ocracy policy, the closeness to Debian, the different installation scenarios possible (from stand-alone workstation to complex multi-server sites), the transparency within project communication, honest communication within the group of developers, etc. What do you see as the disadvantages of Skolelinux/Debian Edu? Every coin has two sides: Technically: BTS issue #311188, tricky upgradability of a Debian Edu main server, network client installations on top of a plain vanilla Debian installation should become possible sometime in the near future, one could think about splitting the very complex package debian-edu-config into several portions (to make it easier for new developers to contribute). Another issue I see is that we (as Debian Edu developers) should find out more about the network of people who do the marketing for Debian Edu / Skolelinux. There is a very active group in Germany promoting Skolelinux on the bigger Linux Days within Germany. Are there other groups like that in other countries? How can we bring these marketing people together (marketing group A with group B and all of them with the group of Debian Edu developers)? During the last meeting of the German Skolelinux group, I got the impression of people there being rather disconnected from the development department of Debian Edu / Skolelinux. Which free software do you use daily? For my daily business, I do not use commercial software at all. For normal stuff I use Iceweasel/Firefox, Libreoffice.org. For serious text writing I prefer LaTeX. I use gimp, inkscape, scribus for more artistic tasks. I run virtual machines in KVM and Virtualbox. I am one of the upstream developers of X2Go. In 2010 I started the development of a Python based X2Go Client, called PyHoca-GUI. PyHoca-GUI has brought forth a Python X2Go Client API that currently is being integrated in Ubuntu's software center. For communications I have my own Kolab server running using Horde as web-based groupware client. For IRC I love to use irssi, for Jabber I have several clients that I use, mostly pidgin, though. I am also the Debian maintainer of Coccinella, a Jabber-based interactive whiteboard. My favourite terminal emulator is KDE's Yakuake. Which strategy do you believe is the right one to use to get schools to use free software? Communicate, communicate, communicate. Enrol people, enrol people, enrol people.

1 June 2012

Pietro Abate: apt-get with external solvers : call for testers

Last year we invited David to work with us for a few days to add a generic interface to apt to call external solvers. After a few iterations, this patch finally landed in master and recently (about 3 months ago), in debian unstable.
  [ David Kalnischkies ]
  * [ABI-Break] Implement EDSP in libapt-pkg so that all front-ends which
    use the internal resolver can now be used also with external
    ones as the usage is hidden in between the old API
  * provide two edsp solvers in apt-utils:
    - 'dump' to quickly output a complete scenario and
    - 'apt' to use the internal as an external resolver
Today the new version of apt-cudf was upload in unstable and with it the latest bug fixes that makes it ready for daily use. I've used it quite a lot myself to upgrade my machine and it seems working pretty well so far... The most important difference with the old version is the support for multi-arch enabled machines. This marks an important milestone in our efforts to integrate external solvers, built using different technologies, directly into apt. From a user prospective, this means that (s)he will have the possibility to check if there exists a better (best ?) solution to an installation problem then what proposed by the internal apt solver. Moreover, even if apt-get gives very satisfactory answers, there are occasions where it fails miserably, leaving the user wondering how to unravel the complex web of dependencies to accomplish his task. Available cudf solvers in debian are at the moment : aspcud, mccs and packup. From an architectural point of view this is accomplished by abstracting the installation problem via a simple textual protocol (EDSP) and using an external tool to do the heavy duty translation. Since all solvers now available in debian are not meant to be debian-specific, using them involve a two step translation. The EDSP protocol specification is for the moment "hidden" in the apt documentation. I hope to find a better place for it soon : it would be cool if other package managers as smart or cupt could add an implementation on EDSP in their code so to automatically benefit of this technology. To make it happen, Apt first creates an EDSP file that is then handled to apt-cudf that takes care of the translation to cudf and back into EDSP that is then read back by apt. Apt-cudf is the bridge between edsp and the external solvers and takes care of doing the book keeping and to select the right optimization criteria. Roberto recently wrote a very nice article explaining how to use apt-get with an external solver. In a nutshell, if you want to try this out you just need to install apt-cudf, one external solver like aspcud from the university of Pasdam and then call apt-get using the --solver option (that is not yet documented #67442 ). For example :
apt-get install -s gnome --solver aspcud
This will install gnome while using the a optimization criteria that tries to minimizing the changes on the system. Various other optimization criteria for all apt-get default actions can be specified in the apt-cudf configuration file /etc/apt-cudf.conf I hope the new release of apt-cudf make it into testing before the freeze. Time to test !!!

Raphaël Hertzog: My Debian Activities in May 2012

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (338.26 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg Like last month, I did almost nothing concerning dpkg. This will probably change in June now that the book is out The only thing worth noting is that I have helped Carey Underwood who was trying to diagnose why btrfs was performing so badly when unpacking Debian packages (compared to ext4). Apparently this already resulted in some btrfs improvements. But not as much as what could be hoped. The sync_file_range() calls that dpkg are doing only force the writeback of the underlying data and not of the meta-data. So the numerous fsync() that follow still create many journal transactions that would be better handled as one big transaction. As a proof of this, replacing the fsync() with a sync() brings the performance on par with ext4. (Beware this is my own recollection of the discussion, while it should be close to the truth, it s probably not 100% accurate when speaking of the brtfs behaviour) Packaging I uploaded new versions of smarty-gettext and smarty-validate because they were uninstallable after the removal of smarty. The whole history of smarty in Debian/Ubuntu has been a big FAIL since the start. Once upon a time, there was a smarty package and some plugins. Everything was great except that the files were installed in a way that differs from the upstream recommendations. So Ubuntu changed the path in their version of the package and did not check whether it broke anything else (and it did break all the plugins). Despite the brokenness of the plugins, this divergence survived for years. So several packages that were using Smarty were modified to use dpkg-vendor to use the correct path depending on whether it was built on Debian or Ubuntu. In 2010, Smarty 3.0 has been released and instead of upgrading the smarty package to this version, one of the smarty co-maintainers introduced a smarty3 package that used yet another path (despite the fact that smarty 3 had a mode to be compatible with smarty 2).
At some point, I informed him that he had to handle the migration of users of smarty to smarty3 he acknowledged and then lost interest in smarty ( I m no longer using it ) and did nothing. After some more bitrot, smarty has been forcefully orphaned in August 2011 by a member of the security team. And in March this year, it has been removed from unstable despite the fact that it still had reverse dependencies (usually removals only happen when they impact no other packages, I don t know why this wasn t the case here). At least the brokenness attracted some attention to the situation and Mike Gabriel contacted me about it. I offered him to take over the various packages since they all needed a real maintainer and he accepted. I sponsored his uploads of all smarty related packages (bringing in the latest upstream versions at the same time). In the end, the situation is looking better now, except that there s no migration path from users who rely on smarty in Squeeze. They will discover that they need smarty3 in Wheezy and that the various paths have to be adjusted. It s probably acceptable since the new upstream versions are no longer backwards compatible with smarty 2 The Debian Administrator s Handbook At the start of the month, I was busy preparing the release of the book. I introduced the publican-debian package to unstable, it s a Publican brand (aka a set of CSS and XSL stylesheets to tailor the output of Publican) using the Debian colors and using the Debian logo. This brand is used by the book. I also created the debian-handbook package and setup the public Git repository on alioth.debian.org. I was ready or so I thought. A few hours after the announce, the website became unusable because the numerous visitors were exhausting the maximum number of client connections. And I could not increase the limit due to Apache s memory usage (with PHP and WordPress). We quickly off-loaded most of the static files traffic to another machine and we setup bittorrent. The problem was solved for the short term. Thousands of persons downloaded the ebook and to this date, 135 copies of the paperback have been sold. Then I took a one-week vacation. Even though I had no Internet at the place I was, I wandered in the street to find a Freewifi wifi network (customers of the Free ISP can use those freely) to stay on top of incoming email. We quickly received some bug reports and I dealt with the easy ones (typos and the like) on the fly. When I came back at home, I manually placed 54 lulu orders for the people who opted for the paperback as reward during the fundraising campaign. A bit tedious but it had to be done (if only Lulu supported a way to batch many orders at once ). I also wanted a long term solution to avoid the use of an external host to serve static files (should a new traffic spike arrive ). So I installed nginx as a front-end. It serves static files directly, as well as WordPress pages which have been cached by wp-super-cache. Apache is still here listening on a local port and responding to the remaining queries forwarded by nginx. Once I ll migrate to wheezy, I might completely ditch apache in favor of php5-fpm to handle the PHP pages. Last but not least, I wanted to bootstrap the various translations that people offered to contribute. I wrote some documentation for interested translators and blogged about it. It s shaping up nicely check it out if you re interested to help! Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

31 May 2012

Maximilian Attems: New klibc release 2.0

The new klibc features prominently the merged stdio branch (causing the version bump). The buffered I/0 allows a faster fstype and thus faster boot. Plus klibc gained the support of several stream I/O functions, for details see the longer story on: klibc 2.0 release announcement or klibc git repository. The dhcp cmd ipconfig can now generate enough info for a proper lease file too and got fixed concerning the min length of it's replies. Google sponsored the /etc/fstab or cmdline boot mounting in kinit. 2.0 is shipping several arch fixes including i386 signal fixes, m68 support, s390x, sh4 compile fixes. There is still portability work todo: #439181 ia64 shared lib buggy (longdated unresolved ia64 mystery), #634890 amrhf all binaries segfault (maybe signal related), status on parisc is unknown, avr32 patch is untested (might as well just merge) and sh4 SIGILL. We got nice arch testcoverage thanks to mksh testsuite and working mksh klibc support. We are confident of this release on all this archs: alpha, armel, m68k, mips, ppc, ppc64, sparc, s390, s390x and x86. (:

Matthew Garrett: Implementing UEFI Secure Boot in Fedora

Edit 16:17EDT 31/5/12: Clarification of who gets the $99
Edit 02:10EDT 01/6/12: Clarification that it's a one-off payment

(Brief disclaimer - while I work for Red Hat, I'm only going to be talking about Fedora here. Anything written below represents only my opinions and my work on Fedora, not Red Hat's opinions or future plans)

Fedora 17 was released this week. It's both useful and free, and serves as a welcome addition to any family gathering. Do give it a go. But it's also noteworthy for another reason - it's the last Fedora release in the pre-UEFI secure boot era. Fedora 18 will be released at around the same time as Windows 8, and as previously discussed all Windows 8 hardware will be shipping with secure boot enabled by default. While Microsoft have modified their original position and all x86 Windows machines will be required to have a firmware option to disable this or to permit users to enrol their own keys, it's not really an option to force all our users to play with hard to find firmware settings before they can run Fedora. We've been working on a plan for dealing with this. It's not ideal, but of all the approaches we've examined we feel that this one offers the best balance between letting users install Fedora while still permitting user freedom.

Getting the machine booted
Most hardware you'll be able to buy towards the end of the year will be Windows 8 certified. That means that it'll be carrying a set of secure boot keys, and if it comes with Windows 8 pre-installed then secure boot will be enabled by default. This set of keys isn't absolutely fixed and will probably vary between manufacturers, but anything with a Windows logo will carry the Microsoft key[1].

We explored the possibility of producing a Fedora key and encouraging hardware vendors to incorporate it, but turned it down for a couple of reasons. First, while we had a surprisingly positive response from the vendors, there was no realistic chance that we could get all of them to carry it. That would mean going back to the bad old days of scouring compatibility lists before buying hardware, and that's fundamentally user-hostile. Secondly, it would put Fedora in a privileged position. As one of the larger distributions, we have more opportunity to talk to hardware manufacturers than most distributions do. Systems with a Fedora key would boot Fedora fine, but would they boot Mandriva? Arch? Mint? Mepis? Adopting a distribution-specific key and encouraging hardware companies to adopt it would have been hostile to other distributions. We want to compete on merit, not because we have better links to OEMs.

An alternative was producing some sort of overall Linux key. It turns out that this is also difficult, since it would mean finding an entity who was willing to take responsibility for managing signing or key distribution. That means having the ability to keep the root key absolutely secure and perform adequate validation of people asking for signing. That's expensive. Like millions of dollars expensive. It would also take a lot of time to set up, and that's not really time we had. And, finally, nobody was jumping at the opportunity to volunteer. So no generic Linux key.

The last option wasn't hugely attractive, but is probably the least worst. Microsoft will be offering signing services through their sysdev portal. It's not entirely free (there's a one-off $99 fee to gain access edit: The $99 goes to Verisign, not Microsoft - further edit: once paid you can sign as many binaries as you want), but it's cheaper than any realistic alternative would have been. It ensures compatibility with as wide a range of hardware as possible and it avoids Fedora having any special privileges over other Linux distributions. If there are better options then we haven't found them. So, in all probability, this is the approach we'll take. Our first stage bootloader will be signed with a Microsoft key.

Bootloaders
We've decided to take a multi-layer approach to our signing for a fairly simple reason. Signing through the Microsoft signing service is a manual process, and that's a pain. We don't want to have bootloader updates delayed because someone needs to find a copy of Internet Explorer and a smartcard and build packages by hand. Instead we're writing a very simple bootloader[2]. This will do nothing other than load a real bootloader (grub 2), validate that it's signed with a Fedora signing key and then execute it. Using the Fedora signing key there means that we can build grub updates in our existing build infrastructure and sign them ourselves. The first stage bootloader should change very rarely, and we don't envisage updating it more than once per release cycle. It shouldn't be much of a burden on release management.

What about grub? We've already switched Fedora 18 over to using grub 2 by default on EFI systems, but it still needs some work before it's ready for secure boot. The first thing is that we'll be disabling the module loading. Right now you can load arbitrary code into grub 2 at runtime, and that defeats the point of secure boot. So that'll be disabled. Next we'll be adding support for verifying that the kernel it's about to boot is signed with a trusted key. And finally we'll be sanitising the kernel command line to avoid certain bits of functionality that would permit an attacker to cause even a signed kernel to launch arbitrary code. These restrictions will all vanish if secure boot is disabled.

Kernel
Secure boot is built on the idea that all code that can touch the hardware directly is trusted, and any untrusted code must go through the trusted code. This can be circumvented if users can execute arbitrary code in the kernel. So, we'll be moving to requiring signed kernel modules and locking down certain aspects of kernel functionality. The most obvious example is that it won't be possible to access PCI regions directly from userspace, which means all graphics cards will need kernel drivers. Userspace modesetting will be a thing of the past. Again, disabling secure boot will disable these restrictions.

Signed modules are obviously troubling from a user perspective. We'll be signing all the drivers that we ship, but what about out of tree drivers? We don't have a good answer for that yet. As before, we don't want any kind of solution that works for us but doesn't work for other distributions. Fedora-only or Ubuntu-only drivers are the last thing anyone wants, and this really needs to be handled in a cross-distribution way.

Wait signed what
Secure boot is designed to protect against malware code running before the operating system. This isn't a hypothetical threat. Pre-boot malware exists in the wild, and some of it is nastier than you expect. So obviously bootloaders need to be signed, since otherwise you'd just replace the signed bootloader with an unsigned one that installed malware and booted your OS.

But what about the kernel? The kernel is just code. If I take a signed Linux bootloader and then use it to boot something that looks like an unsigned Linux kernel, I've instead potentially just booted a piece of malware. And if that malware can attack Windows then the signed Linux bootloader is no longer just a signed Linux bootloader, it's a signed Windows malware launcher and that's the kind of thing that results in that bootloader being added to the list of blacklisted binaries and suddenly your signed Linux bootloader isn't even a signed Linux bootloader. So kernels need to be signed.

And modules? Again, modules are just code. It's a little trickier, but if your signed kernel loads an unsigned module then that unsigned module can set up a fake UEFI environment and chain into a compromised OS bootloader. Now the attacker just has to include a signed kernel and a minimal initramfs that loads their malware module. It'd slow down boot by a couple of seconds, but other than that it'd be undetectable. X? If you can access the registers on a GPU then you can get the GPU to DMA over the kernel and execute arbitrary code. Trickier again, but still achievable - and if you've locked down every other avenue of attack, it's even attractive.

If we produce signed code that can be used to attack other operating systems then those other operating systems are justified in blacklisting us. That doesn't seem like a good outcome.

Customisation
A lot of our users want to build their own kernels. Some even want to build their own distributions. Signing our bootloader and kernel is an impediment to that. We'll be providing all the tools we use for signing our binaries, but for obvious reasons we can't hand out our keys. There's three approaches here. The first is for a user to generate their own key and enrol it in their system firmware. We'll trust anything that's signed with a key that's present in the firmware. The second is to rebuild the shim loader with their own key installed and then pay $99 and sign that with Microsoft. That means that they'll be able to give copies to anyone else and let them install it without any fiddling. The third is to just disable secure boot entirely, at which point the machine should return to granting the same set of freedoms as it currently does.

But I don't trust Microsoft
A system in custom mode should allow you to delete all existing keys and replace them with your own. After that it's just a matter of re-signing the Fedora bootloader (like I said, we'll be providing tools and documentation for that) and you'll have a computer that will boot Fedora but which will refuse to boot any Microsoft code. It may be a little more awkward for desktops because you may have to handle the Microsoft-signed UEFI drivers on your graphics and network cards, but this is also solvable. I'm looking at ways to implement a tool to allow you to automatically whitelist the installed drivers. Barring firmware backdoors, it's possible to configure secure boot such that your computer will only run software you trust. Freedom means being allowed to run the software you want to run, but it also means being able to choose the software you don't want to run.

You've sold out
We've been working on this for months. This isn't an attractive solution, but it is a workable one. We came to the conclusion that every other approach was unworkable. The cause of free software isn't furthered by making it difficult or impossible for unskilled users to run Linux, and while this approach does have its downsides it does also avoid us ending up where we were in the 90s. Users will retain the freedom to run modified software and we wouldn't have accepted any solution that made that impossible.

But is this a compromise? Of course. There's already inequalities between Fedora and users - trademarks prevent the distribution of the Fedora artwork with modified distributions, and much of the Fedora infrastructure is licensed such that some people have more power than others. This adds to that inequality. It's not the ideal outcome for anyone, and I'm genuinely sorry that we weren't able to come up with a solution that was better. This isn't as bad as I feared it would be, but nor is it as good as I hoped it would be.

What about ARM
Microsoft's certification requirements for ARM machines forbid vendors from offering the ability to disable secure boot or enrol user keys. While we could support secure boot in the same way as we plan to on x86, it would prevent users from running modified software unless they paid money for a signing key. We don't find that acceptable and so have no plans to support it.

Thankfully this shouldn't be anywhere near as much of a problem as it would be in the x86 world. Microsoft have far less influence over the ARM market, and the only machines affected by this will be the ones explicitly designed to support Windows. If you want to run Linux on ARM then there'll be no shortage of hardware available to you.

Is this all set in stone?
No. We've spent some time thinking about all of this and are happy that we can implement it in the Fedora 18 timescale, but there's always the possibility that we've missed something or that a new idea will come up. If we can increase user freedom without making awful compromises somewhere else then we'll do it.

[1] In fact, chances are that everything will carry the Microsoft key. Secure boot requires that UEFI drivers also be signed. The signing format only permits a single signature per binary. For compatibility, approximately all add-on hardware shipped will be signed with Microsoft's key, and that means that all system vendors have to recognise Microsoft's key in order to permit that hardware to run on their systems.
[2] Current source is here. It relies on a port of the UEFI crypto library and OpenSSL that I have building with some handholding, and which I'll upload as soon as possible.

comment count unavailable comments

30 May 2012

Jaldhar Vyas: Dovecot 2.1.7 Uploaded to Unstable

This is the latest upstream version and will, barring any unforseen circumstances, be the version in wheezy, the upcoming stable release of Debian GNU/Linux. In order to ensure it is the high quality our users have come to expect from Debian stable, I need your help. First and foremost, please test the hell out of it. Upgrades during a stable release are in general not allowed unless the problem is security related so we need to shake all the bugs out now. I am particularly interested in knowing if there are any issues with upgrades from squeeze. I waved my hands ineffectually at the bug list but it needs more attention. Could some kind person go through them and let me know which ones are no longer valid, need further attention etc.? I attempted to add hardening flags but lintian still complains a lot. I think the warnings are probably false positives but I'm not a 100% sure so I would appreciate some advice from those who know more about such things. Those are the issues that come to mind at the moment but feel free to let me know if there's something I've missed.

Lior Kaplan: Debian packaging for beginners @ FOSDEM the video

I recently got a question by email about the lecture I presented at FOSDEM, and by that mail I discovered that the video of the lecture was uploaded to YouTube (FOSDEM 2012 Debian packaging for beginners). While I thank the person who did the upload, he/she did not keep the right license for the video which is CC-BY-SA 3.0 (as noted at FOSDEM archive). You can also grab the webm file directly. Well, the next task will be to listen to myself in order to improve for FOSDEM 2013 See you there.
Filed under: Debian GNU/Linux, FOSDEM

Sorina Sandu: Initial commit

Hello!

I'm Sorina and this summer I will be working on Debian Installer network setup during Google Summer of Code. I am a second year bachelor's student at Politehnica University of Bucharest, I am studying Computer Science. I very much enjoy what I am doing even if sometimes I feel that the amount of stuff I want/have to learn/do/work on is infinite and I don't really know where to start.

I am at the beginning of the project, I believe now I am accustomed to the way things work and I am waiting forward to finishing the exam period so that I could really start working.

The hardest part for me in what I've done so far is testing what I am doing. It took me a while to figure out at least a part of how the installer functions and to solve the problems that occurred while building. Luckily, there is documentation and internet and even more important, I had help and support from my mentors.

So far I have only written two patches for netcfg, for bugs #610752 and #636211. For the first one I had a version for a while now, but I was unable to test it (unfortunately, it's likely that you will hear this a lot in the future). The idea was to make the link detection timeout preseedable and for this I have added two new templates, asking the user to provide a value for that matter.

The second one involved a missing "Go Back" button. This seemed to be the result of not letting debconf know that the program is capable of handling the user pressing a back button. That could be solved using deb_capb.

As I said before, testing was my main problem. Eventually, I built the image, wrote it to an USB and rebooted my laptop. This served its purpose, since I had access to anything I needed such as the wireless interface and physical access to the network cable. Running it through a virtual machine did not help much in this case, because I was unable to provide those.

Another thing I considered to be hard was blog posting, but now that I have just finished my first post it seems that I overreacted.

That's all for now, have a nice day!

Michal Čihař: Four counferences in Prague this October

I'm participating on organizing this year's openSUSE conference and as we've finally finalized place and time and CFP is open, it's time to share some information. This will be special - there is not a single conference, but four of them are sharing same space and the motto - Bootstrapping awesome!!!. The 4th openSUSE conference will be held together with LinuxDays, Czech conference following tradition of canceled LinuxExpo, what should be the biggest free software event in Czech republic. In addition there will be also 12th SUSE Labs conference (so you can meet quite a lot of kernel hackers and other strange guys) and first Gentoo mini summit (the website is empty so far). All that will happen on weekend from 20th October to 21st October, the SUSE conferences will then continue for 22nd and 23rd October. I believe it will be great mixture of conferences and I hope to meet lot of people there.

Filed under: English Gentoo Suse 0 comments Flattr this!

Christian Perrier: 2012 update 24 for Debian Installer localization

Status for D-I level 1 (core D-I files): Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.): Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader) Full 100% completeness (hall of fame) for 26 languages: Asturian, Belarusian, Bulgarian, Catalan, Czech, Danish, German, Esperanto, Spanish, Persian, French, Hebrew, Indonesian, Italian, Japanese, Kazakh, Khmer, Dutch, Polish, Portuguese, Russian, Slovak, Slovenian, Swedish, Turkish, Simplified Chinese

29 May 2012

Niels Thykier: Lintian 2.5.7 and 2.5.8

The new version of Lintian (2.5.8) can pretty much be summed up as:
Its like 2.5.7, only with less false positives and no FTBFS.
Especially people annoyed by the hardening flags will hopefully find that 2.5.8 greatly reduces the number of false positives. I believe this is best demonstrated with an example:
$ lintian --print-version && lintian -q -C binaries amarok_2.5.0-1_i386.deb   wc -l 
2.5.7
94
[...]
$ lintian --print-version && lintian -q -C binaries amarok_2.5.0-1_i386.deb   wc -l
2.5.8
4
However, nothing comes for free. We dropped hardening-no-stackprotector from the default profile (and demoted it to an I tag). For hardening-no-fortify-functions we made a false positive -> false negative trade-off by ignoring binaries if their only unprotected function is memcpy. For more information, please refer to #673112. hardening-no-stackprotector is still available and can be used via the debian/extra-hardening profile (or via the tags argument). For the 2.5.7 behaviour of hardening-no-fortify-functions, you have to use hardening-check directly. But 2.5.7 had other changes besides the myriad of false-positives: A consequence of this is that we no longer warn if you use things like dpkg assert-working-epoch or your postinst creates a usr/doc transition symlink . With all the Lintian modules under the Lintian namespace, we can install them in the standard perl @INC path. Admittedly, I am not certain we are ready to commit to the current API in these modules, which is one of the reasons why they are still installed in /usr/share/lintian. But in a couple of releases things may look differently. :) Lintain 2.5.7 ships over 50 data files of various kinds (usually white- or blacklists of some kind). Before 2.5.7, all data files with a few exceptions would be loaded eagerly (i.e. as soon as the check was run for the first time). A few data files had been special cased with manual laziness . In 2.5.7, Lintian::Data was updated to lazily load the data. So depending on the packages being checked, Lintian may now load fewer data files. While this is not something exclusive to 2.5.7, we have separated quite a few checks from their data tables by now. My personal favorites are the table of known interpreters and (from 2.5.8) the table of known versioned interpreters. I am certainly biased here. But this is probably the most awesome feature in Lintian 2.5.7. It is now possible for vendors to extend or simply shadow the core Lintian data files. Allow me to demonstrate how this can be used:
$ lintian --tags build-depends-on-obsolete-package a2ps_4.14-1.1.dsc 
E: a2ps source: build-depends-on-obsolete-package build-depends: dpatch
$ cat ~/.lintian/profiles/local/main.profile 
Profile: local/main
Extends: debian/main
$ cat  ~/.lintian/vendors/local/main/data/fields/obsolete-packages 
@include-parent
@delete dpatch
bison
$ lintian --profile local/main --tags build-depends-on-obsolete-package a2ps_4.14-1.1.dsc
E: a2ps source: build-depends-on-obsolete-package build-depends: bison
$
It is a toy example, but I believe it is a good demonstration of the feature. The full documentation of Vendor specific data files can be found in the Lintian User manual (in lintian/2.5.7 or newer). It will also be on lintian.debian.org when we find time to update lintian there. :)

Martin-Éric Racine: OpenWRT: accepting Router Adverts from the ISP in White Russian?

As we are getting closer to the World IPv6 Day, I thought that I'd see if I can upgrade my WRT54GL's OpenWRT (White Russian) installation to accept Router Adverts from the ISP to acquire an IPv6 address (rather than via a 4-to-6 tunnel broker, which seems to be the most documented case), and to propagate its IPv6 address space to my local network via my WRT45GL's radvd. Unfortunately, most of the instructions out there concern newer OpenWRT releases to which I cannot upgrade to because their software base takes up significantly more space than White Russian does. Still, I figure that someone in the community must have already succeeded at configuring their WRT54GL for this and might be able to help. Anyone? :)

28 May 2012

Vincent Bernat: Integration of NetSNMP into an event loop

NetSNMP comes with its own event loop based on the select() system call. While you can build your program around it, you may prefer to use an existing event loop, either a custom one or something like libevent, libev or Twisted.

Own custom event loop Let s start with the easiest case: you have written your own event loop. This means you make a call to select(), poll(), epoll(), kqueue() or something similar. All those functions take a set of file descriptors and wait for any of them to become available in a specified time frame. Here is a typical use of select() (adapted from Quagga)
readfd = m->readfd;
writefd = m->writefd;
exceptfd = m->exceptfd;
timer_wait = thread_timer_wait (&m->timer);
num = select (FD_SETSIZE,
              &readfd, &writefd, &exceptfd,
              timer_wait);
if (num < 0)
   
    if (errno == EINTR)
      continue; /* signal received - process it */
    zlog_warn ("select() error: %s",
               safe_strerror (errno));
    return NULL;
   
if (num == 0)
   
    /* Timeout handling */
    thread_timer_process (&m->timer);
   
if (num > 0)
   
    thread_process_fd (&readfd);
    thread_process_fd (&writefd);
   
thread_process_fd (fds) function iterates on each file descriptor fd and executes the appropriate action if FD_ISSET (fds, fd) is true. Integrating NetSNMP into such an event loop is easy: NetSNMP provides the snmp_select_info() function which alters a set of file descriptors to insert its own. Here is the new code:
#if defined HAVE_SNMP
struct timeval snmp_timer_wait;
int snmpblock = 0;
int fdsetsize;
#endif
/* ... */
#if defined HAVE_SNMP
fdsetsize = FD_SETSIZE;
snmpblock = 1;
if (timer_wait)
   
    snmpblock = 0;
    memcpy(&snmp_timer_wait, timer_wait,
           sizeof(struct timeval));
   
snmp_select_info(&fdsetsize, &readfd,
                 &snmp_timer_wait, &snmpblock);
if (snmpblock == 0)
  timer_wait = &snmp_timer_wait;
#endif
num = select (FD_SETSIZE,
              &readfd, &writefd, &exceptfd,
              timer_wait);
if (num < 0)   /* ... */  
#if defined HAVE_SNMP
if (num > 0)
  snmp_read(&readfd);
else if (num == 0)
   
    snmp_timeout();
    run_alarms();
   
netsnmp_check_outstanding_agent_requests();
#endif
/* ... */
snmp_select_info() may modify the provided set of file descriptors (and therefore, its size). It may also modify the provided timer in case it needs to schedule an action before the original timeout. snmpblock is a tricky variable. While select() can be called with a timeout set to NULL, this is not the case of snmp_select_info(): you have to pass a valid pointer. snmpblock is set to 0 if the provided timeout must be considered or to 1 otherwise. snmp_select_info() will set snmpblock to 0 if and only if it alters the timeout. Here are a two examples of integration. Both are for a subagent but the code for a manager is the same:

Third-party event loop With a third party event loop, you don t have access to the select() system call anymore. Therefore, you cannot alter it with snmp_select_info(). Instead, at each iteration of the loop, a list of SNMP related file descriptors is kept up-to-date with the help of snmp_select_info(): new ones are added and old ones are removed.

libevent Let s assume that snmp_fds is the list of current SNMP related events1. A function levent_snmp_update() calls snmp_select_info() to update this list. Here is a partial implementation (error handling removed and some declarations omitted, look at the complete version from lldpd):
static void
levent_snmp_update()
 
    int maxfd = 0;
    int block = 1;
    FD_ZERO(&fdset);
    snmp_select_info(&maxfd, &fdset, &timeout, &block);
    /* We need to untrack any event whose FD is not in  fdset 
       anymore */
    for (snmpfd = TAILQ_FIRST(snmp_fds);
         snmpfd;
         snmpfd = snmpfd_next)  
        snmpfd_next = TAILQ_NEXT(snmpfd, next);
        if (event_get_fd(snmpfd->ev) >= maxfd  
            (!FD_ISSET(event_get_fd(snmpfd->ev), &fdset)))  
            event_free(snmpfd->ev);
            TAILQ_REMOVE(snmp_fds, snmpfd, next);
            free(snmpfd);
          else
            FD_CLR(event_get_fd(snmpfd->ev), &fdset);
     
    /* Invariant: FD in  fdset  are not in list of FD */
    for (int fd = 0; fd < maxfd; fd++)
        if (FD_ISSET(fd, &fdset))  
            snmpfd->ev = event_new(base, fd,
                EV_READ   EV_PERSIST,
                levent_snmp_read,
                NULL);
            event_add(snmpfd->ev, NULL);
            TAILQ_INSERT_TAIL(snmp_fds, snmpfd, next);
         
    /* If needed, handle timeout */
    evtimer_add(snmp_timeout, block?NULL:&timeout);
 
Then, replace the main loop (usually, a call to event_base_dispatch()) with this:
do  
    if (event_base_got_break(base)  
        event_base_got_exit(base))
        break;
    netsnmp_check_outstanding_agent_requests();
    levent_snmp_update();
  while (event_base_loop(base, EVLOOP_ONCE) == 0);
Here is how are defined the two callbacks:
static void
levent_snmp_read(evutil_socket_t fd, short what, void *arg)
 
    FD_ZERO(&fdset);
    FD_SET(fd, &fdset);
    snmp_read(&fdset);
    levent_snmp_update();
 
static void
levent_snmp_timeout(evutil_socket_t fd, short what, void *arg)
 
    snmp_timeout();
    run_alarms();
    levent_snmp_update();
 

Twisted reactor As a second example, here is how to integrate NetSNMP into Twisted reactor. Twisted is an event-driven network programming framework written in Python. It does not come with support for SNMP2. Because of the language mismatch, the integration of NetSNMP in Twisted needs to be done using a C extension or the ctypes module. Twisted event loop is called a reactor. Like for libevent, events need to be registered. It is possible to register file descriptor-like objects using a class implementing a handful of methods. Here is the implementation of such a class (adapted from PyNetSNMP, a subproject of Zenoss using the ctypes module):
class SnmpReader:
    "Respond to input events"
    implements(IReadDescriptor)
    def __init__(self, fd):
        self.fd = fd
    def doRead(self):
        netsnmp.snmp_read(self.fd)
    def fileno(self):
        return self.fd
Like for libevent, we have a function to update the list of SNMP related events (stored as a mapping between file descriptors and SnmpReader instances):
class Timer(object):
    callLater = None
timer = Timer()
fdMap =  
def updateReactor():
    "Add/remove event handlers for SNMP file descriptors and timers"
    fds, t = netsnmp.snmp_select_info()
    for fd in fds:
        if fd not in fdMap:
            reader = SnmpReader(fd)
            fdMap[fd] = reader
            reactor.addReader(reader)
    current = Set(fdMap.keys())
    need = Set(fds)
    doomed = current - need
    for d in doomed:
        reactor.removeReader(fdMap[d])
        del fdMap[d]
    if timer.callLater:
        timer.callLater.cancel()
        timer.callLater = None
    if t is not None:
        timer.callLater = reactor.callLater(t, checkTimeouts)
Contrary to libevent, we cannot alter the main loop to call updateReactor() at each iteration. Therefore, it must be called after each SNMP related function. For another example, have a look at my equivalent implementation as a C extension.

Miscellaneous

Lack of asynchronicity Integrating NetSNMP into an event-based program is not risk-free. I have written a fairly comprehensive article on the lack of asynchronicity in NetSNMP AgentX protocol implementation. I urge you to read it if you want to integrate a SNMP subagent into an existing program. On the manager side, you will get similar drawbacks when using SNMPv3. To retrieve or manipulate management information using SNMPv3, it is necessary to know the identifier of the remote SNMP protocol engine. This identifier can be configured directly on each manager but it is usually discovered by querying SNMP-FRAMEWORK-MIB:: snmpEngineID, as described in RFC 5343. Unfortunately, this discovery is done synchronously by NetSNMP. There is a bug report for this issue but it seems difficult to fix. I have not been able to come up with a proper patch but I have described a workaround around snmp_sess_async_send(). There is no such problem with SNMPv2.

Limitation of file descriptor sets snmp_select_info() and snmp_read() uses the fd_set type to handle a set of file descriptors. This type should be manipulated with FD_CLR(), FD_ISSET(), FD_SET() and FD_ZERO(). Those may be defined as functions but they usually are macros. Moreover, no file descriptor greater than FD_SETSIZE (which is usually set to 1024) can be handled with the fd_set type. This means that if you have a file descriptor greater than 1024, you won t be able to use snmp_select_info() with it. Starting from NetSNMP 5.5, you can use snmp_select_info2() and snmp_read2() instead of snmp_select_info() and snmp_read(). They use the netsnmp_large_fd_set. If you want to keep compatibility with NetSNMP 5.4, here is another twist. The fd_set type is usually a fixed-size array of long integers. FD_CLR(), FD_ISSET() and FD_SET() are size-independant. The size only matters for FD_ZERO() which is not used by either snmp_select_info() or snmp_read(). You can therefore allocate a larger fd_set. A common way is to compile your program with -D__FD_SETSIZE=4096. You can still use FD_ZERO() yourself. However, this is not portable (GNU C Library only). Another way is to allocate your own array of long integers and cast it to fd_set. You ll have to redefine FD_ZERO():
typedef struct  
   long int fds_bits[4096/sizeof(long int)];
  my_fdset;
#undef FD_ZERO
#define FD_ZERO(fdset) memset((fdset), 0, sizeof(my_fdset))
 
   /* ... */
   my_fdset fdset;
   FD_ZERO(&fdset);
   /* ... */
   rc = snmp_select_info(&n, (fd_set *)&fdset, &timeout, &block);
   /* ... */
 

Threads NetSNMP provides two API:
  • the traditional session API which is not thread-safe and
  • the single session API.
In the above examples, I have used the traditional session API. If you are using threads, you need to ensure that all SNMP operations are done in a single thread. Otherwise, you need to adapt the examples to use the single session API. snmp_select_info() is part of the traditional session API. You should replace it with snmp_sess_select_info() and keep a list of SNMP sessions. This API is not available for the agent side of NetSNMP.

  1. The event list is global but some code could be added to bind a different list for each base in case you use different bases. This is not needed in the case of lldpd which uses only one event base.
  2. TwistedSNMP is an SNMP protocol implementation for Twisted based on PySNMP but is not maintained anymore. PySNMP comes with examples on how to integrate with Twisted.

Pietro Abate: managing puppet manifest with gitolite

Managing the puppet manifest using a vcs is a best practice and there is a lot of material on the web. The easier way to do it, is to use git directly in the directory /etc/puppet and use a simple synchronization strategy with an external repo, either to publish your work, or simply to keep a backup somewhere. Things are a bit more complicated when you would like to co-administer the machine with multiple people. Setting up user accounts, permission and everything can be a pain in the neck. Moreover working from your desktop is always more comfortable then logging in as root on a remote system and make changes there ... The solution I've chosen to make my life a bit easier is to use gitolite, that is a simple git gateway that works using ssh public keys for authentication and does not require the creating of local users on the server machine. Gitolite is available in debian and installing it is super easy : apt-get install gitolite . If you use puppet already you might be tempted to use puppet to manage your gitolite installation. This is all good, but I don't advice you to use modules like this one http://forge.puppetlabs.com/gwmngilfen/gitolite/1.0.0 as it's going to install gitolite from source and on debian it's not necessary ... For my purposes, I didn't find necessary to manage gitolite with puppet as all the default config options where good enough for me. Once your debian package is installed, in order to initialize your repo, you just need to pass to gitolite the admin public key, that is your .ssh/id_rsa.pub key and then run this command:
sudo -H -u gitolite gl-setup /tmp/youruser.pub
This will create the admin and testing repo in /var/lib/gitolite/repositories and setup few other things. At this point you are ready to test you gitolite installation by cloning the admin repo :
git clone gitolite@example.org:gitolite-admin.git
Gitolite is engineered to use only the gitolite use to manage all your repositories. To add more repositories and users you should have a look at the documentation and then editing the file conf/gitolite.conf to add your new puppet repository. At this point, you can go two ways. If you use git to manage your puppet directory, you can just make a copy of it somewhere and then add gitolite as a remote
git remote add origin gitolite@example.org:puppet.git
If you didn't use git before, you can just copy the manifest in your new git repository, make a first commit and push it on the server.
git push origin master
Every authorized users can now use git to clone your puppet repository, hack, commit, push ...
git clone gitolite@example.org:puppet
One last step is to add a small post-receive hook on the server to synchronize your gitolite repository with the puppet directory in /etc : This will sync your main puppet directory and trigger changes on the nodes for the next puppetd run. First I created a small shell script in /usr/local/bin/puppet-post-receive-hook.sh :
#!/bin/bash
umask 0022
cd /etc/puppet
git pull -q origin master
This script presuppose that your git repo in /etc/puppet has the gitolite repo as origin ... Then I added a simple hook in the gitolite git repo that calls this script using sudo :
sudo /usr/local/bin/puppet-post-receive-hook.sh
And since you are at it you should also add a pre-commit hook to check the manifest syntax. This will save you a lot of useless commits. If you use a more complicated puppet setup using environments (I'm not there yet, and I don't think my setup will evolve in that direction in the near future), you can use puppet-sync that seems a neat script to do the job. For the moment this setup works pretty well. I'm tempted to explore mcollective to trigger puppet runs on my nodes, but I'm there yet...

27 May 2012

Gregor Herrmann: RC bugs 2012/21

new week, new report about my work on RC bugs. again, mostly gcc 4.7 related, & again, mostly using patches prepared by others.

Petter Reinholdtsen: Debian Edu interview: Ralf Gesellensetter

In 2003, a German teacher showed up on the Debian Edu and Skolelinux mailing list with interesting problems and reports proving he setting up Linux for a (for us at the time) lot of pupils. His name was Ralf Gesellensetter, and he has been an important tester and contributor since then, helping to make sure the Debian Edu Squeeze release became as good as it is.. Who are you, and how do you spend your days? I am a teacher from Germany, and my subjects are Geography, Mathematics, and Computer Science ("Informatik"). During the past 12 years (since 2000), I have been working for a comprehensive (and soon, also inclusive) school leading to all kind of general levels, such as O- or A-level ("Abitur"). For quite as long, I've been taking care of our computer network. Now, in my early 40s, I enjoy the privilege of spending a lot of my spare time together with my wife, our son (3 years) and our daughter (4 months). How did you get in contact with the Skolelinux/Debian Edu project? We had tried different Linux based school servers, when members of my local Linux User Group (LUG OWL) detected Skolelinux. I remember very well, being part of a party celebrating the Linux New Media Award ("Best Newcomer Distribution", also nominated: Ubuntu) that was given to Skolelinux at Linux World Exposition in Frankfurt, 2005 (IIRC). Few months later, I had the chance to join a developer meeting in Ulsrud (Oslo) and to hand out the award to Knut Yrvin and others. For more than 7 years, Skolelinux is part of our schools infrastructure, namely our main server (tjener), one LTSP (today without thin clients), and approximately 50 work stations. Most of these have the option to boot a locally installed Skolelinux image. As a consequence, I joined quite a few events dealing with free software or Linux, and met many Debian (Edu) developers. All of them seemed quite nice and competent to me, one more reason to stick to Skolelinux. What do you see as the advantages of Skolelinux/Debian Edu? Debian driven, you are given all the advantages of a community project including well maintained updates. Once, you are familiar with the network layout, you can easily roll out an entire educational computer infrastructure, from just one installation media. As only free software (FOSS) is used, that supports even elderly hardware, up-sizing your IT equipment is only limited by space (i.e. available labs). Especially if you run a LTSP thin client server, your administration costs tend towards zero. What do you see as the disadvantages of Skolelinux/Debian Edu? While Debian's stability has loads of advantages for servers, this might be different in some cases for clients: Schools with unlimited budget might buy new hardware with components that are not yet supported by Debian stable, or wish to use more recent versions of office packages or desktop environments. These schools have the option to run Debian testing or other distributions - if they have the capacity to do so. Another issue is that Debian release cycles include a wide range of changes; therefor a high percentage of human power seems to be absorbed by just keeping the features of Skolelinux within the new setting of the version to come. During this process, the cogs of Debian Edu are getting more and more professional, i.e. harder to understand for novices. Which free software do you use daily? LibreOffice, Wikipedia, Openstreetmap, Iceweasel (Mozilla Firefox), KMail, Gimp, Inkscape - and of course the Linux Kernel (not only on PC, Laptop, Mobile, but also our SAT receiver) Which strategy do you believe is the right one to use to get schools to use free software?
  1. Support computer science as regular subject in schools to make people really "own" their hardware, to make them understand the difference between proprietary software products, and free software developing.
  2. Make budget baskets corresponding: In Germany's public schools there are more or less fixed budgets for IT equipment (including licenses), so schools won't benefit from any savings here. This privilege is left to private schools which have consequently a large share among German Skolelinux schools.
  3. Get free software in the seminars where would-be teachers are trained. In many cases, teachers' software customs are respected by decision makers rather than the expertise of any IT experts.
  4. Don't limit ourself to free software run natively. Everybody uses free software or free licenses (for instance Wikipedia), and this general concept should get expanded to free educational content to be shared world wide (school books e.g.).
  5. Make clear where ever you can that the market share of free (libre) office suites is much above 20 p.c. today, and that you pupils don't need to know the "ribbon menu" in order to get employed.
  6. Talk about the difference between freeware and free software.
  7. Spread free software, or even collections of portable free apps for USB pen drives. Endorse students to get a legal copy of Libreoffice rather than accepting them to use illegal serials. And keep sending documents in ODF formats.

Andrew Pollock: [life] Dear World, Sarah's Twitter account is okay

I've received two separate emails now about a spammy looking tweet from Sarah's Twitter account, so I thought I'd attempt to head off any more. Yes, it appears her account got hacked (weak password), but surprisingly the password didn't get changed. Sarah has changed her password to a stronger one and deleted the offending tweet. All is right with the world.

Lars Wirzenius: Larch 1.20120527 (Python B-tree library)

It has been a long time since I announced a version of Larch on my blog: about 14 months. A fair bit has happened since then: see the NEWS file for details. Below is the snippet for today's release. This is an implementation of particular kind of B-tree, based on research by Ohad Rodeh. See "B-trees, Shadowing, and Clones" (copied here with permission of author) for details on the data structure. This is the same data structure that btrfs uses. Note that my implementation is independent from the btrfs one, and might differ from what the paper describes. The distinctive feature of this B-tree is that a node is never modified (sort-of). Instead, all updates are done by copy-on-write. Among other things, this makes it easy to clone a tree, and modify only the clone, while other processes access the original tree. This is utterly wonderful for my backup application, and that's the reason I wrote larch in the first place. See the home page and README for more information; the API docs may be amusingly sparse. Version 1.20120527, released 2012-05-27

Next.

Previous.