Search Results: "ag"

2 May 2011

Michal Čihař: Photo of a week 17

Finally some picture published on time. Going to slightly larger animals this week. Picture taken at parkour race in Prague. Race Taken again using PENTAX K7 + PENTAX-DA 70mm F2.4 Limited.

Filed under: English Photography 1 comments Flattr this!

Lucas Nussbaum: Debian rolling discussion on -devel@

Here is my attempt at a summary of the rolling discussion currently happening on debian-devel@. It might not be complete, it s probably a bit biased, but I hope that it s still better than nothing. It was also posted on debian-devel@. If you are involved in Debian development, please discuss it on debian-devel@, rather than in the comments of this blog. If you want to provide feedback about the Debian rolling idea, use this Doodle poll and/or the comments of this blog.
Update : Someone had the good idea to vandalize the poll, and remove all the votes from people in favor of rolling. Before the vadalism (log), the results were:
37 votes: I am a Debian user, I use testing or unstable, and I would like rolling to happen.
6 votes: I am a Debian user, I use testing or unstable, and I DON T want rolling to happen.
17 votes: I am a Debian user, I use stable, and I would like rolling to happen.
3 votes: I am a Debian user, I use stable, and I DON T want rolling to happen.
7 votes: I am NOT a Debian user, but I would probably use it if rolling happened.
1 vote: I am NOT a Debian user, and I don t care about rolling. Motivations There s some user demand for rolling releases. For evidence, one can look at the usage of Debian testing or unstable which clearly goes further than the Debian development community. Or at the quickly growing market share of ArchLinux. Or at the interest in LinuxMint and Aptosid. Or at the DPL s report of his interactions with the press. Debian s only official product is its stable releases. While it s a clearly great and useful product, many users and developers don t recognize themselves in it: it contains software that is too old for their needs. The success of Ubuntu is related to this problem: Ubuntu proposes a different compromise between stability and up-to-date-ness. While concurrencing Ubuntu with more frequent stable releases (released every 6-months, for example) doesn t look like the right thing to do, Debian is in a perfect position to provide a rolling release with marginal additional work, since Debian already has testing (and unstable) to build on. The rolling discussion investigates how we could endorse the concept of a rolling release, provided as a product in addition to stable releases. This rolling release would be based on the current testing branch. Benefits for Debian: Current proposed plan for rolling (Disclaimer: this is mostly my view. It is shared by others, but some details might not be) rolling is mostly about (external) communication. It is not expected to change our development processes fundamentally.
It would be a statement by the project (through a GR, for example). A very preliminary draft was proposed by Raphael Hertzog:
Title: Debian endorses usage of testing by end-users, and renames it to rolling The Debian project recognizes that the Debian testing distribution initially created to make it easier to prepare and test the next stable release has become a useful product of its own. It satisfies the needs of users who are looking for the latest stable versions of software and who can cope (or even appreciate) a system that s constantly evolving. The Debian project decides to endorse this usage and will strive to provide a good experience to users of testing . To better communicate this policy change to our users, testing will be renamed rolling .
Yes, it s mostly PR bullshit , and I don t expect it to significantly change Debian development processes. However, communication is necessary if we want to attract new users. What would change is more attention from developers to what happens in testing/rolling, which is a good thing since a better testing/rolling makes it easier to create stable releases from it. Open questions Most of the discussion is about the influence of the introduction of rolling on Debian development processes, and in particular, on the painful process resulting in stable releases. Many fear that, with rolling , it will be harder to get stable releases out. The root question is: if we do rolling, what do we do during freezes? Several options have been mentioned in the thread:
  1. We could decide not to do anything special: just freeze rolling for ~6 months, as we used to freeze testing. That might bore people who like the constant flux of new upstream releases, but if we decide that it s the only way to ensure high-quality stable releases, so be it.
  2. We could decide to fork a frozen branch when we freeze, and continue to manage rolling using migrations from unstable.
  3. We could mix both solutions: freeze rolling for 3 months, so that most of the stabilization work occurs with a single active branch, and then, for the final release preparation, fork frozen off rolling, and unfreeze rolling.
Two kinds of objections have been raised: Other things that were discussed: Tentative personal conclusion I have the impression that advertising testing as a rolling release usable by end-users is generally considered a good thing.
The renaming of testing to rolling is not as consensual, but most opponents have a whatever ; if you want position. How we deal with freezes is the hard point in this discussion. I m personnally in favor of the freeze rolling for 3 months, then fork frozen and unfreeze rolling plan, though it has some problems too (it is not clear whether the required manpower really decreases at the end of freezes). Where do we go from there? After another round of discussion, we might postpone the how to deal with freezes question to later, and proceed with a GR to measure the support for the testing for end-users + s/testing/rolling/ proposal.

Russell Coker: This Event Will be Photographed

sign saying This event will be photographed. Images may be used in Banyule City Council promotional material (posters, website, flyers etc).  If you do not want your image used, please go to Event Control staff and tell them. The above sign was prominently displayed at a fair that was recently run by the Banyule Council. We have had some discussions in the Linux community about what policies should be used in regard to photography at events, it seems that this is a precedent for being able to opt-out of public photography.

Christian Perrier: 2011 week 17 Debian work

Will I succeed maintaining a weekly log of what I'm doing in Debian? We'll see. At least, let's try. D-I work: After spending most of the end of week 16 uploading D-I components, I completed this during this week. I hunted down the remaining arch:all, arch:i386 and arch:amd64 packages that need an upload because of l10n changes. The main point was bringing published D-I work inline with the work of translators. That was mostly meant to publicly have an image with Uyghur activated (which I blogged about separately). Finally, I ended up doing some changes to the fonts-ukij-uyghur package in order to use the right font to display the language in D-I, as well as improving the readability by increasing the font size just as we're already doing for Arabic and Persian. Samba packaging: I got release managers approval to upload a new release to stable-proposed-updates. It fixes several important bugs in samba 3.5.6, often affecting Win7 clients. I'm always happy to see that we can do that even when issues are ot security or "release critical" issues. I also integrated the newly-published samba 3.6.0-pre3 in the package maintenance SVN. However, we're still having problems with symbols disappearing from libwbclient, that prevent me to upload 3.6.0pre3 in experimental. Steve Langasek's help here will be welcome. Pytrainer and "gant" packaging: I setup a mailing list for the pkg-running maintenance team so that we can move the few packages some of us maintain (pytrainer and garmin-forerunner-tools as of now, and soon "garmin-ant-downloader", the helper tool for 405 watches). I also began looking at pytrainer bugs and forwarding some of them upstream. I also began packaging "gant" (had to rename it as we already have a gant package, totally unrelated). I just got a notice from Klaus Ethgen about improvements he did to the software. We'll try to figure out what to do as this software doesn't really have a clear upstream and published versions. l10n work: a few packages introduced debconf templates and I launched reviews of English for them (while mumbling about uncoordinated work, of course..;-)). I also reviewed about one hundred package description translations in French DDTP (something I recently began to commit to do even though it means onlinen work for me, which I'm not used to). No l10n NMUs this week but a few maintainer uploads after I (gently) kicked their ass..:-)

1 May 2011

Christian Perrier: [life] Bhutanese music

I really don't know how I ended up spending this entire evening listening to Bhutanese music. But I did well. I particularly enjoyed Rang Sem Kar Ga Mikar and Gayser Jang Lu Jonpi Gang, which I both found on YouTube. The former reminded be the pleasure and (apparent) simplicity of Bhutanese dances, which I had the privilege to share with guests in Thimphu a few years ago. The latter gives a great idea of the peacefulness of that place of the world (I recommend scenery views some of which probably taken at Dochula Pass). And that language is so beautiful even when one doesn't understand a word. Maybe some readers of this blog will become listeners of Bhutanese music for some time after reading this. I just can wish this. And, doh, I really can't wait until I go back there. I will, one day or another. "Ku Zu Zang Po La" to those who made me discover that place.

Michal Čihař: Photo of a week 16

Slightly delayed, but here comes photo for last week. I simply failed to download it from camera and process it. Continuing in animal macro, this time it is frog somewhere in Beskydy mountains. Frog Taken again using same combo PENTAX K7 + PENTAX-DA 35mm F2.8 Macro Limited.

Filed under: English Photography 0 comments Flattr this!

Luca Falavigna: Blessed Ioannes Paulus II

I will never forget.

Christian Perrier: Preview of Debian Installer in Uyghur

Last week, I announced the activation of Uyghur language in Debian Installer. For once, I even went further: I rebuilt all D-I packages that include localization (at least for i386/amd64 architectures) and uploaded them. As a consequence, D-I daily builds now allow previewing how an install done in Uyghur can be like. For this, please go to the Debian Installer development page, download the netboot ISO image (not netinst) and launch D-I with "suite=unstable" passed as argument when booting the ISO image. You should end up with something like this, this or this. There are a few glitches, still: for instance, as the font we chose has no Bold variant, the screens "titles" do not use an Uyghur font but an Arabic or Persian one. As a consequence, glyphs ar enot properly joined together. I'm investigating this issue with Uyghur translators and we're trying to choose an appropriate font among the gazillion that have been created by UKIJ. Please notice that Uyghur looks like Arabic but, according to those who can read Arabic, is really different from Arabic (or Persian). I hope that this great work by Gheyret Tohti, Abduqadir Abliz and other people working on this will give ideas to other so-called "minorities", here or there in the world, who want to be able to install Debian in their language. There is still a lot of work to do, such as having end-user software localization for Uyghur and, then, a uyghur-desktop task, but this is at least the first language that gets added for wheezy. In case you would be wondering, the current count of activated languages in D-I is 71.

30 April 2011

Russell Coker: Autism Awareness and the Free Software Community

It s Autism Awareness Month April is Autism Awareness month, there are lots of articles about Autism, most of them are at best misleading if not wrong. The most common error seems to be describing non-verbal Autistic children and then claiming that Autistic people comprise about 1% of the population. The number of non-verbal Autistic children is a lot less than 1% of the population, and at least 1/3 of the people on the Autism Spectrum have good verbal skills and often talk more than average. To be precise, they are describing the symptoms of people who would be diagnosed with Severity 3 according to the American Psychiatric Association s Diagnostic and Statistical Manual (the DSM) as applying to all people who are on the Autism Spectrum even though of the currently diagnosed people probably about 1/3 might be classified as Severity 2 and 1/3 as Severity 1. It seems that severity 1 Autism is going to be the replacement for Asperger Syndrome in the new DSM-5. The problem with misleading information in regard to what comprises Autism is that parents really don t want to have a child who s abnormal and search for evidence to make their wishes come true. At the extreme there are parents who can convince themselves that a 4yo non-verbal child isn t on the Autism Spectrum. When people are given the impression that Autism is only about non-verbal children then they will probably think that a 6yo who can give a half-hour lecture on a topic of interest (regardless of whether the audience is interested) isn t on the Autism Spectrum. There is also a similar issue with undiagnosed adults, I would have probably got assessed at least a decade earlier if I had learned more about Autism instead of believing some of the stories. In a quick Google search for Autism Awareness Month I haven t found any clear and accurate information on what Autism is or what should be done about it, so it seems to me that the awareness project has largely been a failure. The Autism page on Wikipedia [1] and the page for Asperger Syndrome [2] are good places to start reading. By merely pointing to those two Wikipedia pages I ve made this post a lot more useful than every article I ve seen in the mainstream media. To find some useful and insightful posts about Autism awareness you have to add some search terms such as neurodiversity, which turns up posts such as a post by Leah Jane about Autism Acceptance [3]. It seems that the Autism Awareness articles that are most useful at informing people will only be found by people who are already aware of some of the issues. Most People Who Read this are Geeks The majority of the readers for my blog come from aggregators that cater to fairly geeky people. Geek culture is largely Aspie culture. So I think that most people who read my blog have a good general knowledge of what people who might be diagnosed with Asperger Syndrome or High Functioning Autism are like they just think that such people are merely a bit more geeky than is typical for the Free Software community. So it s not as if all Autistic people are hidden away somewhere because they can t live independently. Many of us work in the computer industry and in academia. Probably the majority of people who have a university degree related to science or engineering have had a professor who was on the Autism Spectrum, probably most people who have any significant experience in the computer industry have had colleagues who are on the Spectrum. After having become used to the geek culture I initially found it difficult to believe that I could be on the Autism Spectrum because I knew heaps of people in the Free Software community who are at least as geeky as me but apparently weren t on the Autism Spectrum. I now think that the majority of people who are as geeky as me would get diagnosed with Asperger Syndrome (or High Functioning Autism under DSM-5) if they were assessed. Adult Diagnosis If you are wondering whether you might be on the Autism Spectrum then it s worth reading the pages linked from the DSM 5 draft process have the full diagnostic criteria (both for DSM-4 and DSM-5) [4]. Also Glenn Rowe has a web page with tests designed by Professor Simon Baron-Cohen [5] and Leif Ekblad has a project to do long-term analysis of changes to personality test scores which starts with a quick Aspie quiz [6] (and also has lots of links to other relevant sites). If you think that you might be on the Autism Spectrum (which means Asperger Syndrome in the case of most people who read my blog) then I recommend that you get assessed if possible. As assessment by a psychologist isn t cheap there is the option of self-diagnosis in which case Chaotic Idealism s post about self-diagnosis is worth reading [7].

Thomas Girard: ACE+TAO Debian packaging moved to git

We recently converted Debian ACE+TAO package repository from Subversion to git. This was a long and interesting process; I learned a lot on git in the course. I had been using git for a while for other packages: BOUML, dwarves and GNU Smalltalk. But I did not really get it. A preliminary study led by Pau[1] showed that out of the following three tools: the last one was giving results that look better.
The conversion svn-all-fast-export requires physical access to the repo, so the Alioth SVN repo was copied on my machine svn-pkg-ace/ before running the tool:
svn-all-fast-export --identity-map authors.txt --rules pkg-ace.rules svn-pkg-ace
Here's the content of the pkg-ace.rules configuration file that was used:
create repository pkg-ace
end repository
match /trunk/
  repository pkg-ace
  branch master
end match
match /(branches tags)/([^/]+)/
  repository pkg-ace
  branch \2
end match
The author mapping file authors.txt being:
markos = Konstantinos Margaritis <email-hidden>
mbrudka-guest = Marek Brudka <email-hidden>
pgquiles-guest = Pau Garcia i Quiles <email-hidden>
tgg = Thomas Girard <email-hidden>
tgg-guest = Thomas Girard <email-hidden>
The tool sample configuration file merged-branches-tags.rules recommends to post-process tags, which are just a branch in SVN. That's why the configuration file above treats branches as tags. The conversion was indeed fast: less than 1 minute.
Post-conversion observations Invoking gitk --all in the converted repo revealed different kind of issues:
  • svn tags as branches: http://thomas.g.girard.free.fr/ACE/tags-as-branches.png Branches are marked with green rectangles, and tags with yellow arrows. What we have here (expected given our configuration of the tool) are branches (e.g. 5.4.7-5) corresponding to tags, and tags matching the SVN tagging commit (e.g. backups/5.4.7-5@224). We'll review and fix this.

  • merged code that did not appear as such: http://thomas.g.girard.free.fr/ACE/missing-merge-metadata.png Branches that were not merged using svn merge look like they were not merged at all.

  • commits with wrong author: http://thomas.g.girard.free.fr/ACE/wrong-author.png Before being in SVN, the repository was stored in CVS. When it was imported into SVN, no special attention was given to the commit author. Hence I got credited for changes I did not write.

  • obsolete branches: http://thomas.g.girard.free.fr/ACE/obsolete-branches.png The tool leaves all branches, including removed ones (with tag on their end) so that you can decide what to do with them.

  • missing merges: http://thomas.g.girard.free.fr/ACE/missing-merge.png The branch 5.4.7-12 was never merged into the trunk!

Learning git Based on observations above, I realized my limited knowledge won't do to complete the conversion and clean the repository. There are tons of documentation on git out there, and you can find a lot of links from the git documentation page. Here's the one I've used:
The Git Object Model It's described with pictures here. You really need to understand this if you haven't already. Once you do, you understand that git is built bottom-up: the plumbing then the porcelain. If you can't find the tool you need, it's easy to write it.
git fast-import The Migrating to Git chapter explains how you can use the git fast-import tool to manually import anything into git. I've used it to create tags with dates in the past, slightly changing the Custom Importer example in the book:
#!/usr/bin/env ruby
#
# retag.rb
#
# Small script to create an annotated tag, specifying commiter as well as
# date, and tag comment.
#
# Based on Scott Chacon "Custom Importer" example.
#
# Arguments:
#  $1 -- tag name
#  $2 -- sha-1 revision to tag
#  $3 -- committer in the form First Last <email>
#  $4 -- date to use in the form YYYY/MM/DD_HH:MM:SS

def help
  puts "Usage: retag <tag> <sha1sum> <committer> <date> <comment>"
  puts "Creates a annotated tag with name <tag> for commit <sha1sum>, using "
  puts "given <committer>, <date> and <comment>"
  puts "The output should be piped to git fast-import"
end
def to_date(datetime)
  (date, time) = datetime.split('_')
  (year, month, day) = date.split('/')
  (hour, minute, second) = time.split(':')
  return Time.local(year, month, day, hour, minute, second).to_i
end
def generate_tag(tag, sha1hash, committer, date, message)
  puts "tag # tag "
  puts "from # sha1hash "
  puts "tagger # committer  # date  +0000"
  print "data # message.size \n# message "
end
if ARGV.length != 5
  help
  exit 1
else
  (tag, sha1sum, committer, date, message) = ARGV
  generate_tag(tag, sha1sum, committer, to_date(date), message)
end
graft points (graft means greffe in French) Because of missing svn:mergeinfo some changes appear unmerged. To fix this there are graft points: they override git idea of parents of a commit. To create a graft point, assuming 6a6d48814d0746fa4c9f6869bd8d5c3bc3af8242 is the commit you want to change, currently with a single parent 898ad49b61d4d8d5dc4072351037e2c8ade1ab68, but containing changes from commit 11cf74d4aa996ffed7c07157fe0780ec2224c73e:
me@mymachine$ echo 6a6d48814d0746fa4c9f6869bd8d5c3bc3af8242 11cf74d4aa996ffed7c07157fe0780ec2224c73e 898ad49b61d4d8d5dc4072351037e2c8ade1ab68 >> .git/info/grafts
git filter-branch git filter-branch allows you to completely rewrite history of a git branch, changing or dropping commits while traversing the branch. As an additional benefit, this tool use graft points and make them permanent. In other words: after running git filter-branch you can remove .git/info/grafts file. I've used it to rewrite author of a given set of commits, using a hack on top of Chris Johnsen script:
#!/bin/sh

br="HEAD"
TARG_NAME="Raphael Bossek"
TARG_EMAIL="hidden"
export TARG_NAME TARG_EMAIL
filt='
    if test "$GIT_COMMIT" = 546db1966133737930350a098057c4d563b1acdf -o \
            "$GIT_COMMIT" = 23419dde50662852cfbd2edde9468beb29a9ddcc; then
        if test -n "$TARG_EMAIL"; then
            GIT_AUTHOR_EMAIL="$TARG_EMAIL"
            export GIT_AUTHOR_EMAIL
        else
            unset GIT_AUTHOR_EMAIL
        fi
        if test -n "$TARG_NAME"; then
            GIT_AUTHOR_NAME="$TARG_NAME"
            export GIT_AUTHOR_NAME
        else
            unset GIT_AUTHOR_NAME
        fi
    fi
'
git filter-branch $force --tag-name-filter cat --env-filter "$filt" -- $br
(Script edited here; there were much more commits written by Raphael.)

Important

It's important to realize that the whole selected branch history is rewritten, so all objects id will change. You should not do this if you already published your repository.

The --tag-name-filter cat argument ensures our tags are copied during the traversal; otherwise they would be untouched, and hence not available in the new history.

Hint

Once git filter-branch completes you get a new history, as well as a new original ref to ease comparison. It is highly recommended to check the result of the rewrite before removing original. To shrink the repo after this, git clone the rewritten repo with file:// syntax -- git-filter-branch says it all.

Cleaning up the repo To recap, here's how the ACE+TAO git repo was changed after conversion:
  1. Add graft points where needed.

  2. Clean tags and branches. Using git tag -d, git branch -d and the Ruby script above it was possible to recreate tags. During this I was also able to add missing tags, and remove some SVN errors I did -- like committing in a branch created under tags/.

  3. Remove obsolete branches.

  4. Merge missing pieces. There were just two missing debian/changelog entries. I did this before git filter-branch because I did not find a way to use the tool correctly with multiple heads.

  5. Fix commit author where needed. Using the shell script above Raphael is now correctly credited for his work.

That's it. The ACE+TAO git repository for Debian packages is alive at http://git.debian.org/?p=pkg-ace/pkg-ace.git;a=summary.
[1]http://lists.alioth.debian.org/pipermail/pkg-ace-devel/2011-March/002421.html
[2]available in Debian as svn-all-fast-export

Julien Blache: forked-daapd: v0.16

forked-daapd v0.16 is a bugfix release, mainly addressing a bad regression introduced in v0.15. Fixes: The DAAP bug is what causes an incorrect display and sorting in iTunes, the tags issue could cause a segfault in some cases, and finally fixing the file size in the HTTP streaming code may or may not fix some issues but was a bad bug anyway. Tarballs available at http://alioth.debian.org/~jblache/forked-daapd/ ; GPG signatures made with my Debian key F5D65169.

Christian Perrier: Pytrainer: finally

This week, I finally switched to use pytrainer to track my running data. Up to now, I was still using Garmin proprietary software on Elizabeth's Windows copmuter for this. Mostly because my GPW watch is a Forerunner 405, that uses a bloody crappy stupid USB stick with proprietary communication protocol (ANT+). So, the use of proprietary crappy software was kinda mandatory. There is however some code floating around to handle this device in Linux, named "Gant". This is far from being very polished, but I finally managed to make it work more or less reliably to get the GPS traces from my watch to my Debian laptop. So, during the week , I brought all my old running data back from "Garmin Training Center" (the Windows software provided by Garmin) back to my laptop and imported it in pytrainer. And I imported new data from my daily runs with help of that "gant" software. So, this is now time to really jump into maintenance of pytrainer and help my friend No l (that will be easy as pytrainer is kinda well maintained), then package "gant" for Debian, as part of the work in the "pkg-running" maintenance team (Ralf Treinen, No le K the and /me). New challenge for me: learn how to create a git public repo and maintain a package in it (I already participate to maintenance work with git for several things, but I never initiated any). Other plans: some bug triage on pytrainer (if some is needed, I indeed don't know yet) and eventually deal with things that have to be reported upstream. Looking at Launchpad might also be a good idea as I suspect that many are using pytrainer with Ubuntu.

29 April 2011

Ian Wienand: Toolchains, libc and interesting interactions

When building a user space binary, the -lc that gcc inserts into the final link seems pretty straight forward link the C library. As with all things system-library related there is more to investigate. If you look at /usr/lib/libc.so; the "library" that gets linked when you specify -lc, it is not a library as such, but a link script which specifies the libraries to link, which also includes the dynamic linker itself:
$ cat /usr/lib/libc.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a  AS_NEEDED ( /lib/ld-linux-x86-64.so.2 ) )
On very old glibc's the AS_NEEDED didn't appear; so every binary really did have a DT_NEEDED entry for the dynamic linker itself. This can be somewhat confusing if you're ever doing forensics on an old binary which seems to have these entries for not apparent reason. However, we can see that that /lib/libc.so itself does actually require symbols from the dynamic linker:
$ readelf --dynamic /lib/libc.so.6   grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
Although you're unlikely to encounter it, a broken toolchain can cause havoc if it gets the link order wrong here, because the glibc dynamic linker defines minimal versions of malloc and friends -- you've got a chicken-and-egg problem using libc's malloc before you've loaded it! You can simulate this havoc with something like:
$ cat foo.c
#include <stdio.h>
#include <syslog.h>
int main(void)
 
   syslog(LOG_DEBUG, "hello, world!");
   return 0;
 
$ cat libbroken.so
GROUP ( /lib/ld-linux-x86-64.so.2 )
$ gcc -o -Wall -Wl,-rpath=. -L. -lbroken -g -o foo foo.c
$ ./foo
Inconsistency detected by ld.so: dl-minimal.c: 138: realloc: Assertion  ptr == alloc_last_block' failed!
Depending on various versions of things, you might see that assert or possibly just strange, corrupt output in your logs as syslog calls the wrong malloc. You could debug something like this by asking the dynamic linker to show you its bindings as it resolves them:
$ LD_DEBUG_OUTPUT=foo.txt LD_DEBUG=bindings ./foo
Inconsistency detected by ld.so: dl-minimal.c: 138: realloc: Assertion  ptr == alloc_last_block' failed!
$ cat foo.txt.11360   grep "\ malloc'"
     11360:	binding file /lib/libc.so.6 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol  malloc' [GLIBC_2.2.5]
     11360:	binding file /lib64/ld-linux-x86-64.so.2 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol  malloc' [GLIBC_2.2.5]
     11360:	binding file /lib/libc.so.6 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol  malloc' [GLIBC_2.2.5]
Above, because the dynamic loader comes first in the link order, libc.so.6's malloc has bound to the minimal implementation it provides, rather the full-featured one it provides internally. As an aside, AFAICT, there is really only one reason why a normal library will link against the dynamic loader -- for the thread-local storage support function __tls_get_addr. You can try this yourself:
$ cat tls.c
char __thread *foo;
char* moo(void)  
	return foo;
 
$ gcc -fPIC -o libtls.so -shared tls.c
$ readelf -d ./libtls.so    grep NEED
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
 0x00000001 (NEEDED)                     Shared library: [ld-linux.so.2]
 0x6ffffffe (VERNEED)                    0x314
 0x6fffffff (VERNEEDNUM)                 2
Thread-local storage is worthy of a book of its own, but the gist is that this support function says "hey, give me an address of foo in libtls.so", the magic being that if the current thread has never accessed foo then it may not actually have any storage for foo yet, so the dynamic linker can allocate some memory for it lazily and then return the right thing. Otherwise, every thread that started would need to reserve room for foo "just in case", even if it never cares about moo. But looking a little closer at the symbols of libc.so is also interesting. libc.so doesn't actually have many functions you can override. You can see what it is possible to override by checking the relocations against the procdure-lookup table (PLT).
Relocation section '.rela.plt' at offset 0x1e770 contains 8 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
00000035b000  084600000007 R_X86_64_JUMP_SLO 00000000000a2100 sysconf + 0
00000035b008  02e000000007 R_X86_64_JUMP_SLO 0000000000075e50 calloc + 0
00000035b010  01dd00000007 R_X86_64_JUMP_SLO 0000000000077910 realloc + 0
00000035b018  029300000007 R_X86_64_JUMP_SLO 00000000000661c0 feof + 0
00000035b020  046f00000007 R_X86_64_JUMP_SLO 00000000000768c0 malloc + 0
00000035b028  000400000007 R_X86_64_JUMP_SLO 0000000000000000 __tls_get_addr + 0
00000035b030  01b400000007 R_X86_64_JUMP_SLO 0000000000076dd0 memalign + 0
00000035b038  086000000007 R_X86_64_JUMP_SLO 00000000000767e0 free + 0
i.e. instead of jumping directly to the malloc defined in the libc code section, any internal calls will jump to this stub which, the first time, asks the dynamic linker to go out and find the address of malloc (it then saves it, so the second time the stub just jumps to the saved location). This is an interesting list, seemingly without much order. feof, for example, stands out as worth checking out a bit closer why would that be there when fopen isn't, say? We can track down where it comes from with a bit of detective work; knowing that the value of the symbol feof will be placed into 0x35b018 we can disassemble libc.so to see that this address is used by the feof PLT stub at 0x1e870 (luckily, objdump has done the math to offest from the rip for us; i.e. 0x1e876 + 0x33c7a2 = 0x35b018)
000000000001e870 <feof@plt>:
   1e870:       ff 25 a2 c7 33 00       jmpq   *0x33c7a2(%rip)        # 35b018 <_IO_file_jumps+0xb18>
   1e876:       68 03 00 00 00          pushq  $0x3
   1e87b:       e9 b0 ff ff ff          jmpq   1e830 <h_errno+0x1e7dc>
From there we can search for anyone jumping to that address, and find out the caller:
$ objdump --disassemble-all /lib/libc.so.6   grep 1e870
000000000001e870 <feof@plt>:
   1e870:	ff 25 a2 c7 33 00    	jmpq   *0x33c7a2(%rip)        # 35b018 <_IO_file_jumps+0xb18>
   f19f7:	e8 74 ce f2 ff       	callq  1e870 <feof@plt>
$ addr2line -e /lib/libc.so.6 f19f7
/home/aurel32/eglibc/eglibc-2.11.2/sunrpc/bindrsvprt.c:70
Which turns out to be part of a local patch which probably gets things a little wrong, as described below. The sysconf relocation is from a similar add-on patch (being used to find the page size, it seems). libc, like all sensible libraries, uses the hidden attribute on symbols to restrict what is exported by the library. The benefit of this is that when the linker knows you are referencing a hidden symbol it knows that the value can never be overridden, and thus does not need to emit extra code to do indirection just in case you ever wish to redirect the symbols. In the above, it appears that feof has never been marked as hidden, probably because no internal glibc functions used it until that add-on patch, and since it is not considered an internal function the linker must allow for the possibility that it will be overridden at run time and provide a PLT slot for it. There are consequences; if this was on a fast-path then the extra jumps required to go via the PLT may matter for performance and it may also cause strange behaviour if somebody had preloaded something that took over feof. Note, this is different from saying that your library can override symbols provided by libc.so; such as when you LD_PRELOAD a library to wrap an open call. What you can not override is the open call that say, the internal libc.so function getusershell does to read /etc/shells. Having the malloc related calls as preemptable seems intentional and sane; although I can not find a comment to the effect of "we deliberately leave these like this so that users may use alternative malloc implementations", it makes sense so that libc.so is internally using the same malloc as everything else if you choose to use something such as tc-malloc. tl;dr? Digging into your system libraries is always interesting. Be careful with your link order when creating toolchains, and be careful about symbol visibility when you're working with libraries.

Dirk Eddelbuettel: Slides from Rcpp workshop / master class yesterday

Romain and I just posted our slides from yesterday's Rcpp workshop and class (preceding the now-ongoing R/Finance conference). You can access the slides via my presentation page, or directly from here as Part 1 (Introduction), Part 2 (Details), Part 3 (Advanced) and Part 4 (Applications). The page also contains a link to the examples as a tar.gz and zip file.

Update: One link corrected.

Rob Bradford: Nicely formatting number types

I came across the following compiler warning today:
    CC     libplurk_la-plurk-item-view.lo
  plurk-item-view.c: In function  construct_image_url :
plurk-item-view.c 233  warning: format  %lld  expects type  long long int ,
but argument 3 has type  gint64 
Tut tut tut .. a compiler warning! How could the commiter had let this happen? Lets look at the code
url = g_strdup_printf ("http://avatars.plurk.com/%s-medium%lld.gif", ...);
Ahaha .. they re probably using a 32-bit system, the nice thing to do here is:
url = g_strdup_printf ("http://avatars.plurk.com/%s-medium%" G_GINT64_FORMAT ".gif", ...);
To handle the case that on a 64-bit system you can represent a gint64 as a long rather than needing to go for a long long. I ve seen this quite a bit with debugging output for size_t for which G_GSIZE_FORMAT is definitely your friend.

Steve Kemp: Goodbye, world.

Today I resigned from the Debian project. The following packages are up for adoption: I'll remove myself from Planet Debian tomorrow, assuming the keyring revokation isn't swift and merciless. ObQuote: This space is intentionally blank.

Rapha&#235;l Hertzog: Do we need project-wide support for Debian rolling?

The discussion about Debian rolling started sooner than expected on debian-devel (see the thread here). I initially wanted to iron out the biggest problems through discussions on my blog and try to submit a somewhat polished proposal instead we ended up discussing the same things both on -devel and on my blog. That said it s not that bad (except for the time I lost to have similar conversations in both places) because the debian-devel discussions included members of the release team and it looks like they are not fundamentally opposed to the idea. Despite this, the introduction of a frozen suite branched off from testing/rolling is not really consensual (yet?). But the idea of officially supporting testing on a best-effort basis appears to have almost no opposition. While some will undoubtedly believe that this is a useless exercise, I believe it would help if the project stated this in a somewhat official manner. My answer to the question in the title is thus: YES I am thus considering to submit a general resolution to that effect. My current draft is below.
Title: Debian endorses usage of testing by end-users, and renames it to rolling The Debian project recognizes that the Debian testing distribution initially created to make it easier to prepare and test the next stable release has become a useful product of its own. It satisfies the needs of users who are looking for the latest stable versions of software and who can cope (or even appreciate) a system that s constantly evolving. The Debian project decides to endorse this usage and will strive to provide a good experience to users of testing . To better communicate this policy change to our users, testing will be renamed rolling . While we believe that this is a good move, we would like to remind our users that Debian is a volunteer project and that our resources are not infinite. Package maintainers are contributing to Debian on a best-effort basis. This means that they might not be able to properly support their package(s) in all distributions. In that case, the project recommends that maintainers apply the following priorities:
  1. Support in stable (security updates, release critical bugs)
  2. Preparation of the next stable release
  3. Support in rolling
Note that this general resolution could have amendments with s/rolling/current/ and that would solve the bikeshedding over the name that started several times already. I deliberately separated Preparation of the next stable release and Support in rolling so that priorities are clear even if we decide to not freeze rolling and to introduce a frozen distribution to finalize the next stable release. I also did not go into too much details on the implications that it might have, it s best to leave that up to each contributor/team/etc. I hesitated to add a paragraph stating that we want to try to gradually improve the usability of testing but in the end I think it would be somewhat redundant. We re always trying to do our best when we decide to take on something. All comments welcome (even if you just agree and would be willing to second such a GR). Update: I will tweak the draft included in this article when I get good suggestions. Thanks to Lucas Nussbaum for the first one.

10 comments Liked this article? Click here. My blog is Flattr-enabled.

Petter Reinholdtsen: Initial notes on adding Open311 server API on FixMyStreet

The last few days I have spent some time trying to add support for the Open311 API in the Norwegian FixMyStreet service. Earlier I believed Open311 would be a useful API to use to submit reports to the municipalities, but when I noticed that the New Zealand version of FixMyStreet had implemented Open311 on the server side, it occurred to me that this was a nice way to allow the public, press and municipalities to do data mining directly in the FixMyStreet service. Thus I went to work implementing the Open311 specification for FixMyStreet. The implementation is not yet ready, but I am starting to get a draft limping along. In the process, I have discovered a few issues with the Open311 specification. One obvious missing feature is the lack of natural language handling in the specification. The specification seem to assume all reports will be written in English, and do not provide a way for the receiving end to specify which languages are understood there. To be able to use the same client and submit to several Open311 receivers, it would be useful to know which language to use when writing reports. I believe the specification should be extended to allow the receivers of problem reports to specify which language they accept, and the submitter to specify which language the report is written in. Language of a text can also be automatically guessed using statistical methods, but for multi-lingual persons like myself, it is useful to know which language to use when writing a problem report. I suspect some lang=nb,nn kind of attribute would solve it. A key part of the Open311 API is the list of services provided, which is similar to the categories used by FixMyStreet. One issue I run into is the need to specify both name and unique identifier for each category. The specification do not state that the identifier should be numeric, but all example implementations have used numbers here. In FixMyStreet, there is no number associated with each category. As the specification do not forbid it, I will use the name as the unique identifier for now and see how open311 clients handle it. The report format in open311 and the report format in FixMyStreet differ in a key part. FixMyStreet have a title and a description, while Open311 only have a description and lack the title. I'm not quite sure how to best handle this yet. When asking for a FixMyStreet report in Open311 format, I just merge title an description into the open311 description, but this is not going to work if the open311 API should be used for submitting new reports to FixMyStreet. The search feature in Open311 is missing a way to ask for problems near a geographic location. I believe this is important if one is to use Open311 as the query language for mobile units. The specification should be extended to handle this, probably using some new lat=, lon= and range= options. The final challenge I see is that the FixMyStreet code handle several administrations in one interface, while the Open311 API seem to assume only one administration. For FixMyStreet, this mean a report can be sent to several administrations, and the categories available depend on the location of the problem. Not quite sure how to best handle this. I've noticed SeeClickFix added latitude and longitude options to the services request, but it do not solve the problem of what to return when no location is specified. Will have to investigate this a bit more. My distaste for web forums have kept me from bringing these issues up with the open311 developer group. I really wish they had a email list available via Gmane to use for discussions instead of only a forum. Oh, well. That will probably resolve itself, one way or another. I've also tried visiting the IRC channel #open311 on FreeNode, but no-one seem to reply to my questions there. This make me wonder if I just fail to understand how the open311 community work. It sure do not work like the free software project communities I am used to.

28 April 2011

Russell Coker: What is Valid SE Linux Policy?

Guido Trentalancia started an interesting discussion on the SE Linux policy development list about how to manage the evolution of the policy [1]. The Problem The SE Linux policy is the set of rules that determine what access is granted. It assigns types to files and domains to processes and has a set of rules that specify all the permitted interactions between processes and files (among many other things). The policy evolves over time to match the requirements of programs (applications and daemons). As a program evolves the things that it does will change and the SE Linux policy will tend to evolve to permit the set of all operations that were requested by all versions because people only complain when things stop working not when excessive privilege is granted. So we need to periodically remove old allow rules from the policy. One difficulty in this regard is the fact that multiple versions of programs are often available for use at the same time. Debian in particular has a good history of providing separate packages for the old and new versions of programs such as Apache to meet the needs of users who want the tried and tested version and of users who want the newer version with better performance, more features, better documentation, or something else good. There is also a demand to have the same policy work with multiple versions of a distribution without excessive effort. Finally all the distributions that have SE Linux support have different people deciding when the new version of a daemon is ready for inclusion and therefore there is a need to support multiple versions for multiple distributions. So support for older versions of daemons can t be removed easily. One of the things I do to make these things a little easier to manage is to put ifdef( distro_debian', before any Debian specific bits of policy. When policy is conditional and only used in Debian I can freely remove it at any future time if Debian works well without it. Also it doesn t matter if such Debian specific policy allows access that is not needed or desired in other distributions, the only down-side to this is that sometimes other distributions need to repeat work that I did, they determine what access is needed for their configuration and discover that it was already enabled for Debian. What is Valid Policy? We went to only have Valid Policy (as described by Christopher J. PeBenito), so the challenge is determining what is Valid Policy. It seems to me that there are three type of access granted by valid policy (it is debatable whether type #3 is valid):
  1. Access that is needed for an application to perform it s minimal designed task.
  2. Access that is needed for the application to perform all the optional configurations, EG an ftpd running from inetd or as a daemon, and daemons like http server being granted access to ssl keys or not.
  3. Access that is needed to perform all the operations the application requests, but which the application doesn t require or shouldn t require if it worked correctly.
Some common operations that aren t required include opening utmp for write, searching /root, and many other relatively innocuous access attempts which don t affect the program operation if they are denied. There are also many things such as writing temporary files to /root that don t seem unusual if the application developer is not considering SE Linux (but which are often considered bad practice anyway). Some of these things (like using /root for stuff that belongs in /var/lib) have the potential to break things (for the daemon or for other system processes) even if you don t consider SE Linux. How to deal with those types:
  1. In most cases this can be determined without too much effort. For example a web server needs to listen on port 80 and read files and directories that relate to data. When writing policy I can write a lot of the allow rules without even testing the application because I know from the design what it will do. A large part of the other access is obvious in a I can t believe I didn t realise it would need this sense.
  2. The main question here is whether we have booleans (settings which can be tuned at run-time by the sysadmin which determine how the policy works) to specify which optional tasks or whether we allow all access for optional configurations by default. The secondary question is when certain unusual corner cases should be not supported at all such that the people who do such unusual corner cases need to use audit2allow to generate local policy to allow their operations.
  3. Sometimes we have to allow things that we really don t like. Even when we write policy to allow a daemon to do unusual things (such as using /root instead of /var/lib) it s still a lot better than running without SE Linux. Also SE Linux policy to allow such obviously broken things stands out and is a constant reminder that the daemon needs fixing, this is better than allowing symptoms of broken design to be forgotten.
How to Improve the Situation We could have comments in the policy source for everything that is in category 3. If the comments had a fixed format so that a recursive grep could find them all then it would allow us to more easily remove the gross things from the policy at a later date. But it seems to me that the main problem is a lack of people working on this. I am not aware of any people actively testing Debian policy for excessive privilege in regard to such issues.

Bastian Venthur: Installing Dropbox in Debian/Sid

A few months ago dropbox was a Debian packge in non-free. Recently I noticed that there are several new versions available and tried to notifiy the maintainer and ask him to update the package. But unfortunately it looks like the package is not available any more (Here s why). However getting Dropbox is quite easy if you know what to do. The obvious choice would be to go to dropbox.com and download either the source package and compile it yourself or try to install the precompiled Ubuntu packages. The first option is not really an option for most users and the second one does currently not work on Debian/Sid (aka unstable). However, there is an easy solution: on the download page search for the CLI script (currently it is named dropbox.py) Download that and run it with python dropbox.py start -i. That will download the propritary binaries into ~/.dropbox-dist. Now go into that folder and start dropboxd. There you go, you have a fully working systray app for dropbox. Depending on your desktop environment you might want to put dropboxd in your autostart . So you don t have to start the daemon every time you login. And that s it. It is not really ideal, since a Debian package would be much better for updates, security, yada yada, but it s still better than nothing.

Next.

Previous.