Search Results: "beh"

27 March 2013

Francesca Ciceri: bits.debian.org: challenge accepted

official blog for the Debian project? challenge accepted Ladies and gentlemen, I'm so excited to present you the official blog of the Debian Project. The idea is to make it the voice of the project, in addition to the official press releases and DPN, our bi-weekly newsletter.
I'm really excited about it not only because it will help us to communicate better with our users in letting them know what is going on, but also because it's something we long waited (and planned) for. And it was surely a collective effort: many thanks to Ana Guerrero, who not only is one of the main editors, but also the engine behind the idea and the realization of it; Stefano Zacchiroli, who during his three terms as DPL gave lots of support to this idea; and the Debian System Administrators Team who put all in place (and found the perfect name for it).

Benjamin Mako Hill: The Institute for Cultural Diplomacy and Wikipedia

A month ago, Mark Donfried from the Institute for Cultural Diplomacy (ICD) an organization dedicated to promoting open dialogue sent me this letter threatening me with legal action because of contributions I ve made to Wikipedia. Yesterday, he sent me this followup threat. According to the letters, Donfried has threatened me with legal action because I participated in a discussion on Wikipedia that resulted in his organization s article being deleted. It is not anything I wrote in any Wikipedia article that made Donfried so upset although Donfried is also unhappy about at least one off-hand comment I made during the deletion discussion on a now-deleted Wikipedia process page. Donfried is unhappy that my actions, in small part, have resulted in his organization not having an article in Wikipedia. He is able to threaten me personally because unlike many people I edit Wikipedia using my real, full, name. Donfried s letter is the latest step in a saga that has been ongoing since last June. It has been a frustrating learning experience for me that has made me worried about Wikipedia, its processes, and its future. In Wikipedia, debates can be won by stamina. If you care more and argue longer, you will tend to get your way. The result, very often, is that individuals and organizations with a very strong interest in having Wikipedia say a particular thing tend to win out over other editors who just want the encyclopedia to be solid, neutral, and reliable. These less-committed editors simply have less at stake and their attention is more distributed. The ICD is a non-profit organization based in Berlin. According to its own website, a large part of the organization s activities are based around arranging conferences. Its goals peace, cultural interchange, human rights are admirable and close to my heart. Its advisors and affiliates are impressive. I had never heard of the ICD before their founder, Mark Donfried, emailed me in April 2012 asking me to give a keynote address at their conference on The 2012 International Symposium on Cultural Diplomacy & Human Rights. I replied, interested, but puzzled because my own research seems very far afield of both cultural diplomacy (which I had never heard of) and human rights. I replied saying:
What would you like me to talk about I ask because I don t consider myself an expert in (or even particularly knowledgeable about) cultural diplomacy. Did someone else refer you to me?
Donfried replied with a long message seemingly copy and pasted thanking me for considering attending and asking me for details of my talk. I replied again repeating text from my previous email and asking why he was interested in me. Donfried suggested a phone call to talk about details. But by this point, I had looked around the web for information about the ICD and had decided to decline the invitation. Among things I found was a blog post by my friend James Grimmelmann that suggests that, at least in his case, the ICD had a history of sending unsolicited email and an apparently inability to take folks off their email lists even after repeated requests. I also read the Wikipedia article about the ICD. Although the Wikipedia article was long and detailed, it sent off some internal Wikipedian-alarm-bells for me. The page read, to me, like an advertisement or something written by the organization being described; it simply did not read to me like an encyclopedia article written by a neutral third-party. I looked through the history of the article and found that the article had been created by a user called Icd_berlin who had made no other substantive edits to the encyclopedia. Upon further examination, I found that almost all other significant content contributions were from a series of anonymous editors with IP addresses associated with Berlin. I also found that a couple edits had removed criticism when it had been added to the article. The criticism was removed by an anonymous editor from Berlin. Criticisms on the article included links to a website called Inside the ICD which was a website that mostly consisted of comments by anonymous people claiming to be former interns of the ICD complaining about the working conditions at the organization. There were also many very positive descriptions of work at the ICD. A wide array of pseudonymous users on the site accused the negative commenters of being liars and detractors and the positive commenters of being ICD insiders. I also found that there had been evidence on Wikipedia also removed without discussion by an anonymous IP from Berlin of an effort launched by the youth wing of ver.di one of the largest trade unions in Germany to campaign for good internships at the ICD. Although details of the original campaign have been removed from ver.di s website, the campaigned ended after coming to an agreement with the ICD that made explicit a set of expectations and created an Intern Council. Although the article about ICD on Wikipedia had many citations, many were to the ICD s own website. Most of the rest were to articles that only tangentially mentioned the ICD. Many were about people with ICD connections but did not mention the ICD at all. As Wikipedia editor, I was worried that Wikipedia s policies on conflict of interest, advertising, neutrality, and notability were not being served by the article in its state. But as someone with no real experience or knowledge of the ICD, I wasn t sure what to do. I posted a request for help on Wikipedia asking for others to get involved and offer their opinions. It turns out, there were several editors who had tried to improve the article in the past and had been met by pro-ICD editors reverting their changes. Eventually, those editors lost patience or simply moved on to other topics. By raising the issue again, I kicked off a round of discussion about the article. At the termination of that discussion, the article was proposed for deletion under Wikipedia s Articles for Deletion policy. A new Wikipedia editor began working enthusiastically to keep the article by adding links and by arguing that the article should stay. The new user edited the Wikipedia article about me to accuse me of slander and defamation although they removed that claim after I tried to explain that I was only trying to help. I spent quite a bit of time trying to rewrite and improve the article during the deletion discussion and I went link by link through the many dozens of citations. During the deletion discussion, Mark Donfried contacted me over email and explained that his representatives had told him that I was working against the ICD in Wikipedia. He suggested that we meet. We had a tentative plan to meet in Berlin on an afternoon last July but, in the end, I was too busy trying to submit my thesis proposal and neither of us followed up to confirm a particular time within the time window we had set. I have still never met him. My feeling, toward the end of the deletion discussion on Wikipedia, was mostly exasperation. Somewhat reluctantly, I voted to delete the article saying:
Delete This AFD is a complete mess for all the reasons that the article itself is. Basically: there are a small number of people who seem to have a very strong interest in the Institute for Cultural Diplomacy having an article in Wikipedia and, from what I can tell, very little else. Hessin fahem, like all the major contributors to the page, joined Wikipedia in order to participate in this issue. This article has serious problems. I have posted a detailed list of my problems on the article talk page: primary sources, conflict of interest for nearly all substantive contributions and reading like an advert are the biggest issues. My efforts to list these problems were reverted without discussion by an anonymous editor from Berlin. I have seen no evidence that the Institute for Cultural Diplomacy satisfies WP:ORG but I agree that it is possible that it does. I strongly agree with Arxiloxos that articles should always be fixed, and not deleted, if they are fixable. But I also know that Wikipedia does not deserve this article, that I don t know to fix it, and that despite my efforts to address these issues (and I ll keep trying), the old patterns of editing have continued and the article is only getting worse. This ICD seems almost entirely based around a model that involves organizing conferences and then calling and emailing to recruit speakers and attendees. A large number of people will visit this Wikipedia article to find out more about the organization before deciding to pay for a conference or to join to do an internship. What Wikipedia shows to them reads like an advert, links almost exclusively to of pages on the organizations websites and seems very likely to have been written by the organization itself. We are doing an enormous disservice to our readers by keeping this page in its current form. If somebody wants to make a serious effort to improve the article, I will help and will happily reconsider my !vote. But after quite a bit of time trying to raise interest and to get this fixed, I m skeptical this can be addressed and my decision reflects this fact. mako 05:18, 12 June 2012 (UTC)
I concluded that although the organization might be notable according to Wikipedia s policies and although the Wikipedia article about it might be fixable, the pattern of editing gave me no faith that it could be fixed until something changed. When the article was deleted, things became quiet. Several months later a new article was created again, by an anonymous user with no other edit history. Although people tend to look closely at previously deleted new pages, this page was created under a different name: The Institute of Cultural Diplomacy and was not noticed. Deleted Wikipedia articles are only supposed to be recreated after they go through a process called deletion review. Because the article was recreated out of this process, I nominated it for what is called speedy deletion under a policy specifically dealing with recreated articles. It was deleted again. Once again, things were quiet. In January, it seems, the Inside the ICD website was threatened with a lawsuit by the ICD and the maintainers of the site took it down with the following message:
Apparently, the ICD is considering filing a lawsuit against this blog and it will now be taken down. We completely forgot about this blog. Let s hope no one is being sued. Farewell.
On February 25, the Wikipedia article on ICD was recreated once again out of process and by a user with almost no previous edit history. The next day, I received an email from Mark Donfried. In the message, Donfried said:
Please note that the ICD is completely in favor of fostering open dialogue and discussions, even critical ones, however some of your activities are raising serious questions about the motives behind your actions and some even seem to be motives of sabotage, since they resulted in ICD not having any Wikipedia page at all. We are deeply concerned regarding these actions of yours, which are causing us considerable damages. As the person who initiated these actions with Wikipedia and member of the board of Wikipedia [1], we would therefore request your answer regarding our questions below within the next 10 days (by March 6th). If we do not receive your response we will unfortunately have to consider taking further legal actions with these regards against you and other anonymous editors.
I responded to Donfried to say that I did not think it was prudent to speak with him while he was threatening me. Meanwhile, other Wikipedia editors nominated the ICD article for deletion once again and unanimously decided to delete it. And although I did not participate in the discussion, Donfried emailed again with more threats of legal action hours after the ICD article was deleted:
[A]s the case of the ICD and its presentation on the Wikipedia has seriously worsened in recent days, we see no alternative but to forward this case (including all relevant visible and anonymous contributors) to our legal representatives in both USA and Europe/Germany as well as to the authorities and other corresponded organizations in order to find a remedy to this case.
Donfried has made it very clear that his organization really wants a Wikipedia article and that they believe they are being damaged without one. But the fact that he wants one doesn t mean that Wikipedia s policies mean he should have one. Anonymous editors in Berlin and in unknown locations have made it clear that they really want a Wikipedia article about the ICD that does not include criticism. Not only do Wikipedia s policies and principles not guarantee them this, Wikipedia might be hurt as a project when this happens. The ICD claims to want to foster open dialogue and criticism. I think they sound like a pretty nice group working toward issues I care about personally. I wish them success. But there seems to be a disconnect between their goals and the actions of both their leader and proponents. Because I used my real name and was skeptical about the organization on discussion pages on Wikipedia, I was tracked down and threatened. Donfried insinuated that I was motivated to sabotage his organization and threatened legal action if I do not answer his questions. The timing of his first letter the day after the ICD page was recreated means that I was unwilling to act on my commitment to Wikipedia and its policies. I have no problem with the ICD and I deeply regret being dragged into this whole mess simply because I wanted to improve Wikipedia. That said, Donfried s threat has scared me off from attempts to improve the ICD articles. I suspect I will not edit ICD pages in Wikipedia in the future. The saddest part for me is that I recognize that what is in effect bullying is working. There are currently Wikipedia articles about the ICD in many languages. For several years, ICD has had an article on English Wikipedia. For almost all of that period, that article has consisted entirely of universally positive text, without criticism, and has been written almost entirely by anonymous editors who have only contributed to articles related to the ICD. In terms of the ICD and its article on Wikipedia, I still have hope. I encourage Donfried and his representatives to create accounts on Wikipedia with their full names just like I have. I encourage them to engage in open dialogue in public on the wiki. I encourage them go through deletion review, make any conflicts of interest they have unambiguously clear, and to work with demonstrably non-conflicted editors on Wikipedia to establish notability under Wikipedia s policies. The result still can be an awesome, neutral, article about their organization. I have offered both advice on how to do this and help in that process in the past. I have faith this can happen and I will be thrilled when it does. But the general case still worries me deeply. If I can be scared off by threats like these, anybody can. After all, I have friends at the Wikimedia Foundation, a position at Harvard Law School, and am close friends with many of the world s greatest lawyer-experts on both wikis and cyberlaw. And even I am intimidated into not improving the encyclopedia. I am concerned by what I believe is the more common case where those with skin in the game will fight harder and longer than a random Wikipedian. The fact that it s usually not me on the end of the threat gives me lots of reasons to worry about Wikipedia at a time when its importance and readership continues to grow as its editor-base remains stagnant.
[1] It s a minor mistake but worth pointing out that I am not on the board of Wikipedia ; I am on its advisory board which carries no power or responsbility within the organization. Sometimes, the foundation asks for my advice and I happily give it.

Raphael Geissert: A bashism a week: substrings (dynamic offset and/or length)

Last week I talked about the substring expansion bashism and left writing a portable replacement of dynamic offset and/or length substring expansion as an exercise for the readers.

The following was part of the original blog post, but it was too long to have everything in one blog post. So here is one way to portably replace said code.

Let's consider that you have the file name foo_1.23-1.dsc of a given Debian source package; you could easily find its location under the pool/ directory with the following non-portable code:
file=foo_1.23-1.dsc
echo $ file:0:1 /$ file%%_* /$file

Which can be re-written with the following, portable, code:
file=foo_1.23-1.dsc
echo $ file%$ file#? /$ file%%_* /$file

Now, in the Debian archive source packages with names with the lib prefix are further split, so the code would need to take that into consideration if file is libbar_3.2-1.dsc.

Here's a non-portable way to do it:
file=libbar_3.2-1.dsc
if [ lib = "$ file:0:3 " ]; then
    length=4
else
    length=1
fi

# Note the use of a dynamic length:
echo $ file:0:$length /$ file%%_* /$file

While here's one portable way to do it:
file=libbar_3.2-1.dsc
case "$file" in
    lib*)
        length=4
    ;;
    *)
        length=1
    ;;
esac

length_pattern=
while [ 0 -lt $length ]; do
    length_pattern="$ length_pattern ?"
    length=$(($length-1))
done

echo $ file%$ file#$length_pattern /$ file%%_* /$file

The idea is to compute the number of interrogation marks needed and use them where needed. Here are two functions that can replace substring expansion as long as values are not negative (which are also supported by bash.) 

genpattern()  
    local pat=
    local i="$ 1:-0 "

    while [ 0 -lt $i ]; do
        pat="$ pat ?"
        i=$(($i-1))
    done
    printf %s "$pat"
 

substr()  
    local str="$ 1:- "
    local offset="$ 2:-0 "
    local length="$ 3:-0 "

    if [ 0 -lt $offset ]; then
        str="$ str#$(genpattern $offset) "
        length="$(($ #str  - $length))"
    fi

    printf %s "$ str%$ str#$(genpattern $length) "

Note that it uses local variables to avoid polluting global variables. Local variables are not required by POSIX:2001.

Enough about substrings!

Remember, if you rely on non-standard behaviour or feature make sure you document it and, if feasible, check for it at run-time.

Matthew Garrett: Secure Boot and Restricted Boot.

I gave a presentation at Libreplanet this weekend on the topic of Secure Boot and Restricted Boot. There's a copy of the video here - it should be up on the conference site at some point. It turned out to be excellent timing, in that a group in Spain filed a complaint with the European Commission this morning arguing that Microsoft's imposition of Secure Boot on the x86 client PC market is anticompetitive. I suspect that this is unlikely to succeed (the Commission has already stated that the current implementation appears to conform to EU law), and I fear that it's going to make it harder to fight the real battle we face.

Secure Boot means different things to different people. I think the FSF's definition is a useful one - Secure Boot is any boot validation scheme in which ultimate control is in the hands of the owner of the device, while Restricted Boot is any boot validation scheme in which ultimate control is in the hands of a third party. What Microsoft require for x86 Windows 8 devices falls into the category of Secure Boot - assuming that OEMs conform to Microsoft's requirements, the user must be able to both disable Secure Boot entirely and also leave Secure Boot enabled, but with their own choice of trusted keys and binaries. If the FSF set up a signing service to sign operating systems that met all of their criteria for freeness, Microsoft's requirements would permit an end user to configure their system such that it refused to run non-free software. My system is configured to trust things shipped by Fedora or built locally by me, a decision that I can make because Microsoft require that OEMs support it. Any system that meets Microsoft's requirements is a system that respects the freedom of the computer owner to choose how restrictive their computer's boot policy is.

This isn't to say that it's ideal. The lack of any common UI or key format between hardware vendors makes it difficult for OS vendors to document the steps users must take to assert this freedom. The presence of Microsoft as the only widely trusted key authority leaves people justifiably concerned as to whether Microsoft will be equally aggressive in blacklisting its own products as it will be in blacklisting third party ones. Implementation flaws in a (very) small number of systems have resulted in correctly signed operating systems failing to boot, requiring users to update their firmware before being able to install anything but Windows.

But concentrating on these problems misses the wider point. The x86 market remains one where users are able to run whatever they want, but the x86 market is shrinking. Users are purchasing tablets and other ARM-based ultraportables. Some users are using phones as their primary computing device. In contrast to the x86 market, Microsoft's policies for the ARM market restrict user freedom. Windows Phone and Windows RT devices are required to boot only signed binaries, with no option for the end user to disable the signature validation or install their own keys. While the underlying technology is identical, this differing set of default policies means that Microsoft's ARM implementation is better described as Restricted Boot. The hardware vendors and Microsoft define which software will run on these systems. The owner gets no say.

And, unfortunately, Microsoft aren't alone. Apple, the single biggest vendor in this market, implement effectively identical restrictions. Some Android vendors provide unlockable bootloaders, but others (either through personal preference or at the behest of phone carriers) lock down their platforms. A naive user is likely to end up purchasing a device that will, in the absence of exploited security flaws, refuse to run if any system components are modified. Even in cases where the underlying components are built using free software, there's no guarantee that the user will have the ability to assert any of those freedoms.

Why does this matter? Some of these platforms (notably Windows RT and iOS, but also some Android-based devices) will even refuse to run unsigned applications. Users are unable to write their own software and distribute it to others without agreeing to often onerous restrictions. Users with the misfortune of living in the wrong country may be forbidden from even that opportunity. The vendor may choose to block applications that compete with their own, reducing innovation. The ability to explore and tinker with the components of the system is restricted, making it harder for users to learn how modern operating systems work. If I own a perfectly functional phone that no longer receives vendor updates, I don't even have the option of paying a third party to ensure that I can't be compromised by a malicious website and risk the loss of passwords or financial details. The user is directly harmed by these restrictions.

I won't argue that there are no benefits to curated software ecosystems. I won't even argue against devices shipping with a locked down policy by default. I will strongly argue that the owner of a device should not only have the freedom to choose whether they wish to remain within those locked-down boundaries, but should also have the freedom to impose their own boundaries. There should be no forced choice between freedom and security.

Those who argue against Secure Boot risk depriving us of the freedom to make a personal decision as to who we trust. Those who argue against Secure Boot while ignoring Restricted Boot risk depriving us of even more. The traditional PC market is decreasing in importance. Unless we do anything about it, free software will be limited to a niche group of enthusiasts who've carefully chosen from a small set of devices that respect user freedom. We should have been campaigning against Restricted Boot 10 years ago. Don't delay it even further by fighting against implementations that already respect user freedom.

comment count unavailable comments

26 March 2013

Russ Allbery: remctl 3.4

remctl is the client/server protocol that we use for basically all our middleware connections at Stanford. It uses Kerberos GSS-API for authentication and privacy and is designed to be as simple as possible, just running a command on the server with the provided arguments. We've found this design to be extremely effective and a significant improvement over trying to put web servers everywhere to use something like REST. The primary feature in this release is a new Net::Remctl::Backend Perl module that automates a lot of the work of creating remctl backend scripts written in Perl. Over the years, we've developed a lot of conventions for things like help output and command-line processing, and this rolls those conventions into a module that handles a lot of the work of formatting help output and handling command dispatch. This is just the first cut, but it already supports argument validation, handling standard input, formatting help output, and handling per-option command-line flags. Later versions will feature better integration with Kerberos and hopefully better integration into the new remctl help and summary features. Also in this release are new C APIs contributed by Jeffrey Hutzelman that allow one to start a remctl connection over an existing sockaddr, a list of struct addrinfo results, or an open socket. In the bug fix department, this release removes all the prototypes from Net::Remctl functions following current Perl best practices (prototypes cause weird context issues and behave in surprising ways), and rejects the empty command to remctl_command rather than trying to malloc 0 bytes. You can get the latest version from the remctl distribution page.

25 March 2013

Russ Allbery: kadmin-remctl 3.3

kadmin-remctl is a remctl wrapper around the kadmin protocol that we use at Stanford to provide an easier API to Kerberos administration (for Java applications in particular, but also somewhat for humans) and to use a finer-grained ACL than Kerberos kadmin provides. This package is still kind of a mess, with some Stanford-specific bits and a whole lot of messy Perl code, but we had some serious bugs that I needed to deal with. So this is mostly a bug-fix release without the much-delayed cleanup. (More on that in a moment.) The primary problem this release addresses is a very annoying network problem that we've been having at Stanford. Some interaction between our new OpenFlow fabric and our current firewalls causes some number of TCP connections to just be dropped under heavy load. Since we do a lot of kadmin queries (although we're working on various caching strategies to cut down on that), kadmin sees this a lot, which results in failures that cause user-noticable problems in various applications. After some experimentation, though, we found that most of the failures were on the original connection, so catching connection failure and retrying works around most of the problem (at the cost of a delay). That was only the beginning of the odyssey, however. When I added the retry code, I found that we were improperly configuring Heimdal::Kadm5 to actually throw exceptions. And then, even after fixing that, I found that error messages were still printed to standard error. It turns out that the underlying Heimdal libraries print warnings to standard error by default unless the application directs the log messages somewhere else, and Heimdal::Kadm5 doesn't have a look available to do that. So I ended up adding a bunch of ugly code to close standard error while doing the initial connection. This release also cleans up a bunch of the error handling, including going back to the behavior that the code always should have had: check_passwd returns a non-zero status if the password is rejected. I lost an argument with a co-worker about that years ago, but they've since left and I still think that's the right thing to do. The new code also uses IPC::Run to run commands, which makes for much shorter and saner code, and improves error reporting when trying to change the password of a disabled account with Heimdal's kpasswd. It also strips whitespace from the username in passwd_change. At some point, I'm going to rewrite this software completely, but what that really needs is a much saner underlying Kerberos module. One of the problems with all existing Kerberos Perl modules is that they all use various hacks to try to build with different Kerberos libraries within the ExtUtils::MakeMaker framework, which is totally inadequate. (That's also why there aren't any Perl modules linked with the server version of the kadmin libraries, so they're all painfully slow and require a keytab even when running directly on the KDC.) I think the answer is a brand new Perl module that incorporates my portability layer on top of Kerberos libraries and a configure script that does proper probing for Kerberos library functionality, and then integrates that Autoconf probing into a Module::Build infrastructure. There are some bits on CPAN that one could start with, but not all the proper glue, so this is a bit of a project. I also have to figure out how to link different extensions in the project with different libraries, since having separate distributions for the server and client kadmin modules (which share 95% of their code) is dumb, but they have to be separately linked with the appropriate library. Sometime, when I have some time, I'll do that, and then rebuild a remctl interface on top of that infrastructure with a much better protocol. In the meantime, you can get the latest version of kadmin-remctl from the kadmin-remctl distribution page.

21 March 2013

Jonas Smedegaard: What is FreedomBox? And when can I have one?

On wednesday 7. of November I gave a talk at EPFSUG in Brussels about FreedomBox. EPFSUG is an interest group of Free Software users working inside the European Parliament. The FreedomBox is a project to help non-geeks care about their personal privacy when online, same ways as geeks have practiced for a decade or more. The goal is a small, cheap physical box looking like and operating like an internet gateway or wifi router many are accustomed to nowadays - but with three additions: "Privacy" is commonly mistaken as "secrecy": Privacy is to keep you in control of your information - you can then use that control to keep secrets from others or to share with others as you like. Talking about personal privacy in the European Parliament can seem a bit of a stretch. It is not a home but a (huge) office space, where it is perhaps less obvious that you should treat some information as personal, or that you are even allowed to do so. Try watch the videos, and if you have questions then please don't hesitate to get in touch with me about them. At the EPFSUG meeting was also a presentation by an emplyee who had succesfully installed and used Free Software at the internal network of the Parliament, but then later told that it wasn't allowed - because the IT staff need to be in control of your computer activities at the place! The head of IT services at the Parliament attended the meeting and gave a short improvised talk at the end, expressing positive interest in behalf of the established IT services at the place towards our grassroots activities - and even praising explicitly EPFSUG as being the ideal place for all EU citizens to ask questions about Free Software in relation to the European Parliament. See for yourself in the video of Giancarlo! Thanks a lot especially to Erik Josefsson for making this event reality! Slides, sources and videos from the event.

19 March 2013

Michal Čihař: Libravatar support in Weblate

For some time, Weblate was showing avatars for users. Just as I've discovered Libravatar - free and federated alternative to Gravatar, I thought it would be better replacement. Quickly looking at their website, it seems that they transparently provide all avatars from Gravatar as well, so the migration seems to be pretty much painless. Basic replacement to use their server is just matter of changing base URLs, however to support federated behavior, you have to install pyLibravatar. Weblate in Git now supports both these ways. While implementing the client side, I did also setup my own instance of Surrogator to provide avatars for some of my domains. Surprisingly this worked fine without problems, but let's see how much used this will actually become.

Filed under: English Phpmyadmin Weblate 2 comments Flattr this!

Gergely Nagy: Rebuttals, or the lack of thereof

It is customary during Debian Project Leader elections to write a rebuttal of the other candidate's platform. However, this is very hard to do this year, as there are a lot of overlaps between platforms, goals and ideas this time around. It is hard to rebute another platform, without also rebuting one's own. So instead of trying to highlight where I disagree (there's very little of that, if at all), I'd like to point out where I found each platform lacking.Mind you, these notes are primarily based off of the platforms, because I'm sadly still behind on my debian-vote@ reading list. Furthermore, due to the high level of similarity between platforms, I will not always single out one or the other, but mention what I found lacking, or troublesome in general instead. So unless stated otherwise, my thoughts below apply to both Moray's and Lucas' platforms.The thing I miss most from both platforms is the curious lack of plans about non-packaging project members (something which I strongly emphasize in my own platform). Similarly, even though Moray does mention encouraging more local events, I find those few paragraphs a little bit lacking; and Lucas does not even mention local teams in his platform, even though they are a very, very important part of the wider Debian community, and a terrific tool in our asset. They can further our reach by long miles! Considering their importance, little's said about either local teams or local events, even when the candidate does explicitly recognise their importance.On the other hand, Lucas writes about innovation within Debian (which I found lacking in Moray's platform) - but while he does bring up the topic, I sadly found that section of his platform vague, a mistake I made last year myself, and which I tried to avoid this time, by explicitly listing a few of my ideas on how I'd imagine inspiring & motivating people (which, in turn, would foster innovation within the project).From what I see - and I may be wrong here, and I am obviously biased - both platforms are reasonably explicit when it comes to easier topics, things that we know to work, and which we'd all continue doing in the very same spirit as our predecessor. But when it comes to experimenting, to new ideas, I found both platforms too vague. There are exceptions, of course, Moray's intern idea for example is sufficiently well explained, as far as it is possible within the limits of a platform. I found no similar thing in Lucas' platform, however, most of the ideas he explained were technical things, and those are the areas where we're fairly good at, where we already have a reasonably chance of improving - and thus, an easier topic. I was suprised by Lucas' platform, because even though he lists the need for experimentation important, the lack of truly experimental ideas was lacking - I found more in Moray's platform.All in all, I found the discussions on debian-vote@ much more useful so far, than the platforms, a lot of things were asked and answered there that the platforms did not touch.

17 March 2013

Andrew Cater: Kali Linux - Debian derivative for penetration testing/security

New Debian-based security distribution. The company behind BackTrack (Offensive Security) have just launched Kali Linux which will be tightly coupled to Debian and use Debian security updates, for example.

See also: http://www.kali.org/ and Raphael Hertzog's more informative post on Planet Debian.

16 March 2013

Stefano Zacchiroli: bits from the DPL for February 2013 and a half

Dear project members, here's another report of DPL activities, this time for a period longer than usual (February + 1st week of March), so that the next one will be at the very end of the current DPL term. Highlights Appointments DPL helpers Two more DPL helpers IRC meetings have happened, minutes and logs of both are available. Assets Events Past At the beginning of February, I've attended FOSDEM 2013, together with many other Debian people. I didn't have any specific talk this year, but it's been a chance to talk F2F about several ongoing issues (see logs), and help mediating in some conflicts. I've also accepted the invitation to participate in the GNOME Advisory Board meeting, together with Laurent Bigonville of our GNOME team. No report of that has been prepared as of yet (sorry about that), but we have both reported "live" to the rest of the team on IRC. Future Miscellaneous A couple of months ago I've mentioned that I had filed an application, as Debian representative, to participate in a working table to define software procurement rules for the Italian public administration. Good news: my application has been accepted, together with those of other well-known FOSS communities and organizations (e.g. KDE, FSFE). I'll keep you posted of how it goes. Let's go back to elect a new DPL and release Wheezy now,
Cheers.
PS the day-to-day activity logs for February and March 2013 are available at the usual place master:/srv/leader/news/bits-from-the-DPL.txt.20130 2,3

15 March 2013

Hideki Yamane: pbuilder/cowbuilder and piuparts failed

Today, I've failed to create pbuilder's base.tgz and cowbuilder's base.cow, and log says as below

Get:1 http://ftp.jp.debian.org sid Release.gpg [836 B]
Hit http://ftp.jp.debian.org sid Release
Ign http://ftp.jp.debian.org sid Release
Hit http://ftp.jp.debian.org sid/main amd64 Packages/DiffIndex
Hit http://ftp.jp.debian.org sid/main Translation-en
Fetched 836 B in 0s (3011 B/s)
Reading package lists...
W: GPG error: http://ftp.jp.debian.org sid Release: The following signatures were invalid: BADSIG AED4B06F473041FA Debian Archive Automatic Signing Key (6.0/squeeze) <ftpmaster debian.org="">
I: Obtaining the cached apt archive contents</ftpmaster>
"The following signatures were invalid: BADSIG" ?

apt (0.9.7.8) unstable; urgency=criticial
 
  * SECURITY UPDATE: InRelease verification bypass
    - CVE-2013-1051
   
  [ David Kalnischk ]
  * apt-pkg/deb/debmetaindex.cc,
    test/integration/test-bug-595691-empty-and-broken-archive-files,
    test/integration/test-releasefile-verification:
    - disable InRelease downloading until the verification issue is
      fixed, thanks to Ansgar Burchardt for finding the flaw
Maybe this cause it (just wild guess).

And piuparts also failed.


8m16.5s DEBUG: Starting command: ['chroot', '/tmp/tmpivurbJ', 'eatmydata', 'umount', '/proc']
8m16.5s DUMP: 
  umount: /proc: device is busy.
          (In some cases useful info about processes that use
           the device is found by lsof(8) or fuser(1))
8m16.5s DEBUG: Command failed (status=1), but ignoring error: ['chroot', '/tmp/tmpivurbJ', 'eatmydata', 'umount', '/proc']
8m16.5s DEBUG: Starting command: ['rm', '-rf', '--one-file-system', '/tmp/tmpivurbJ']
8m16.6s DUMP: 
  rm: skipping '/tmp/tmpivurbJ/proc', since it's on a different device
8m16.6s ERROR: Command failed (status=1): ['rm', '-rf', '--one-file-system', '/tmp/tmpivurbJ']
  rm: skipping '/tmp/tmpivurbJ/proc', since it's on a different device
  
8m16.7s DEBUG: Starting command: ['chroot', '/tmp/tmpivurbJ', 'umount', '/proc']
8m16.8s DUMP: 
  chroot: failed to run command 'umount': No such file or directory


It mounts binfmt_misc mount under /proc and try to umount /proc without umount it again, again and again... then failed. What causes this annoying behavior?

Please let me know.

Russ Allbery: C TAP Harness 2.0

This package is my pure C test harness that understands the TAP protocol. It's akin to prove, except written in C. It also provides a TAP library for C and shell scripts, which is useful even if you don't want to use the harness. I was going to just do a small release with no major changes, just some cleanup and merging of changes from elsewhere, but I do hate making tiny releases and I'd been meaning to get back to doing some larger development for a while. So this release has several major changes, including one backward-incompatible one. The backward-incompatible change is that, to specify a list of tests to run, one now needs to use the -l option to runtests. That's because the default interpretation of the command line is now that it is a list of tests to run and summarize. This is more consistent with other test harnesses, such as prove. Also in this release, runtests will look for the test name as given if it can't find it by appending -t or .t, which allows one to pass the actual name of the test executable on the command line (with or without the -o option). This makes command-line completion less annoying. There are a few other minor fixes to display: output is flushed after each test executable completes even if the output isn't a terminal, which produces better display when teeing to a log file, and partial status of tests that use lazy plans now show the total test count as "?" to match the behavior of prove. I also fixed a few errors when displaying a test abort to a terminal. You can get the latest version from the C TAP Harness distribution page.

14 March 2013

Iustin Pop: Types as control flow constructs

A bug I've recently seen in production code gave me the idea for this blog post. Probably smarter people already wrote better things on this topic, so this is mostly for myself, to better summarise my own thoughts. Corrections are welcome, please leave a comment! Let's say we have a somewhat standard API in Python or C++: Signalling failures to initialise the object can be done in two ways: either by raising an exception, or by returning a null/None result. There are advantages and disadvantages to both: The None model can create latent bugs, for example in the following code: 
ok = True
for arg in input_list:
  t = init_t(arg)
  if t is None:
    ok = False
    continue
  t.process()
return ok
The presence of the continue statement there is critical. Removing it, or moving it after some other statements which work with t will result in a bug. So using value-based returns forces us to introduce (manually) control points, without having the possibility to validate the model by the compiler (e.g. in C++). So it would seem that this kind of latent bugs pushes us to use the exception model, with its drawbacks. Let's look at how this interface would be implemented in (IMHO) idiomatic Haskell (where the a and b types represent the input and output types): 
initT :: a -> Maybe T
processT :: T -> b
my_fn =
  
 case initT arg of
   Nothing -> -- handle failure
   Just v -> processT v
Yes, this can be written better, but it's beside the main point. The main point is that by introducing a wrapper type around our main type (T), we are forced via the type system to handle the failure case. We can't simply pass the result of initT to a function which accepts T, because it won't type check. And, no matter what we do with the result value, there are no exceptions involved here, so we only have to think about types/values, and not control flow changes. In effect, types become automatically-validated control-flow instructions. Or so it looks to me . So using types properly, we can avoid the exception-vs-return-value debate, and have all advantages without the disadvantages of either. If that is the case, why isn't this technique used more in other languages? At least in statically typed languages, it would be possible to implement it (I believe), via a C++ template, for example. In Python, you can't actually apply it, as there's no static way of enforcing the correct decapsulation. I was very saddened to see that Google's Go language, which is quite recent, has many examples where initialisation functions return a tuple err, value = , separating the actual value from the error, making it not safer than Python. It might be that polymorphic types are not as easy to work with, or it might be the lack of pattern matching. In any case, I don't think this was the last time I've seen a null pointer dereference (or the equivalent AttributeError: 'NoneType' object has no attribute ). Sadly You can even go further in Haskell and introduce more control flow structure via wrapper types. Please bear another contrived example: an HTML form that gets some input data from the user, validates it, saves it to the database, and echoes it back to the user. Without types, you would have to perform these steps manually, and ensure they are kept in the correct order when doing future modifications. With types, you only have to design the types correctly and export only smart contructors (but not the plain ones): 
module Foo ( ValidatedValue
           , validateValue
           , RecordId
           , CommittedValue
           , commitValue
           , buildFeedbackForm
           ) where
data ValidatedValue a = ValidatedValue a
validateValue :: a -> Maybe (ValidatedValue a)
data RecordId =  
data CommittedValue a = CommittedValue a RecordId
commitValue :: ValidatedValue a -> ComittedValue a
buildFeedbackForm :: CommittedValue a -> HTMLDoc
From these types, it follows more or less that the only correct workflow is:
  1. get a value from the user
  2. validate it
  3. commit it, getting a transaction/record ID
  4. send the response to the user
In other words: 
handleFeedbackForm = buildFeedbackForm . commitValue . validateValue
There are still issues here, e.g. the type a is completely hidden behind the wrapper types, and we can't recover some basic properties (even if we use newtype, unless we use GeneralizedNewtypeDeriving). But it does offer a way to improve control flow safety. And that is my 0.02 currency unit for today.

13 March 2013

Raphael Geissert: A bashism a week: assigning to variables and special built-ins

Assigning a value to a variable when executing a command is a way to populate the command's environment, without the variable assignment persisting after the command completes. This is not true, however, when a special built-in is the command being executed.

POSIX:2001 states that "Variable assignments specified with special built-in utilities remain in effect after the built-in completes".

Not only this is tricky because it depends on whether a utility is a special built-in or not, but the bash interpreter does not respect that behaviour of the POSIX standard. That is, special built-ins are not so "special" to the bash interpreter.

This leaves two things to take into account when assigning to a variable when executing a command: whether the command is a special built-in, and whether bash is interpreting the script.

Now, the list of special built-ins is rather short and it would be a bit unusual to perform variable assignments when calling them, except for some cases: "exec", "eval", "." (dot), and ":" (colon).

It is important to note that ":" and "true" differ in this regard; the former is a special built-in, the latter is just a utility. Watch out for this kind of differences when using ":" or "true" to nullify a command. E.g.

Compare
$ dash -c '
method=sed
# some condition or user setting ends up making:
method=true
# later:
foo=bar $method
echo foo: $foo'
foo:
To (redacted for brevity):
$ dash -c '
method=:
foo=bar $method
echo foo: $foo'
foo: bar

12 March 2013

Richard Hartmann: Gitify your life

These Open Source Days were kind of a mixed experience. The social part Things started off quite nicely; the speaker dinner was very pleasant and the people at my table interesting to talk with. The way back to the hotel and breakfast on Saturday were even more interesting; talking with Amelia Andersdotter, a member of the European Parliament for the Swedish Pirate Party, about anti-trust measures, privacy protection, lobbyists, and creating the right incentives for companies to actually protect user data was fun and engaging. My own claim to fame is that I told her that while Microsoft is willing to sign UEFI payloads for x86, they refuse to do so for ARM which she didn't know. From my understanding, their interpretation is that any anti-trust rulings apply to PCs only, and not general computing devices like tablets. It will be interesting to see how that plays out in the long run, especially once ARM processors start appearing in laptops and desktops as main CPUs where Microsoft clearly still has a near monopoly. It was fascinating to discuss all those issues with someone who's actively involved with legislation, mainly because her style of talking was markedly different from what I am used to. One point which she stressed repeatedly is that the most important skill within the context of political work in the EU is to be able to track down, read, and then reference all the content that's being produced in Brussels and Strasbourg. Maybe that tidbit is useful to people who want to try themselves at political activism. She nicely, and maybe unknowingly, emphasized that point by always referring to specific papers, directives, and studies after making any point. Statement of opinion, reference to supporting document. Statement of opinion, reference to supporting document. Statement of opinion, reference to supporting document. Rinse, repeat. It's no secret that I'm continuously disappointed by the political process in general and politicians in particular. Yet, I have to say that this encounter had a lasting and positive impression on me. Amelia has an incredible amount of energy and genuinely cares about the issues which too many other politicians ignore completely. It's good to know someone like this is on our side. If you're Swedish, please inform yourself about her and consider voting in her favour. Other than that, I met several other people, new and old. This year's OSD felt a lot more relaxed and there were more informal and spontaneous conversations going on in hallways and at tables. Even though I failed to have any intense discussion of Mercurial vs git this year, all those conversation were still really interesting. The "workshop" I already knew that the room assignment for my workshop on Saturday afternoon would be a last-minute thing. What I didn't expect was that the final confirmation by Copenhagen Business School would be given on Friday evening. Mixed with a very cumbersome process to update the slides of the CBS-owned and -operated information beamer in the lobby, this resulted in the official schedule showing my room as "Workshop" instead of "SP 214". The official schedule repeatedly stated that people had to pre-register for the workshop which was a left-over from last year where they actually had to plan the amount of hardware kits needed; the registration itself was was very well hidden behind a rather generic link you saw exactly once after purchasing your ticket. Add to this that my room was literally in the farthest corner of the building and on the top floor and things start to turn sour real quick. All other rooms were either close to the main entrance or on the ground floor. Helpers at the front desk had a lot of people asking where my room was and gave out directions, yet... ...the grand total of six people who arrived were all telling me how they couldn't find the room and almost gave up on finding it. As more people asked for directions than turned up at the workshop, this means that even with directions, the room was really hard to find. Myself, it took me two attempts to find the room after which I was led there by one of the helpers who wasn't completely sure we were going in the right direction, himself. Still, I managed to get everyone who had a laptop and access to a UNIX-based system set up with working mr and vcsh repositories they can work from as well as a demo git-annex repository to play with. Sadly, not everyone had a laptop, or access to UNIX, or the tools installed, or even an idea where to install them from, even though I had noted these "requirements" in the description. Next time I do this workshop, I will need to make sure to include information on what distributions have the relevant packages and how to find and install the software on Max OS X in the workshop description. Feedback from the very few people who actually managed to find me was very good and one of the attendants asked if I would be willing to hold the workshop again at another conference in Denmark. I am still awaiting details on this, but I would obviously enjoy having the chance to reach more people. At least I fared better than the guys who did the workshop on "Binary exploitation 101" in yet another room. Two guys held a workshop with a total attendance of... one... I know the organizers of OSD were really unhappy with how this turned out and I believe them when they say that something like this will never happen again. It's the second year that this team is running the conference, running conferences is an incredible amount of work, and things were smoother this year than last year. Iterate and improve, etc. The talk Saturday evening, Alex offered me the chance to hold a reparation talk on Sunday morning to compensate for the botched workshop. Said talk was scheduled for the first slot and not really announced (but the schedule on both www and the beamer was changed over night). Additionally, this was merely an updated and extended version of the talk which I already held at OSD 2012 so at least part of the visitors knew most of what I had to say, already. To be honest I didn't expect too much, but more than twenty people turned up which is a very decent turnout, all things considered. Same as with the workshop, feedback was extremely positive and I was told afterwards by Bryan that several people mentioned my talk as one of the highlights, or even the highlight, of OSD when he was talking to them. To make things even better this talk has, finally, been videotaped; I will update this blog with a link as soon as the video is online.

10 March 2013

Gergely Nagy: DPL platform

Last year, I invited you all to walk the plank with me, to follow me on a road, a road that even I was unsure where it led. I had a vision, I had ideas, but I had little to show for a plan. Even worse, some of my ideas, some of the things I've thought I knew, turned out to be wrong this past year. While my original plan was to pursue the goals I set out in my previous platform, I did none of that, and before I get to the meat of the platform, I owe you an explanation why: After walking the plank last year, I spent some time swimming to the shore, determined that no matter what, I will get things done. But how shall I do that? And when, with whom? Those questions needed an answer, therefore I took on a coat of an ordinary hacker, and started a journey to find the answers.That is how the past year's been spent: looking and learning. I was - and still am - following every major Debian list, I'm on IRC, absorbing all the information that fits into my tiny mind. I see now, that there's a lot more to be done than I thought last year, that some of the assumptions I made then, were wrong. And having spent a long time learning, it is my hope, that I did learn, and that this time, the plans I came up with will serve the same goal I had last year, better.That goal is as simple as growing Debian from a community of mostly technical folk to something much bigger, a community of enthusiastic, bright-eyed people who all want to do their part of making the world a better place. Big words, big task.The problemOne of the most fundamental problems with the project is that it is old, and hasn't grown up yet. We're still very focused on technical merit - don't get me wrong, that is important too, but there's much more we can do than that! We have tremendous amounts of talent, but pretty much every team is suffering from lack of people, and we seem to be slow and ineffective in recruiting new contributors (either from within the project, or outside of it), and even worse at keeping them motivated. This is not new, I worried about it last year, and I'm not the only one to see things this way, respectable members of the project expressed their pessimism publicly too.This is our biggest issue. We can be excellent at the technical level, but not being able to keep people motivated, failing to find new contributors is hurting us more and more, and it is my firm belief that without quick improvement in these areas, it won't be just me virtually walking the plank.On the flip side, we have every tool in our possession to steer ourselves into a better, brighter direction. We have excellent hackers, we have people who are terrific at communication, we have people who inspire others (just see the pattern on the dpl-game!), we have everything we need. Read on then, dear reader, and see my plan!Dancing on a tightropeTo tread on a path we have not taken so consciously before, we need to have a vision to strive for. A vision for the project as a whole, and a vision for ourselves, to see where we'd like to be a few years from now.As Stefano put it so well in his platform last year, Debian is a very unique member of the larger Free Software ecosystem - go and read his description, it can hardly be written better. This is the same vision I share, where Debian grows up to be much more than hundreds of talented people putting together a distribution. I want to see Debian becoming more than that, something that attracts not only the technical geniuses, but those who wield the power of speech and inspiration too, for these traits rarely go together.I envision a project, where people recognise each other's strengths, and make use of these strengths; where we can - and will - turn to each other, and work closely together, to achieve such good balance of skills of all kinds, as we already achieved in creating and maintaining the distribution itself.Let's dance!But alas, a vision is useless without a plan. That is a mistake I made last year, one that I will not repeat again. The approaches I choose may seem strange, or weird at first, but rest assured, they were carefully considered, and most of them tried out in a smaller setting. CommunicationWhat I've seen and experienced over the past year (and thinking back, far more than that - I just didn't know where to put the experience at that time) is that written communication is rarely the best medium to reach out to people. In this age we live in, written text reaches those who were already curious, it rarely reaches those who're still waiting for enlightenment. And even if it does, more often than not, there is no direct channel between the writer and the reader, which hurts not only the latency of the communication, but the quality too. Not to mention, that - however sad it may sound - the recipient is often too shy to engage.What does help, is face to face time. Events, where we can reach out to people, and receive immediate feedback, where we can notice the shy ones, and help them. Where we can immediately adapt to expectations.For this reason, I feel it necessary to encourage and help local teams in every possible way, to organise hackfests, code retreats, and all kinds of events that they feel will attract the bright-eyed youth (be them 20 or 80, it's the spirit that counts) we so direly need. These events need not be technical at all: a session about Debian publicity, about representing the project in press or on conferences is just as useful (if not more). Along with hackfests, we could - and should - have events focused on improving communication, for that is just as important.With strong focus placed on non-technical matters, I believe we can step on a path of rapid progress: my experience shows that techies and non-techies can inspire one another very well, much more than two techies would. With a focus on areas we are lacking in, we are indirectly improving those we are already good at. Inspiration & motivationBut finding new contributors, be them technically-sawwy or not is only part of the task, and the easier one too. It is much harder to keep people interested and motivated over a long period of time, than to find them once.On one hand, I would like to rely on those members of our community, who we all hold in high esteem (including, but not limited to those mentioned during the dpl-game). They know what keeps them going, they know what drives them away, we need to learn from that. I'd like to hear them speak more often, on conferences, if possible, as they are, I believe, a much more credible source than any elected official, at least in some situations.I'd also like to hear everyone's voice, from old members to the newest ones, I'd like to read - and if so need be, conduct - interviews with people who recently became Debian Developers, to hear about their experience, to learn what they think is good or bad, because a fresh view is just as important as a decade long experience.On the other hand, we need to act on the issues discovered too. We already know that quite a lot of trouble stems from lack of time, overburden and eventually burn out. We know the solution to this: recruit more people, and utilise the resources we do have better. Part of that can be solved by expanding our numbers, but for the other part, we may need deeper changes. What those may be, I cannot tell yet, there simply wasn't enough time to dig deeper, and I'm likely not the best person to undertake the task, either. I am, however, confident that we'll discover the solution as we travel along.The first step is gaining knowledge, without this, no further plans can be made. RecruitmentI already touched the recruitment part earlier, at least the how of the approach. What I'd like to expand on is the why, in particular: why encourage recruiting non-packaging contributors, when we are building a distribution, and we're lacking packaging manpower too?Let me answer those questions separately.First of all, I still like to think - but perhaps I am wrong - that we have an unbelievable amount of packaging manpower, which we fail to use to its full extent. We can use more, we always will be able to. But it also helps if we manage to put our existing resources to better use.And that is where non-packaging contributions come in. If we improve our skills in areas we're lacking in, that in turn, will improve the general quality of the project, which in turns improves morale, and a vibrant, well functioning, welcoming community attracts even more people, and keeps us motivated in the long run too.Mouse on the tightropeWith all that out, you may be wondering who I may be, and why you have not seen my name over the mailing lists in recent years. I'm not a terribly vocal person anymore, not a hot-headed teen I used to be some ten years ago. Most who know me, know me by my online nick name, algernon, and in most cases, I prefer that in real life too, for silly reasons I'd rather not dive into (but the primary one being that people have a hard time correctly pronouncing my name, and I have a hard time recognising my name unless pronounced the way I learnt it). Nevertheless, my name is Gergely Nagy, and I'm a recovering hacker, who intends to become a bachelor of (hungarian) arts, by majoring in Hungarian grammar & literature sometime in the next three to five years.Over the course of the past year, I also strived to observe and learn from exceptional leaders at my day job, because there, I have the advantage of meeting with them face to face, to interact with them on a daily basis. Fortunately for me, there are people among them from whom I learned a great deal about not only motivation, but about behind-the-scenes politics too - both a valuable lesson.And while I still have a tremendous amount of things to learn, I like to believe that I can do that. I know I can learn technical things extremely quickly (my day job depends on that, I would be out of a job if I failed at that), I'm positive that I can do the same in other areas as well, at least as long as it doesn't involve extreme amounts of economics. That proved to be a major barrier in the past.But back to the more important point: my organising, presentation and leading skills. I'm a co-organiser of the Budapest Clojure User Group meetups, I started to encourage GSoC participation at my day job last year (we participated thanks to OpenSUSE who gave us a slot), and I'm driving it this year, aiming to become a mentoring organisation of our own. I'm doing regular talks and presentations, participate in code retreats, meetups and other similar events where I can both practice and learn the art of interacting with people, and the fine art of organising events.

Gergely Nagy: DPL platform

Last year, I invited you all to walk the plank with me, to follow me on a road, a road that even I was unsure where it led. I had a vision, I had ideas, but I had little to show for a plan. Even worse, some of my ideas, some of the things I've thought I knew, turned out to be wrong this past year. While my original plan was to pursue the goals I set out in my previous platform, I did none of that, and before I get to the meat of the platform, I owe you an explanation why: After walking the plank last year, I spent some time swimming to the shore, determined that no matter what, I will get things done. But how shall I do that? And when, with whom? Those questions needed an answer, therefore I took on a coat of an ordinary hacker, and started a journey to find the answers.That is how the past year's been spent: looking and learning. I was - and still am - following every major Debian list, I'm on IRC, absorbing all the information that fits into my tiny mind. I see now, that there's a lot more to be done than I thought last year, that some of the assumptions I made then, were wrong. And having spent a long time learning, it is my hope, that I did learn, and that this time, the plans I came up with will serve the same goal I had last year, better.That goal is as simple as growing Debian from a community of mostly technical folk to something much bigger, a community of enthusiastic, bright-eyed people who all want to do their part of making the world a better place. Big words, big task.The problemOne of the most fundamental problems with the project is that it is old, and hasn't grown up yet. We're still very focused on technical merit - don't get me wrong, that is important too, but there's much more we can do than that! We have tremendous amounts of talent, but pretty much every team is suffering from lack of people, and we seem to be slow and ineffective in recruiting new contributors (either from within the project, or outside of it), and even worse at keeping them motivated. This is not new, I worried about it last year, and I'm not the only one to see things this way, respectable members of the project expressed their pessimism publicly too.This is our biggest issue. We can be excellent at the technical level, but not being able to keep people motivated, failing to find new contributors is hurting us more and more, and it is my firm belief that without quick improvement in these areas, it won't be just me virtually walking the plank.On the flip side, we have every tool in our possession to steer ourselves into a better, brighter direction. We have excellent hackers, we have people who are terrific at communication, we have people who inspire others (just see the pattern on the dpl-game!), we have everything we need. Read on then, dear reader, and see my plan!Dancing on a tightropeTo tread on a path we have not taken so consciously before, we need to have a vision to strive for. A vision for the project as a whole, and a vision for ourselves, to see where we'd like to be a few years from now.As Stefano put it so well in his platform last year, Debian is a very unique member of the larger Free Software ecosystem - go and read his description, it can hardly be written better. This is the same vision I share, where Debian grows up to be much more than hundreds of talented people putting together a distribution. I want to see Debian becoming more than that, something that attracts not only the technical geniuses, but those who wield the power of speech and inspiration too, for these traits rarely go together.I envision a project, where people recognise each other's strengths, and make use of these strengths; where we can - and will - turn to each other, and work closely together, to achieve such good balance of skills of all kinds, as we already achieved in creating and maintaining the distribution itself.Let's dance!But alas, a vision is useless without a plan. That is a mistake I made last year, one that I will not repeat again. The approaches I choose may seem strange, or weird at first, but rest assured, they were carefully considered, and most of them tried out in a smaller setting. CommunicationWhat I've seen and experienced over the past year (and thinking back, far more than that - I just didn't know where to put the experience at that time) is that written communication is rarely the best medium to reach out to people. In this age we live in, written text reaches those who were already curious, it rarely reaches those who're still waiting for enlightenment. And even if it does, more often than not, there is no direct channel between the writer and the reader, which hurts not only the latency of the communication, but the quality too. Not to mention, that - however sad it may sound - the recipient is often too shy to engage.What does help, is face to face time. Events, where we can reach out to people, and receive immediate feedback, where we can notice the shy ones, and help them. Where we can immediately adapt to expectations.For this reason, I feel it necessary to encourage and help local teams in every possible way, to organise hackfests, code retreats, and all kinds of events that they feel will attract the bright-eyed youth (be them 20 or 80, it's the spirit that counts) we so direly need. These events need not be technical at all: a session about Debian publicity, about representing the project in press or on conferences is just as useful (if not more). Along with hackfests, we could - and should - have events focused on improving communication, for that is just as important.With strong focus placed on non-technical matters, I believe we can step on a path of rapid progress: my experience shows that techies and non-techies can inspire one another very well, much more than two techies would. With a focus on areas we are lacking in, we are indirectly improving those we are already good at. Inspiration & motivationBut finding new contributors, be them technically-sawwy or not is only part of the task, and the easier one too. It is much harder to keep people interested and motivated over a long period of time, than to find them once.On one hand, I would like to rely on those members of our community, who we all hold in high esteem (including, but not limited to those mentioned during the dpl-game). They know what keeps them going, they know what drives them away, we need to learn from that. I'd like to hear them speak more often, on conferences, if possible, as they are, I believe, a much more credible source than any elected official, at least in some situations.I'd also like to hear everyone's voice, from old members to the newest ones, I'd like to read - and if so need be, conduct - interviews with people who recently became Debian Developers, to hear about their experience, to learn what they think is good or bad, because a fresh view is just as important as a decade long experience.On the other hand, we need to act on the issues discovered too. We already know that quite a lot of trouble stems from lack of time, overburden and eventually burn out. We know the solution to this: recruit more people, and utilise the resources we do have better. Part of that can be solved by expanding our numbers, but for the other part, we may need deeper changes. What those may be, I cannot tell yet, there simply wasn't enough time to dig deeper, and I'm likely not the best person to undertake the task, either. I am, however, confident that we'll discover the solution as we travel along.The first step is gaining knowledge, without this, no further plans can be made. RecruitmentI already touched the recruitment part earlier, at least the how of the approach. What I'd like to expand on is the why, in particular: why encourage recruiting non-packaging contributors, when we are building a distribution, and we're lacking packaging manpower too?Let me answer those questions separately.First of all, I still like to think - but perhaps I am wrong - that we have an unbelievable amount of packaging manpower, which we fail to use to its full extent. We can use more, we always will be able to. But it also helps if we manage to put our existing resources to better use.And that is where non-packaging contributions come in. If we improve our skills in areas we're lacking in, that in turn, will improve the general quality of the project, which in turns improves morale, and a vibrant, well functioning, welcoming community attracts even more people, and keeps us motivated in the long run too.Mouse on the tightropeWith all that out, you may be wondering who I may be, and why you have not seen my name over the mailing lists in recent years. I'm not a terribly vocal person anymore, not a hot-headed teen I used to be some ten years ago. Most who know me, know me by my online nick name, algernon, and in most cases, I prefer that in real life too, for silly reasons I'd rather not dive into (but the primary one being that people have a hard time correctly pronouncing my name, and I have a hard time recognising my name unless pronounced the way I learnt it). Nevertheless, my name is Gergely Nagy, and I'm a recovering hacker, who intends to become a bachelor of (hungarian) arts, by majoring in Hungarian grammar & literature sometime in the next three to five years.Over the course of the past year, I also strived to observe and learn from exceptional leaders at my day job, because there, I have the advantage of meeting with them face to face, to interact with them on a daily basis. Fortunately for me, there are people among them from whom I learned a great deal about not only motivation, but about behind-the-scenes politics too - both a valuable lesson.And while I still have a tremendous amount of things to learn, I like to believe that I can do that. I know I can learn technical things extremely quickly (my day job depends on that, I would be out of a job if I failed at that), I'm positive that I can do the same in other areas as well, at least as long as it doesn't involve extreme amounts of economics. That proved to be a major barrier in the past.But back to the more important point: my organising, presentation and leading skills. I'm a co-organiser of the Budapest Clojure User Group meetups, I started to encourage GSoC participation at my day job last year (we participated thanks to OpenSUSE who gave us a slot), and I'm driving it this year, aiming to become a mentoring organisation of our own. I'm doing regular talks and presentations, participate in code retreats, meetups and other similar events where I can both practice and learn the art of interacting with people, and the fine art of organising events.

7 March 2013

Vincent Sanders: Man cannot discover new oceans unless he has the courage to lose sight of the shore

Continuing with my whirlwind introduction to NetSurf Development now is the time to start examining the code, how its arranged and how to interact with the existing developers.

The way the NetSurf source is structured is around the idea of the frontends each being a native browser. While this implies that there are nine separate browsers that happen to share a common code base the separation is not quite that well defined.

Each frontend provides the OS entry point (the main() function in a c program) and calls out to standard browser initialisation entry function netsurf_init() and then starts running the browsers main dispatch loop with netsurf_main_loop() when that exits the frontend cleans up with netsurf_exit().

The frontends provide a large selection of functions which are called from the core code. These routines run from running the event scheduler through to rendering graphics and text.
Finding your way aroundThe browsers directory layout is fairly shallow consisting of some Makefiles, the nine frontend directories and eleven others.

The Makefiles are GNU make and represent a pretty straightforward linear build system. We do not use recursive make or autotools. There are plans to use the core buildsystem that all the other components use.

The frontend directories contain the code for the frontend and the makefile fragments to build them which are included by the top level Makefile.

In addition there is:
desktop
This contains the non frontend specific code that actually behaves like a browser. For example desktop/netsurf.c contains the three primary functions we outlined in the introduction. You will also find much of the function and data structures interfaces the frontend must provide. It is unlikely someone new to the project will need to change anything in here (there be dragons) but is an important set of routines.
utils
Here you will find the utility and compatibility interfaces, things like url handling, logging, user messages, base64 handling. These are utility interfaces that do not justify splitting out their functionality to a separate library but are useful everywhere. Changing an interface in here would likely result in a major refactor.
For example one quirk is the logging macro was created before varadic C preprocessor macros were universal so it must be called as LOG(("hello %s world", world_type)) e.g. with double brackets. Fixing this and perhaps improving the logging functionality would be "nice" but the changes would be massive and potentially conflict with ongoing work.
content
This contains all the core code to handle contents i.e. html, css, javascript, images. The handling includes retrieval of the resources from URI , correct caching of the received objects and managing the objects state. It should be explicitly stated that the content handlers are separate and use this core functionality. The actual content handlers to deal with object contents such as the routines to decode image files or render html are elsewhere.
image
This is where the content handlers for the various image types are kept. The majority of these image types (jpeg, png, webp, gif, bmp) use a standard library to perform the actual decode (libjpeg, libpng etc.). One special feature used by most image handlers is that of a decoded image cache which is distinct and separate from the content cache.
The decoded image cache manages decoding of the source images (the jpegs and pngs) into frontend specific "render" bitmaps. For example the gtk frontend keeps the decoded images as a cairo surface ready for immediate plotting.
The cache uses a demand based (when the browser actually displays the image) just in time strategy which has been carefully balanced, with real world input, to reduce the overhead of unnecessary image decoding and processing against memory usage for the render bitmaps.
css
The css content handlers provide for the processing of css source text and use the NetSurf libcss library to process into a bytecode suitable for applying style selection at render time.
javascript
The javascript handlers (strictly speaking this should be named the "script" directory as all types of script are handled here) provide basic functionality to bind a javascript engine to the rest of the browser, principally the Document Object Model (DOM) accessed with libdom. The only engine that currently has bindings is Spidermonkey from the Mozilla foundation.
render
This is the heart of the browser containing the content handler for html (and plain text). This handler deals with:
  • Acquiring the html.
  • Running the base parser as data arrives which generates the DOM and hence DOM events from which additional resource (stylesheets etc.) fetches are started.
  • Deal with script loading
  • Constructing the box model used for layout
  • Performing the Layout and rendering of the document.
Because this module has so many jobs to do it has inevitably become very complex and involved, it is also the principle area of core development . Currently NetSurf lacks a dynamic renderer so changes made by scripts post document load event are not visible. This also has the side effect that the render is only started after the DOM has finished construction and all the resources on a page have completed their fetch which can lead to undesirable display latency.
Docs
Documentation about building and using NetSurf. If anyone wants a place to start improving NetSurf, this is it, it is very incomplete. It must be noted this is not where dynamically generated documentation is found. For the current Doxygen output the best place to look is the most recent build on the Continuous Integration system.
resources
These are runtime resources which are common to all frontends. To be strictly correct they may be the sources which get converted into runtime resources e.g. The FatMessages file which is teh message text for all frontends in all languages, this gets processed at build time into separate files ready.
!NetSurf
This is another resources directory and technically the resources for the riscos frontend. The naming and reliance on this directory are historical. To allow the RISC OS frontend to be run directly from the source directory and an inability of RISC OS to process symbolic links most common runtime resources end up in here and linked to from elsewhere.
test
These are some basic canned test programs and files, principally to test elements of the utils and perform specific exercise of various javascript components.
Getting started
Once a developer has a checked out working build environment and can run the executable for their chosen frontend (and maybe done some web browsing :-) it is time to look at contributing.

If a developer does not have a feature or bug in mind when they begin the best way to get started is often to go bug hunting. The NetSurf bugtacker has lots to choose from unfortunately. Do remember to talk to us (IRC is the best bet if you are bug hunting) about what you are up to but do not be impatient. Some of those bugs are dirty great Shelob types and are not being fixed because even the core developers are stumped!

When first getting going I cannot recommend reading the code enough, this seems to be a skill that many inexperienced open source developers have yet to acquire, especially if they are from a predominately proprietary development background. One wonderful feature of open source software is you get to see all of it, all the elegant nice code and all the "what the hell were they thinking" code too.

One important point is to use your tools well the source is in git, if you learn how to use git well you will gain a skill that is readily portable, not just for NetSurf. And not just revision control tools, learn to use your debugger well and tools like valgrind. Those skills will replay the time spent learning them themselves many times over.

When using git one thing to remember is commit early and often, it does not matter if you have lots of junk commits and dead ends, once you have something viable you can use 
git rebase --interactive
and rewrite it all into a single sensible set of commits. Also please do remember to develop on a branch, or if you did not 
git pull --rebase
is your friend to avoid unnecessary merges.
Playing nicely with othersThe NetSurf community is a small band of unpaid volunteers. On average we manage to collectively put in, perhaps, ten hours a week with the occasional developer weekend where we often manage over twenty hours each.

The result is that developer time is exceptionally valuable, add in a mature codebase with its fair share of technical debt and you get a group who, when they get to work on NetSurf, are incredibly busy. To be fair we are just like every other small open source project in that respect.

What this means to a new contributor is that they should try and do their homework before asking the obvious questions. The documentation is there for a reason and in spite of its obvious shortcomings please read it!

When asking questions it should be noted that currently the majority of active contributors are in the Europe so if you visit the IRC channel or post questions to lists the time difference is something to keep in mind.

I carefully said contributor above and not developer, users trying the CI builds and reporting results are welcome...as long as they report useful bugs to the bug tracker. Simon Tatham has produced an excellent resource on this subject.

Also we are always happy to receive translations to new languages (diff against the FatMessages file would be outstandingly useful but anything is welcome), artwork, documentation. Just recall what I mentioned about busy developers. Surest way to get us to see something is the development mailing list, you will probably get a reply, though I will not promise how fast!

Some of the more common mistakes when interacting with the community are:
In conclusion
Hopefully this has been of some use although I had hoped to cover more and provide deeper insights and advice on the codebase but there is only so much generalisation to be done before it is just easier for the developer to go read the code for themselves.

I look forward to lots of new contributions :-) though I fear this may all end up as more of a crib sheet for next time we do GSOC, time will tell.

6 March 2013

Keith Packard: composite-swap

Composite and Swap Getting it Right Where the author tries to make sure DRI3000 is going to do what we want now and in the future DRI3000 The basic DRI3000 plan seems pretty straightforward:
  1. Have applications allocate buffers full of new window contents, attach pixmap IDs to those buffers and pass them to the X server to get them onto the screen.
  2. Provide a mechanism to let applications know when those pixmaps are idle so that they can reuse them instead of creating new ones for every frame.
  3. Finally, allow the actual presentation of the contents to be scheduled for a suitable time in the future, generally synchronized with the monitor. Let the client know when this has happened in case they want to synchronize themselves to vblank.
The DRI3 extension provides a way to associate pixmap IDs and buffers, and given the MIT-SHM prototype I ve already implemented, I think we can safely mark this part as demonstrably implementable. That leaves us with a smaller problem, that of taking pixmap contents and presenting them on the screen at a suitable time and telling applications about the progress of that activity. In the absence of compositing, I m pretty sure the initial Swap extension design would do this job just fine, and should resolve some of the known DRI2 limitations related to buffer management. And, I think that goal is sufficient motivation to go and implement that. However, I wanted to write up some further ideas to see if the DRI3000 plan can be made to do precisely what we want in a composited world. The Composited Goal To make sure we re all on the same page, here s what I expect from the Swap extension in a composited world:
  1. Application calls Swap with new window pixmap
  2. Compositor hears about the new pixmap and uses that to construct a new screen pixmap
  3. Compositor calls Swap with new screen pixmap
  4. Vertical retrace happens, executing the pending swap operation
  5. Compositor hears about the swap completion for the screen
  6. Application hears about the swap completion for its window
In particular, applications should not hear that their swap operations are complete until the contents appear on the screen. This allows for applications to throttle themselves to the screen rate, either doing double or triple buffering as they choose. I didn t add steps here indicating buffers going idle or being allocated, because I think that should all happen behind the scenes from the application s perspective. Many applications won t care about the swap completion notification either, but some will and so that needs to be visible. Redirected Swaps? Owen Taylor suggested that one way of getting the compositor involved would be to have it somehow redirect Swap operations, much like we do with window management operations today. I think that idea may be a good direction to try:
  1. Application calls Swap with new window pixmap
  2. Swap is redirected to compositor, passing along the new window pixmap
  3. Compositor constructs a new screen pixmap using the new window pixmap
  4. Compositor calls Swap on the screen and the window, passing the new screen pixmap and the new window pixmap. When the screen update occurs, the screen and the window both receive swap completion events.
This has the added benefit that the X server knows when the compositor is expecting window pixmaps to change like this the compositor has to explicitly request Swap redirection. Window Pixmap Names and GEM Buffer Handles One issue that swapping window pixmaps around like this brings up is how to manage existing names for the window pixmap. Right now, applications expect that window pixmaps will only change when the window is resized. If the Swap extension is going to actually replace the window pixmap when running with a suitable compositor, then we need to figure out what the old names will reference. Are there non-compositor applications using NameWindowPixmap that matter to us? How about non-compositor applications using TextureFromPixmap to get a GEM handle for a window pixmap? For now, I m very tempted to just break stuff and see who complains, but knowing what we re breaking might be nice beforehand. Idling Pixmaps When an application is done drawing to a window pixmap and has passed it off to the X server for presentation, we d like for that pixmap to be automatically marked as discardable as soon as possible. This way, when memory is tight, the kernel can come steal those pages for something critical. Of course, applications may not want to let the server mark the pixmap as idle after being used, so a flag to the Swap call would be needed. Ideally, the pixmap would become idle immediately after the pixmap contents have been extracted. In the absence of a compositor, that would probably be when the Swap operation completes. With a compositor running, we d need explicit instruction from the compositor telling us that the window pixmap was now idle : 
 
    SwapIdle
    drawable: Drawable
    pixmap: Pixmap
Furthermore, the application needs to know that the pixmap is in fact idle. I think that we ll need a synchronous X request that marks a buffer as no longer idle and have that return whether the buffer was discarded while idle. It doesn t seem sufficient to use events here as the application will need to completely reconstruct the pixmap contents in this case. This reply could also contain information about precisely what contents the pixmap does contain. 
 
    SwapReuse
    drawable: Drawable
    pixmap: Pixmap
       
    valid: BOOL
    swap-hi: CARD32
    swap-lo: CARD32
Pixmap Lifetimes and Triple Buffered Applications If we redirect the Swap operation and send the original application window pixmap ID to the compositor, what happens when the application frees that pixmap before the compositor gets around to using the contents? Surely the Compositor must handle such cases, and not just crash. However, I m fine with requiring that the application not free the pixmap until told by the compositor.

Next.

Previous.