Search Results: "will"

13 March 2013

Lars Wirzenius: cliapp 1.20130313 released (Python command line utility framework)

I've just released, to code.liw.fi, version 1.20130313 of cliapp, my Python framework for Unix-like command line programs. It contains the typical stuff such programs need to do, such as parsing the command line for options, and iterating over input files. Version 1.20130313 Bug fixes:

Lars Wirzenius: ttystatus 0.22 (terminal status and progress reporting for Python)

I've just released version 0.22 of ttystatus, my Python library for showing progress reporting and status updates on terminals, for (Unix) command line programs. Output is automatically adapted to the width of the terminal: truncated if it does not fit, and re-sized if the terminal size changes. Available on code.liw.fi only, due to the Debian freeze. Version 0.22, released 2013-03-12

12 March 2013

Alexander Reichle-Schmehl: A pledge about a certain Debian mailing list

I hereby pledge, that for every mail I send into a thread on the private Debian list with more than 10 mails, I will fix a release critical bug in a foreign package.

Richard Hartmann: Gitify your life

These Open Source Days were kind of a mixed experience. The social part Things started off quite nicely; the speaker dinner was very pleasant and the people at my table interesting to talk with. The way back to the hotel and breakfast on Saturday were even more interesting; talking with Amelia Andersdotter, a member of the European Parliament for the Swedish Pirate Party, about anti-trust measures, privacy protection, lobbyists, and creating the right incentives for companies to actually protect user data was fun and engaging. My own claim to fame is that I told her that while Microsoft is willing to sign UEFI payloads for x86, they refuse to do so for ARM which she didn't know. From my understanding, their interpretation is that any anti-trust rulings apply to PCs only, and not general computing devices like tablets. It will be interesting to see how that plays out in the long run, especially once ARM processors start appearing in laptops and desktops as main CPUs where Microsoft clearly still has a near monopoly. It was fascinating to discuss all those issues with someone who's actively involved with legislation, mainly because her style of talking was markedly different from what I am used to. One point which she stressed repeatedly is that the most important skill within the context of political work in the EU is to be able to track down, read, and then reference all the content that's being produced in Brussels and Strasbourg. Maybe that tidbit is useful to people who want to try themselves at political activism. She nicely, and maybe unknowingly, emphasized that point by always referring to specific papers, directives, and studies after making any point. Statement of opinion, reference to supporting document. Statement of opinion, reference to supporting document. Statement of opinion, reference to supporting document. Rinse, repeat. It's no secret that I'm continuously disappointed by the political process in general and politicians in particular. Yet, I have to say that this encounter had a lasting and positive impression on me. Amelia has an incredible amount of energy and genuinely cares about the issues which too many other politicians ignore completely. It's good to know someone like this is on our side. If you're Swedish, please inform yourself about her and consider voting in her favour. Other than that, I met several other people, new and old. This year's OSD felt a lot more relaxed and there were more informal and spontaneous conversations going on in hallways and at tables. Even though I failed to have any intense discussion of Mercurial vs git this year, all those conversation were still really interesting. The "workshop" I already knew that the room assignment for my workshop on Saturday afternoon would be a last-minute thing. What I didn't expect was that the final confirmation by Copenhagen Business School would be given on Friday evening. Mixed with a very cumbersome process to update the slides of the CBS-owned and -operated information beamer in the lobby, this resulted in the official schedule showing my room as "Workshop" instead of "SP 214". The official schedule repeatedly stated that people had to pre-register for the workshop which was a left-over from last year where they actually had to plan the amount of hardware kits needed; the registration itself was was very well hidden behind a rather generic link you saw exactly once after purchasing your ticket. Add to this that my room was literally in the farthest corner of the building and on the top floor and things start to turn sour real quick. All other rooms were either close to the main entrance or on the ground floor. Helpers at the front desk had a lot of people asking where my room was and gave out directions, yet... ...the grand total of six people who arrived were all telling me how they couldn't find the room and almost gave up on finding it. As more people asked for directions than turned up at the workshop, this means that even with directions, the room was really hard to find. Myself, it took me two attempts to find the room after which I was led there by one of the helpers who wasn't completely sure we were going in the right direction, himself. Still, I managed to get everyone who had a laptop and access to a UNIX-based system set up with working mr and vcsh repositories they can work from as well as a demo git-annex repository to play with. Sadly, not everyone had a laptop, or access to UNIX, or the tools installed, or even an idea where to install them from, even though I had noted these "requirements" in the description. Next time I do this workshop, I will need to make sure to include information on what distributions have the relevant packages and how to find and install the software on Max OS X in the workshop description. Feedback from the very few people who actually managed to find me was very good and one of the attendants asked if I would be willing to hold the workshop again at another conference in Denmark. I am still awaiting details on this, but I would obviously enjoy having the chance to reach more people. At least I fared better than the guys who did the workshop on "Binary exploitation 101" in yet another room. Two guys held a workshop with a total attendance of... one... I know the organizers of OSD were really unhappy with how this turned out and I believe them when they say that something like this will never happen again. It's the second year that this team is running the conference, running conferences is an incredible amount of work, and things were smoother this year than last year. Iterate and improve, etc. The talk Saturday evening, Alex offered me the chance to hold a reparation talk on Sunday morning to compensate for the botched workshop. Said talk was scheduled for the first slot and not really announced (but the schedule on both www and the beamer was changed over night). Additionally, this was merely an updated and extended version of the talk which I already held at OSD 2012 so at least part of the visitors knew most of what I had to say, already. To be honest I didn't expect too much, but more than twenty people turned up which is a very decent turnout, all things considered. Same as with the workshop, feedback was extremely positive and I was told afterwards by Bryan that several people mentioned my talk as one of the highlights, or even the highlight, of OSD when he was talking to them. To make things even better this talk has, finally, been videotaped; I will update this blog with a link as soon as the video is online.

11 March 2013

Gunnar Wolf: So you want to be a leader

So we are at the beginning of this year's Debian Project Leader elections. And yes, after Stefano's long and (IMO) very successful DPL term, I feel as my voting machinery is somewhat stuck; it will not be so easy to get it back up to speed. Anyway, I have glanced over the three platforms, but only actually read 1.5 from the three DPL platforms. I know that whoever succeeds, I will be quite happy with the results. This year there are three runners for the post. I have worked in several teams with two of them, and would love to know better the third. In the same order as presented in the vote:
Gergely Nagy
I have not yet worked with Gergely, but enjoy reading him. The closest I have been to working with him was sketching a packaging tutorial during DebConf11, in Bosnia. Sadly, I was quite busy, and he picked up the complete workload And correspondingly, got the credits. I can say that Gergely has a very important quality, the ability to put in seductively easy words the most complex processes. So, yes, being the Debian Leader post a public-facing one, I am certain he has one of the important qualities.
Moray Allan
We have worked together organizing DebConf for many years, first loosely as orga-team members, and starting two years ago (and together with Holger Levsen) as formal delegates. I think our team is quite well balanced, and Moray plays an important role. Holger and I are sometimes anxious to take measures, measures that IMO would have proven disastrous more than once. Moray is often the voice of reason. Given that another one of the DPL's roles is to mediate in social conflicts and keep Debian working smoothly (or something close to it ;-) ), that is also a very important trait of a DPL, and I'm also sure Moray would shine as a good leader.
Lucas Nussbaum
I have long been part of the pkg-ruby-extras team (although I am way less active than what I used to, where Lucas often dazzles us with his intense streaks of activity. Among this group of three, I see Lucas firstly as the most technically oriented, the biggest implementer. Also, as the proactive bug-finder and team-herder. And yes, Lucas is maybe the most enthusiastic about the (always) important Making Debian Sexy point. So, if elected, I'm sure this facet will also make him shine
So, it's not that I'm trying to bribe our next DPL with sweet nice words about how interesting a person or how good a friend he is, but am trying to look at the election process as something different. It seems for me that we are going to choose which Debian do we want to pursue for this starting period. Now, for our soon-to-be-ex-DPL Stefano: As many will surely tell you (or already have): You rock. I truly enjoyed your DPL term, and there is much we should adopt and learn from your personality and leadership. And, although it has waned over the past few years, many people tend to publish their (stated?) vote during the campaigning period. I (think I) have never done so, and this time I will surely not do so. Choosing a DPL involves personal feelings, sympathies, and many non-objective things. And although I know nobody will feel hurt if I don't put them in the first place, I prefer not to expose such issues. I can only assure you that this year, "None of the above" will sink to the bottom of my ballot.

Daniel Pocock: Firewalling IPv6

With my new ISP providing native IPv6 transit, enabling IPv6 in Debian is pretty trivial, just update /etc/networking/interfaces:
iface eth0 inet6 static
	pre-up modprobe ipv6
	address 2001:67c:1388::2
	netmask 64
	gateway 2001:67c:1388::1
if you have a Linux router, enable packet forwarding:
echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
and add AAAA records to DNS:
danielpocock.com.        IN      AAAA    2001:67c:1388:1000::9
Yes, that really is enough to get up and running. Apache was already configured to listen on IPv6 and just needed a restart. However, it is not quite secure: the configuration above routes all traffic on all ports. If somebody has the time to port scan my 64 billion trillion possible IPv6 addresses (every IPv6 customer is allocated a /48 as standard), they might end up connecting to some service that should not be public. ip6tables The ip6tables tool for IPv6 does just about everything that iptables does for IPv4. It is tempting to just rewrite an existing firewall script, adding an extra rule for the IPv6 version of every host. For example, my web server is 195.8.117.9, so I copy the 9 off the end and assign the IPv6 address 2001:67c:1388:1000::9 to the same server and add the ip6tables firewall rule:
iptables --append FORWARD -p tcp -d 195.8.117.9 --dport http -j ACCEPT
ip6tables --append FORWARD -p tcp -d 2001:67c:1388:1000::9 --dport http -j ACCEPT
Notice that my IPv6 addressing simply involves re-using the same last byte as the IPv4 equivalent? This is a very easy way to start. However, it is not quite so simple. IPv6 requires a couple of extra rules, in the simplest form:
# Allow traffic for link-local addresses:
ip6tables -A INPUT -s fe80::/10 -j ACCEPT
# Normally, link-local packets should NOT be forwarded and don't need an
# entry in the FORWARD rule.
# However, when bridging in Linux (e.g. in Xen or OpenWRT), the FORWARD rule is
# needed:
ip6tables -A FORWARD -s fe80::/10 -j ACCEPT
# Allow all ICMPv6 traffic (optional for IPv4, mandatory for IPv6):
ip6tables -A INPUT -p icmpv6 -j ACCEPT
ip6tables -A FORWARD -p icmpv6 -j ACCEPT
This works, but it turns out that it is less than ideal. Beyond that, the firewall can be significantly enhanced by rate-limiting the ICMPv6 packets and various other mechanisms as demonstrated with a practical netfilter example in Appendix B of RFC 4890 The question that arises, how to enable wide use of IPv6 without every sysadmin having to familiarise themselves with all these details of filtering ICMPv6 packets? Debian firewall packages My next step will be to look over the existing firewall packages in Debian to see which of them may answer that question. It would be really useful to hear feedback from anybody else who has already undertaken such an evaluation. If none of them are suitable, and if permitted by the licensing terms of RFC 4890, I may well wrap their sample code into a Debian package that can be used as a foundation for people building their own custom firewall scripts.

Russ Allbery: Small non-fiction haul

Long time no write. A variety of things, including getting sick with a particularly bad cold (I'm still coughing up phlegm), led to my schedule and general life organization getting turned on its head, and I'm only slowly recovering. Work gets top priority (including a security release of OpenAFS), so I'm afraid non-work things such as reviews and journal posts (and Debian, and non-work-related software development, and conversations with friends...) have been getting second shrift. Hopefully this is finally starting to improve, and I have a block of time for working remotely coming up that should help. In the meantime, more books have, of course, been acquired. This particular order was mostly to get one book (The Making of the Indebted Man), which I'm going to read and discuss as part of a surprisingly fun political discussion about basic income guarantee and, somewhat by extension, Marxist analysis of capitalism. But it's impossible to buy just one book, similar to how it's impossible to eat just one potato chip: Neil Barofsky Bailout (non-fiction)
E. Gabriella Coleman Coding Freedom (non-fiction)
Joshua Foer Moonwalking with Einstein (non-fiction)
Maurizio Lazzarato The Making of the Indebted Man (non-fiction)
Isabel Wilkerson The Warmth of Other Suns (non-fiction) I of course preordered Coding Freedom and am quite looking forward to reading it, although it will surprise no one to hear that I have rather a long queue. The Warmth of Other Suns I picked up based on a recommendation by Ta-Nehisi Coates, whose writing for The Atlantic I wholeheartedly recommend. I have such a large pile of books that I've read but not yet reviewed that I cringe to think about it. But no progress will be made on that this evening (damn you, Benjamin Franklin! *shakes fist*). More tomorrow if I survive five and a half hours of scheduled forced social interaction.

10 March 2013

Gergely Nagy: DPL platform

Last year, I invited you all to walk the plank with me, to follow me on a road, a road that even I was unsure where it led. I had a vision, I had ideas, but I had little to show for a plan. Even worse, some of my ideas, some of the things I've thought I knew, turned out to be wrong this past year. While my original plan was to pursue the goals I set out in my previous platform, I did none of that, and before I get to the meat of the platform, I owe you an explanation why: After walking the plank last year, I spent some time swimming to the shore, determined that no matter what, I will get things done. But how shall I do that? And when, with whom? Those questions needed an answer, therefore I took on a coat of an ordinary hacker, and started a journey to find the answers.That is how the past year's been spent: looking and learning. I was - and still am - following every major Debian list, I'm on IRC, absorbing all the information that fits into my tiny mind. I see now, that there's a lot more to be done than I thought last year, that some of the assumptions I made then, were wrong. And having spent a long time learning, it is my hope, that I did learn, and that this time, the plans I came up with will serve the same goal I had last year, better.That goal is as simple as growing Debian from a community of mostly technical folk to something much bigger, a community of enthusiastic, bright-eyed people who all want to do their part of making the world a better place. Big words, big task.The problemOne of the most fundamental problems with the project is that it is old, and hasn't grown up yet. We're still very focused on technical merit - don't get me wrong, that is important too, but there's much more we can do than that! We have tremendous amounts of talent, but pretty much every team is suffering from lack of people, and we seem to be slow and ineffective in recruiting new contributors (either from within the project, or outside of it), and even worse at keeping them motivated. This is not new, I worried about it last year, and I'm not the only one to see things this way, respectable members of the project expressed their pessimism publicly too.This is our biggest issue. We can be excellent at the technical level, but not being able to keep people motivated, failing to find new contributors is hurting us more and more, and it is my firm belief that without quick improvement in these areas, it won't be just me virtually walking the plank.On the flip side, we have every tool in our possession to steer ourselves into a better, brighter direction. We have excellent hackers, we have people who are terrific at communication, we have people who inspire others (just see the pattern on the dpl-game!), we have everything we need. Read on then, dear reader, and see my plan!Dancing on a tightropeTo tread on a path we have not taken so consciously before, we need to have a vision to strive for. A vision for the project as a whole, and a vision for ourselves, to see where we'd like to be a few years from now.As Stefano put it so well in his platform last year, Debian is a very unique member of the larger Free Software ecosystem - go and read his description, it can hardly be written better. This is the same vision I share, where Debian grows up to be much more than hundreds of talented people putting together a distribution. I want to see Debian becoming more than that, something that attracts not only the technical geniuses, but those who wield the power of speech and inspiration too, for these traits rarely go together.I envision a project, where people recognise each other's strengths, and make use of these strengths; where we can - and will - turn to each other, and work closely together, to achieve such good balance of skills of all kinds, as we already achieved in creating and maintaining the distribution itself.Let's dance!But alas, a vision is useless without a plan. That is a mistake I made last year, one that I will not repeat again. The approaches I choose may seem strange, or weird at first, but rest assured, they were carefully considered, and most of them tried out in a smaller setting. CommunicationWhat I've seen and experienced over the past year (and thinking back, far more than that - I just didn't know where to put the experience at that time) is that written communication is rarely the best medium to reach out to people. In this age we live in, written text reaches those who were already curious, it rarely reaches those who're still waiting for enlightenment. And even if it does, more often than not, there is no direct channel between the writer and the reader, which hurts not only the latency of the communication, but the quality too. Not to mention, that - however sad it may sound - the recipient is often too shy to engage.What does help, is face to face time. Events, where we can reach out to people, and receive immediate feedback, where we can notice the shy ones, and help them. Where we can immediately adapt to expectations.For this reason, I feel it necessary to encourage and help local teams in every possible way, to organise hackfests, code retreats, and all kinds of events that they feel will attract the bright-eyed youth (be them 20 or 80, it's the spirit that counts) we so direly need. These events need not be technical at all: a session about Debian publicity, about representing the project in press or on conferences is just as useful (if not more). Along with hackfests, we could - and should - have events focused on improving communication, for that is just as important.With strong focus placed on non-technical matters, I believe we can step on a path of rapid progress: my experience shows that techies and non-techies can inspire one another very well, much more than two techies would. With a focus on areas we are lacking in, we are indirectly improving those we are already good at. Inspiration & motivationBut finding new contributors, be them technically-sawwy or not is only part of the task, and the easier one too. It is much harder to keep people interested and motivated over a long period of time, than to find them once.On one hand, I would like to rely on those members of our community, who we all hold in high esteem (including, but not limited to those mentioned during the dpl-game). They know what keeps them going, they know what drives them away, we need to learn from that. I'd like to hear them speak more often, on conferences, if possible, as they are, I believe, a much more credible source than any elected official, at least in some situations.I'd also like to hear everyone's voice, from old members to the newest ones, I'd like to read - and if so need be, conduct - interviews with people who recently became Debian Developers, to hear about their experience, to learn what they think is good or bad, because a fresh view is just as important as a decade long experience.On the other hand, we need to act on the issues discovered too. We already know that quite a lot of trouble stems from lack of time, overburden and eventually burn out. We know the solution to this: recruit more people, and utilise the resources we do have better. Part of that can be solved by expanding our numbers, but for the other part, we may need deeper changes. What those may be, I cannot tell yet, there simply wasn't enough time to dig deeper, and I'm likely not the best person to undertake the task, either. I am, however, confident that we'll discover the solution as we travel along.The first step is gaining knowledge, without this, no further plans can be made. RecruitmentI already touched the recruitment part earlier, at least the how of the approach. What I'd like to expand on is the why, in particular: why encourage recruiting non-packaging contributors, when we are building a distribution, and we're lacking packaging manpower too?Let me answer those questions separately.First of all, I still like to think - but perhaps I am wrong - that we have an unbelievable amount of packaging manpower, which we fail to use to its full extent. We can use more, we always will be able to. But it also helps if we manage to put our existing resources to better use.And that is where non-packaging contributions come in. If we improve our skills in areas we're lacking in, that in turn, will improve the general quality of the project, which in turns improves morale, and a vibrant, well functioning, welcoming community attracts even more people, and keeps us motivated in the long run too.Mouse on the tightropeWith all that out, you may be wondering who I may be, and why you have not seen my name over the mailing lists in recent years. I'm not a terribly vocal person anymore, not a hot-headed teen I used to be some ten years ago. Most who know me, know me by my online nick name, algernon, and in most cases, I prefer that in real life too, for silly reasons I'd rather not dive into (but the primary one being that people have a hard time correctly pronouncing my name, and I have a hard time recognising my name unless pronounced the way I learnt it). Nevertheless, my name is Gergely Nagy, and I'm a recovering hacker, who intends to become a bachelor of (hungarian) arts, by majoring in Hungarian grammar & literature sometime in the next three to five years.Over the course of the past year, I also strived to observe and learn from exceptional leaders at my day job, because there, I have the advantage of meeting with them face to face, to interact with them on a daily basis. Fortunately for me, there are people among them from whom I learned a great deal about not only motivation, but about behind-the-scenes politics too - both a valuable lesson.And while I still have a tremendous amount of things to learn, I like to believe that I can do that. I know I can learn technical things extremely quickly (my day job depends on that, I would be out of a job if I failed at that), I'm positive that I can do the same in other areas as well, at least as long as it doesn't involve extreme amounts of economics. That proved to be a major barrier in the past.But back to the more important point: my organising, presentation and leading skills. I'm a co-organiser of the Budapest Clojure User Group meetups, I started to encourage GSoC participation at my day job last year (we participated thanks to OpenSUSE who gave us a slot), and I'm driving it this year, aiming to become a mentoring organisation of our own. I'm doing regular talks and presentations, participate in code retreats, meetups and other similar events where I can both practice and learn the art of interacting with people, and the fine art of organising events.

Gergely Nagy: DPL platform

Last year, I invited you all to walk the plank with me, to follow me on a road, a road that even I was unsure where it led. I had a vision, I had ideas, but I had little to show for a plan. Even worse, some of my ideas, some of the things I've thought I knew, turned out to be wrong this past year. While my original plan was to pursue the goals I set out in my previous platform, I did none of that, and before I get to the meat of the platform, I owe you an explanation why: After walking the plank last year, I spent some time swimming to the shore, determined that no matter what, I will get things done. But how shall I do that? And when, with whom? Those questions needed an answer, therefore I took on a coat of an ordinary hacker, and started a journey to find the answers.That is how the past year's been spent: looking and learning. I was - and still am - following every major Debian list, I'm on IRC, absorbing all the information that fits into my tiny mind. I see now, that there's a lot more to be done than I thought last year, that some of the assumptions I made then, were wrong. And having spent a long time learning, it is my hope, that I did learn, and that this time, the plans I came up with will serve the same goal I had last year, better.That goal is as simple as growing Debian from a community of mostly technical folk to something much bigger, a community of enthusiastic, bright-eyed people who all want to do their part of making the world a better place. Big words, big task.The problemOne of the most fundamental problems with the project is that it is old, and hasn't grown up yet. We're still very focused on technical merit - don't get me wrong, that is important too, but there's much more we can do than that! We have tremendous amounts of talent, but pretty much every team is suffering from lack of people, and we seem to be slow and ineffective in recruiting new contributors (either from within the project, or outside of it), and even worse at keeping them motivated. This is not new, I worried about it last year, and I'm not the only one to see things this way, respectable members of the project expressed their pessimism publicly too.This is our biggest issue. We can be excellent at the technical level, but not being able to keep people motivated, failing to find new contributors is hurting us more and more, and it is my firm belief that without quick improvement in these areas, it won't be just me virtually walking the plank.On the flip side, we have every tool in our possession to steer ourselves into a better, brighter direction. We have excellent hackers, we have people who are terrific at communication, we have people who inspire others (just see the pattern on the dpl-game!), we have everything we need. Read on then, dear reader, and see my plan!Dancing on a tightropeTo tread on a path we have not taken so consciously before, we need to have a vision to strive for. A vision for the project as a whole, and a vision for ourselves, to see where we'd like to be a few years from now.As Stefano put it so well in his platform last year, Debian is a very unique member of the larger Free Software ecosystem - go and read his description, it can hardly be written better. This is the same vision I share, where Debian grows up to be much more than hundreds of talented people putting together a distribution. I want to see Debian becoming more than that, something that attracts not only the technical geniuses, but those who wield the power of speech and inspiration too, for these traits rarely go together.I envision a project, where people recognise each other's strengths, and make use of these strengths; where we can - and will - turn to each other, and work closely together, to achieve such good balance of skills of all kinds, as we already achieved in creating and maintaining the distribution itself.Let's dance!But alas, a vision is useless without a plan. That is a mistake I made last year, one that I will not repeat again. The approaches I choose may seem strange, or weird at first, but rest assured, they were carefully considered, and most of them tried out in a smaller setting. CommunicationWhat I've seen and experienced over the past year (and thinking back, far more than that - I just didn't know where to put the experience at that time) is that written communication is rarely the best medium to reach out to people. In this age we live in, written text reaches those who were already curious, it rarely reaches those who're still waiting for enlightenment. And even if it does, more often than not, there is no direct channel between the writer and the reader, which hurts not only the latency of the communication, but the quality too. Not to mention, that - however sad it may sound - the recipient is often too shy to engage.What does help, is face to face time. Events, where we can reach out to people, and receive immediate feedback, where we can notice the shy ones, and help them. Where we can immediately adapt to expectations.For this reason, I feel it necessary to encourage and help local teams in every possible way, to organise hackfests, code retreats, and all kinds of events that they feel will attract the bright-eyed youth (be them 20 or 80, it's the spirit that counts) we so direly need. These events need not be technical at all: a session about Debian publicity, about representing the project in press or on conferences is just as useful (if not more). Along with hackfests, we could - and should - have events focused on improving communication, for that is just as important.With strong focus placed on non-technical matters, I believe we can step on a path of rapid progress: my experience shows that techies and non-techies can inspire one another very well, much more than two techies would. With a focus on areas we are lacking in, we are indirectly improving those we are already good at. Inspiration & motivationBut finding new contributors, be them technically-sawwy or not is only part of the task, and the easier one too. It is much harder to keep people interested and motivated over a long period of time, than to find them once.On one hand, I would like to rely on those members of our community, who we all hold in high esteem (including, but not limited to those mentioned during the dpl-game). They know what keeps them going, they know what drives them away, we need to learn from that. I'd like to hear them speak more often, on conferences, if possible, as they are, I believe, a much more credible source than any elected official, at least in some situations.I'd also like to hear everyone's voice, from old members to the newest ones, I'd like to read - and if so need be, conduct - interviews with people who recently became Debian Developers, to hear about their experience, to learn what they think is good or bad, because a fresh view is just as important as a decade long experience.On the other hand, we need to act on the issues discovered too. We already know that quite a lot of trouble stems from lack of time, overburden and eventually burn out. We know the solution to this: recruit more people, and utilise the resources we do have better. Part of that can be solved by expanding our numbers, but for the other part, we may need deeper changes. What those may be, I cannot tell yet, there simply wasn't enough time to dig deeper, and I'm likely not the best person to undertake the task, either. I am, however, confident that we'll discover the solution as we travel along.The first step is gaining knowledge, without this, no further plans can be made. RecruitmentI already touched the recruitment part earlier, at least the how of the approach. What I'd like to expand on is the why, in particular: why encourage recruiting non-packaging contributors, when we are building a distribution, and we're lacking packaging manpower too?Let me answer those questions separately.First of all, I still like to think - but perhaps I am wrong - that we have an unbelievable amount of packaging manpower, which we fail to use to its full extent. We can use more, we always will be able to. But it also helps if we manage to put our existing resources to better use.And that is where non-packaging contributions come in. If we improve our skills in areas we're lacking in, that in turn, will improve the general quality of the project, which in turns improves morale, and a vibrant, well functioning, welcoming community attracts even more people, and keeps us motivated in the long run too.Mouse on the tightropeWith all that out, you may be wondering who I may be, and why you have not seen my name over the mailing lists in recent years. I'm not a terribly vocal person anymore, not a hot-headed teen I used to be some ten years ago. Most who know me, know me by my online nick name, algernon, and in most cases, I prefer that in real life too, for silly reasons I'd rather not dive into (but the primary one being that people have a hard time correctly pronouncing my name, and I have a hard time recognising my name unless pronounced the way I learnt it). Nevertheless, my name is Gergely Nagy, and I'm a recovering hacker, who intends to become a bachelor of (hungarian) arts, by majoring in Hungarian grammar & literature sometime in the next three to five years.Over the course of the past year, I also strived to observe and learn from exceptional leaders at my day job, because there, I have the advantage of meeting with them face to face, to interact with them on a daily basis. Fortunately for me, there are people among them from whom I learned a great deal about not only motivation, but about behind-the-scenes politics too - both a valuable lesson.And while I still have a tremendous amount of things to learn, I like to believe that I can do that. I know I can learn technical things extremely quickly (my day job depends on that, I would be out of a job if I failed at that), I'm positive that I can do the same in other areas as well, at least as long as it doesn't involve extreme amounts of economics. That proved to be a major barrier in the past.But back to the more important point: my organising, presentation and leading skills. I'm a co-organiser of the Budapest Clojure User Group meetups, I started to encourage GSoC participation at my day job last year (we participated thanks to OpenSUSE who gave us a slot), and I'm driving it this year, aiming to become a mentoring organisation of our own. I'm doing regular talks and presentations, participate in code retreats, meetups and other similar events where I can both practice and learn the art of interacting with people, and the fine art of organising events.

Hideki Yamane: great IOPS device and if we'd have it...?


I know some folks work at Fusion IO, maybe you know, one of the leader of PCIe flash storage system. And they say their products are "great", since their device put amazing IOPS as they press released (9,000,000 IOPS, and Wikipedia says "SAS HDDs put 200 IOPS". If you have a SSD in your laptop/desktop, you'd love it. It puts almost 10,000 IOPS or so). Such PCIe SSD can bring "Moon shot", change the rules for managing CPU, Memory, Disk and Network for servers, IMHO.


Then, I'm dreaming - if we Debian would have such MAGICAL device, what is the most efficient way to improve/accelerate/Kaizen our project? To use mailing list server? BTS? UDD? CI?

Dear lazyweb, please let me know what is the bottleneck of our development, and great IOPS capacity will fix it.

(and please don't say about things based on human resource, since PCIe SSD devices cannot improve such things, you know ;)

Axel Beckert: Rendering Markdown, Asciidoc and Friends automatically while Editing

Partially because of Markdown being Github s markup format of choice, I enjoy writing documents in simple markup formats more and more. There s though one common annoyance with these formats compared to writing plain HTML The Annoyance They need to be rendered (i.e. more or less compiled) before you can view your outpourings rendered, e.g. in the web browser. So the workflow usually is:
  1. Saving the current file in your favourite editor
  2. Switch to terminal with commandline
  3. Cursor up, Enter
  4. Switch to your favourite web browser
  5. Hit the reload button
Using a Specialized Editor with Live Preview One choice would be to use a specific editor with live rendering. The one I know in Debian (from Wheezy on) is ReText (Debian package retext). It supports Markdown and reStructuredText. But as with most simple GUI editors, I miss there many of the advanced editing commands possible with Emacs. Using Emacs Markdown Mode Then there is the Markdown Mode for Emacs (part of Debian s emacs-goodies-el package), where you can get a preview by pressing C-c C-c p. But for some reason this takes several seconds, opens a new buffer and window with the rendered HTML code and then starts (hardcoded) Firefox (which is not my preferred web browser). And if you do that a second time without closing Firefox first, it won t just reload the file but will open a new tab. You might think that just hitting reload should suffice. But no, the new tab has a different file name, so reload doesn t help. Additionally it may not use my preferred Markdown implementation. Meh. Well, I probably could fix all those issues with Markdown Mode, it s only Emacs Lisp. Heck, the called command is even configurable. But fixing at least four issues to fix one workflow annoyance? Maybe some other time, but not as long there are other nice choices Using inotifywait to Render on Write So everytime you save the currently edited file, you immediately want to rerender the same HTML file from it. This can be easily automated by using Linux inotify kernel subsystem which notices changes to the filesystem, and reports those to applications which ask for it. One such tool is inotifywait which can either output all or just specific events, or just exit if the first requested event occurs. With the latter it s easy to write a while loop on the commandline which regenerates a file after every write access. I use either Pandoc or Asciidoc for that since both generate full HTML pages including header and footer, but you can use that also with Markdown to render just the HTML body. Most browsers render it correctly anyway:
while inotifywait -q -e modify index.md; do pandoc -s -f markdown -t html -o index.html index.md; done
while inotifywait -q -e modify index.txt; do asciidoc index.txt; done
while inotifywait -q -e modify index.md; do markdown index.md > index.html; done
This solution is even editor- and build-system-agnostic (But not operating-system-agnostic.) inotifywait is part of inotify-tools, a useful set of commandline tools to interface with inotify. They re packaged in Debian as inotify-tools, too. Using mdpress for Markdown plus Impress.js based Slides The ruby-written mdpress is a special case of the previous case. It s a commandline tool to convert Markdown into Impress.js based slide shows and it has an option named --automatic which causes it to keep running and automatically update the presentation as soon as changes are made to the Markdown file. mdpress is not yet in Debian, but there s an ITP for it and Impress.js itself recently entered Debian as libjs-impress. Nevertheless, two dependencies (highlight.js, ITP ed, ruby-launchy, ITP ed) are still missing in Debian.

9 March 2013

Lars Wirzenius: Can the world be saved by coding?

I love programming. I was born to code. I want to save the world by writing code. I feel very strongly about software freedom. As I get older, and my eyes open to see more of the evil in the world, I become more concerned about freedom in general. Software freedom is, at least traditionally, about permission use, study, modify, and re-distribute software. The Free Software Foundation expresses this as the four essential freedoms. Debian expanded on that, and produced the Debian Free Software Guidelines. This is no longer enough. It is not enough to have all the freedom when you using your own computer, to have the source code to every bit of code that runs on your hardware. We live in the era of the Internet. Much of what we use computers for involves communication over the Internet, and the Internet is being actively used to curtail the freedom of people. For example, governments and corporations do large-scale surveillance of everyone, by eavesdropping on private communications, gathering enormous databases of personal data, and by analysing everything they can in order to find patterns and make conclusions both at the statistical level and about individuals. This ruins privacy. Without privacy, democracy cannot survive. Another example: in the name of various strawmen, such as terrorism, copyright violations, drugs, and child pornography, governments and corporations are collaborating in limiting private people's communications. I'm a Finnish citizen living in the UK. Both countries are among the closest ones to an ideal democractic nation state. Both countries arbitrarily block access to websites based on lists provided by private organisations, assuming that those organisations produce accurate lists of sites that contain copyright violations. As a result, both countries blocked, for example, a site where musical artists promoted their own works, bypassing the large media corporations that fund the list-making organisation. These issues transcend software freedom, though they interact with it. Software freedom is a necessary requirement for freedom, in a world where almost everything is done with, or controlled by, the use of computers. Software freedom is not enough to prevent censorship or surveillance: even if all the software in the world were free, including the national firewalls of China, Finland, and the UK, this would not prevent those countries from censoring and surveilling their citizens. The firewall would run free software, but that does not give the citizens the freedom to disable, or modify, the firewall systems. Fixing these issues is not a coding task. It is a job for politics, and it is going to take a long time, I fear. In the mean time, is there something a hacker can do to improve things? My main hobby project is Obnam, my backup program. Does that help people to protect their freedom? I think it does, in a small way: Obnam supports online, encrypted backups (and de-duplication even for encryption is used), which means they can make backups to servers anywhere on the Internet without having to fear their data gets read and analysed by hostile entities such as their own government, other governments, large corporations, or criminals. Obviously this does not help them if their government prohibits the use of encryption, or requires key escrow, or mandates backdoors to all encryption methods. But it's a step in the right direction. I don't claim Obnam will save the world, but a million small such steps by a thousand individual hackers, even without any particular organisation or guidance, will make a big impact. What step can you take?

8 March 2013

Richard Hartmann: Release Critical Bug report for Week 09

I extended UDD's rcblog historical bug overview table until week 18, which is the last week of April. Let's hope we won't need that long and I will be able to delete half of those. The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the Squeeze release cycle?
Week Squeeze Wheezy Diff
43 284 (213+71) 468 (332+136) +184 (+119/+65)
44 261 (201+60) 408 (265+143) +147 (+64/+83)
45 261 (205+56) 425 (291+134) +164 (+86/+78)
46 271 (200+71) 401 (258+143) +130 (+58/+72)
47 283 (209+74) 366 (221+145) +83 (+12/+71)
48 256 (177+79) 378 (230+148) +122 (+53/+69)
49 256 (180+76) 360 (216+155) +104 (+36/+79)
50 204 (148+56) 339 (195+144) +135 (+47/+90)
51 178 (124+54) 323 (190+133) +145 (+66/+79)
52 115 (78+37) 289 (190+99) +174 (+112/+62)
1 93 (60+33) 287 (171+116) +194 (+111/+83)
2 82 (46+36) 271 (162+109) +189 (+116/+73)
3 25 (15+10) 249 (165+84) +224 (+150/+74)
4 14 (8+6) 244 (176+68) +230 (+168/+62)
5 2 (0+2) 224 (132+92) +222 (+132/+90)
6 release! 212 (129+83) +212 (+129/+83)
7 release+1 194 (128+66) +194 (+128/+66)
8 release+2 206 (144+62) +206 (+144/+62)
9 release+3 174 (105+69) +174 (+105/+62)
Graphical overview of bug stats thanks to azhag:

Richard Hartmann: Release Critical Bug report for Week 10

By using highly advanced and secret analytical tools (i.e. stupid correlation), I am guessing that Wheezy will be released in about six weeks. Week 16... mark my words. The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the Squeeze release cycle?
Week Squeeze Wheezy Diff
43 284 (213+71) 468 (332+136) +184 (+119/+65)
44 261 (201+60) 408 (265+143) +147 (+64/+83)
45 261 (205+56) 425 (291+134) +164 (+86/+78)
46 271 (200+71) 401 (258+143) +130 (+58/+72)
47 283 (209+74) 366 (221+145) +83 (+12/+71)
48 256 (177+79) 378 (230+148) +122 (+53/+69)
49 256 (180+76) 360 (216+155) +104 (+36/+79)
50 204 (148+56) 339 (195+144) +135 (+47/+90)
51 178 (124+54) 323 (190+133) +145 (+66/+79)
52 115 (78+37) 289 (190+99) +174 (+112/+62)
1 93 (60+33) 287 (171+116) +194 (+111/+83)
2 82 (46+36) 271 (162+109) +189 (+116/+73)
3 25 (15+10) 249 (165+84) +224 (+150/+74)
4 14 (8+6) 244 (176+68) +230 (+168/+62)
5 2 (0+2) 224 (132+92) +222 (+132/+90)
6 release! 212 (129+83) +212 (+129/+83)
7 release+1 194 (128+66) +194 (+128/+66)
8 release+2 206 (144+62) +206 (+144/+62)
9 release+3 174 (105+69) +174 (+105/+69)
10 release+4 120 (72+48) +120 (+72/+48)
11 release+5
12 release+6
13 release+7
14 release+8
15 release+9
16 release+10
17 release+11
18 release+12
Graphical overview of bug stats thanks to azhag:

Vasudev Kamath: Multi-Arch lintian check for font packages

I've provided a new patch #701061 for lintian to warn about font packages that are not marked as Multi-Arch foreign or allowed. Its already included in the lintian by Neils Thykier and will be part of version 2.5.12. The following tag has been implemented
font-package-not-multi-arch-foreign
A Bit of History for this implementation is as follows: We got a bug report on one of the fonts in pkg-fonts team that its not been installed in i386 architecture on a amd64 multi-arch system #694864. We were first confused but Daniel Kahn Gillmor pointed that we indeed need to mark all font packages as Multi-Arch foreign. He proposed that we should write a lintian check for this which I volunteered to do and then forgot!. Recently I was checking my QA page and landed into Ubuntu's page of my package where I saw they were patching imported font package and marking them as Multi-Arch: foreign and I suddenly remembered my promise! and this patch was the result of same enlightenment :-). Since there is huge number of font packages maintained by pkg-fonts devel we targeted this for Jessie release.
Here by I request all font package maintainers to consider marking their packages as Multi-Arch foreign. I also request people to join us on pkg-fonts-devel and help us doing this for all font packages maintained by the team, we really lack people in the team.

Sylvain Le Gall: Always test your HD first

I just received a WD 500GB Blue, to replace the hard drive of my wife's computer. First thing after unpacking: start to test. And I was right to do it... there are bad blocks on it. WD 500GB DOA The story: I tend to do proactive replacement of hard drive, to avoid losing too much data or be in hurry when the current one fails. There are two ways to monitor a hard drive: Logcheck is pretty straightforward to install. It scans your log every two hours and send you a report of what is happening. It is not a very precise tool and you have to tune it a little bit to just send you what is relevant (install extra rules to ignore what you know is not of interest). Whenever you start to see log entry like that:
Jan 24 18:16:08 foo kernel: [ 1965.343980] ata5.00: exception Emask 0x50 SAct 0x39 SErr 0x800 action 0x6 frozen
Jan 24 18:16:08 foo kernel: [ 1965.343991] ata5.00: irq_stat 0x08000000, interface fatal error
Jan 24 18:16:08 foo kernel: [ 1965.344001] ata5: SError:   HostInt  
Jan 24 18:16:08 foo kernel: [ 1965.344036] ata5.00: failed command: READ FPDMA QUEUED
Jan 24 18:16:08 foo kernel: [ 1965.344055] ata5.00: cmd 60/08:00:18:31:44/00:00:1a:00:00/40 tag 0 ncq 4096 in
Jan 24 18:16:08 foo kernel: [ 1965.344059]          res 40/00:2c:e6:c2:fd/00:00:26:00:00/40 Emask 0x50 (ATA bus error)
Jan 24 18:16:08 foo kernel: [ 1965.344071] ata5.00: status:   DRDY  
Jan 24 18:16:08 foo kernel: [ 1965.344081] ata5.00: failed command: WRITE FPDMA QUEUED
It is a good time to think about changing your hardrive -- but it is maybe too late. Smartmontools (aka smartd) is dedicated tool to monitor hard drives and do a good job. I think it is not installed by default in Debian, but it should be. It scans your hard drive for SMART capabilities and monitor the health of the HD using internal tools. In the case of bad blocks, you will start to see entry like that:
Feb 17 14:05:17 bar smartd[1268]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Feb 17 14:35:17 bar smartd[1268]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Obviously when you see this log, it is also a good time to change your hard drive. In the case of my wife's HD, I just got data from logcheck. It means that the error is not that important (transient failure, something is wrong but the HD can cope with it). But I still decided to get a new one for my wife. Whenever, I receive a new drive, the first thing I do is to check it for errors. You can do that using the program badblocks in write mode. It takes ages to test (count up to 1 day for 1TB on USB), but at the end you know that you have a good candidate -- where it is worth install your data. You just have to follow this procedure
  1. dmesg grep sd
  2. try to find what drive is the one you want to test in the output of 1.
  3. cfdisk /dev/sdX, sdX being the drive you want to test
  4. check that what you see in cfdisk is what you expect to test: right name and capacity for the drive
  5. sudo badblocks -wvs /dev/sdX
  6. run sudo tail -f /var/log/syslog in parallel just in case
  7. wait
If some errors appears in /var/log/syslog, you know something bad is happening. Whenever you have a single failing block, don't think it is ok. It is NOT ok for a HD to start its life with failing blocks. In this case, repack the drive and send it for replacement ASAP. In the case of my hard drive: smartd mail:
The following warning/error was logged by the smartd daemon:
Device: /dev/sda [SAT], 15 Currently unreadable (pending) sectors
syslog entries:
Mar  2 20:52:57 foo kernel: [ 8317.419715] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
Mar  2 20:52:57 foo kernel: [ 8317.419724] ata4.00: irq_stat 0x40000008
Mar  2 20:52:57 foo kernel: [ 8317.419732] ata4.00: failed command: READ FPDMA QUEUED
Mar  2 20:52:57 foo kernel: [ 8317.419747] ata4.00: cmd 60/80:00:00:af:08/00:00:28:00:00/40 tag 0 ncq 65536 in
Mar  2 20:52:57 foo kernel: [ 8317.419750]          res 41/40:00:65:af:08/00:00:28:00:00/40 Emask 0x409 (media error) <F>
Mar  2 20:52:57 foo kernel: [ 8317.419758] ata4.00: status:   DRDY ERR  
Mar  2 20:52:57 foo kernel: [ 8317.419764] ata4.00: error:   UNC  
Mar  2 20:52:57 foo kernel: [ 8317.423959] ata4.00: configured for UDMA/133
I am not blaming any particular brand (like Western Digital), all computer parts I have ever bought had to follow the same procedure and it is a known fact that computer parts as a non-zero percentage of chance to be DOA (dead on arrival) or after a few weeks. But as a consumer you should be aware of that and take action to avoid spending 10h configuring your computer to see it failing after a week... The waste of time to test is a win on the long term.

Paul Tagliamonte: Ubuntu: The Community

Alas. The blogosphere has been slammed with discontent. Count me as one of them. It s been a long time since the last position (I ve held) of significance in Ubuntu, a LoCo Council Member, which I left with a heavy heart. I felt, at the time, that Ubuntu, The Community was no longer in control of Ubuntu The Distro . Technical opinions of Ubuntu community members started to get disregarded (starting with the gawd-awful Yahoo search switch), and it continued to get worse as Canonical stopped becoming a contributor to Ubuntu and started becoming it s sole guiding force. In comes the mess called Ubuntu One which, for the record, is a wrong name, it s not Ubuntu in any sense, it s not Community driven, it just uses Ubuntu as one of it s platforms. Trading on Ubuntu s good name started this mess. Slowly, like the ship of theseus, Canonical started to swap out bit by bit until nothing was left without Canonical assignment. I always assumed it was so that selling the code for another company to use the code for non-free purpose was the reason. Oh well. Look at us now: Starting off on an ill-thought out path because others are unwilling to say they just didn t understand the whole problem. This is a long way of saying: Ex-Ubuntu ers: You re welcome in Debian. I ve set up #ubuntu-expats on oftc. Please join us. You re welcome there. Working in Debian has been an absolute dream; it feels like contributing to Ubuntu in the early days. Come. Have a say. Re-join a Community that isn t just Like us on Facebook . I still consider myself an Ubuntu Community Member, but I don t think there s any community left.

7 March 2013

Vincent Sanders: Man cannot discover new oceans unless he has the courage to lose sight of the shore

Continuing with my whirlwind introduction to NetSurf Development now is the time to start examining the code, how its arranged and how to interact with the existing developers.

The way the NetSurf source is structured is around the idea of the frontends each being a native browser. While this implies that there are nine separate browsers that happen to share a common code base the separation is not quite that well defined.

Each frontend provides the OS entry point (the main() function in a c program) and calls out to standard browser initialisation entry function netsurf_init() and then starts running the browsers main dispatch loop with netsurf_main_loop() when that exits the frontend cleans up with netsurf_exit().

The frontends provide a large selection of functions which are called from the core code. These routines run from running the event scheduler through to rendering graphics and text.
Finding your way aroundThe browsers directory layout is fairly shallow consisting of some Makefiles, the nine frontend directories and eleven others.

The Makefiles are GNU make and represent a pretty straightforward linear build system. We do not use recursive make or autotools. There are plans to use the core buildsystem that all the other components use.

The frontend directories contain the code for the frontend and the makefile fragments to build them which are included by the top level Makefile.

In addition there is:
desktop
This contains the non frontend specific code that actually behaves like a browser. For example desktop/netsurf.c contains the three primary functions we outlined in the introduction. You will also find much of the function and data structures interfaces the frontend must provide. It is unlikely someone new to the project will need to change anything in here (there be dragons) but is an important set of routines.
utils
Here you will find the utility and compatibility interfaces, things like url handling, logging, user messages, base64 handling. These are utility interfaces that do not justify splitting out their functionality to a separate library but are useful everywhere. Changing an interface in here would likely result in a major refactor.
For example one quirk is the logging macro was created before varadic C preprocessor macros were universal so it must be called as LOG(("hello %s world", world_type)) e.g. with double brackets. Fixing this and perhaps improving the logging functionality would be "nice" but the changes would be massive and potentially conflict with ongoing work.
content
This contains all the core code to handle contents i.e. html, css, javascript, images. The handling includes retrieval of the resources from URI , correct caching of the received objects and managing the objects state. It should be explicitly stated that the content handlers are separate and use this core functionality. The actual content handlers to deal with object contents such as the routines to decode image files or render html are elsewhere.
image
This is where the content handlers for the various image types are kept. The majority of these image types (jpeg, png, webp, gif, bmp) use a standard library to perform the actual decode (libjpeg, libpng etc.). One special feature used by most image handlers is that of a decoded image cache which is distinct and separate from the content cache.
The decoded image cache manages decoding of the source images (the jpegs and pngs) into frontend specific "render" bitmaps. For example the gtk frontend keeps the decoded images as a cairo surface ready for immediate plotting.
The cache uses a demand based (when the browser actually displays the image) just in time strategy which has been carefully balanced, with real world input, to reduce the overhead of unnecessary image decoding and processing against memory usage for the render bitmaps.
css
The css content handlers provide for the processing of css source text and use the NetSurf libcss library to process into a bytecode suitable for applying style selection at render time.
javascript
The javascript handlers (strictly speaking this should be named the "script" directory as all types of script are handled here) provide basic functionality to bind a javascript engine to the rest of the browser, principally the Document Object Model (DOM) accessed with libdom. The only engine that currently has bindings is Spidermonkey from the Mozilla foundation.
render
This is the heart of the browser containing the content handler for html (and plain text). This handler deals with:
  • Acquiring the html.
  • Running the base parser as data arrives which generates the DOM and hence DOM events from which additional resource (stylesheets etc.) fetches are started.
  • Deal with script loading
  • Constructing the box model used for layout
  • Performing the Layout and rendering of the document.
Because this module has so many jobs to do it has inevitably become very complex and involved, it is also the principle area of core development . Currently NetSurf lacks a dynamic renderer so changes made by scripts post document load event are not visible. This also has the side effect that the render is only started after the DOM has finished construction and all the resources on a page have completed their fetch which can lead to undesirable display latency.
Docs
Documentation about building and using NetSurf. If anyone wants a place to start improving NetSurf, this is it, it is very incomplete. It must be noted this is not where dynamically generated documentation is found. For the current Doxygen output the best place to look is the most recent build on the Continuous Integration system.
resources
These are runtime resources which are common to all frontends. To be strictly correct they may be the sources which get converted into runtime resources e.g. The FatMessages file which is teh message text for all frontends in all languages, this gets processed at build time into separate files ready.
!NetSurf
This is another resources directory and technically the resources for the riscos frontend. The naming and reliance on this directory are historical. To allow the RISC OS frontend to be run directly from the source directory and an inability of RISC OS to process symbolic links most common runtime resources end up in here and linked to from elsewhere.
test
These are some basic canned test programs and files, principally to test elements of the utils and perform specific exercise of various javascript components.
Getting started
Once a developer has a checked out working build environment and can run the executable for their chosen frontend (and maybe done some web browsing :-) it is time to look at contributing.

If a developer does not have a feature or bug in mind when they begin the best way to get started is often to go bug hunting. The NetSurf bugtacker has lots to choose from unfortunately. Do remember to talk to us (IRC is the best bet if you are bug hunting) about what you are up to but do not be impatient. Some of those bugs are dirty great Shelob types and are not being fixed because even the core developers are stumped!

When first getting going I cannot recommend reading the code enough, this seems to be a skill that many inexperienced open source developers have yet to acquire, especially if they are from a predominately proprietary development background. One wonderful feature of open source software is you get to see all of it, all the elegant nice code and all the "what the hell were they thinking" code too.

One important point is to use your tools well the source is in git, if you learn how to use git well you will gain a skill that is readily portable, not just for NetSurf. And not just revision control tools, learn to use your debugger well and tools like valgrind. Those skills will replay the time spent learning them themselves many times over.

When using git one thing to remember is commit early and often, it does not matter if you have lots of junk commits and dead ends, once you have something viable you can use
git rebase --interactive
and rewrite it all into a single sensible set of commits. Also please do remember to develop on a branch, or if you did not
git pull --rebase
is your friend to avoid unnecessary merges.
Playing nicely with othersThe NetSurf community is a small band of unpaid volunteers. On average we manage to collectively put in, perhaps, ten hours a week with the occasional developer weekend where we often manage over twenty hours each.

The result is that developer time is exceptionally valuable, add in a mature codebase with its fair share of technical debt and you get a group who, when they get to work on NetSurf, are incredibly busy. To be fair we are just like every other small open source project in that respect.

What this means to a new contributor is that they should try and do their homework before asking the obvious questions. The documentation is there for a reason and in spite of its obvious shortcomings please read it!

When asking questions it should be noted that currently the majority of active contributors are in the Europe so if you visit the IRC channel or post questions to lists the time difference is something to keep in mind.

I carefully said contributor above and not developer, users trying the CI builds and reporting results are welcome...as long as they report useful bugs to the bug tracker. Simon Tatham has produced an excellent resource on this subject.

Also we are always happy to receive translations to new languages (diff against the FatMessages file would be outstandingly useful but anything is welcome), artwork, documentation. Just recall what I mentioned about busy developers. Surest way to get us to see something is the development mailing list, you will probably get a reply, though I will not promise how fast!

Some of the more common mistakes when interacting with the community are:
In conclusion
Hopefully this has been of some use although I had hoped to cover more and provide deeper insights and advice on the codebase but there is only so much generalisation to be done before it is just easier for the developer to go read the code for themselves.

I look forward to lots of new contributions :-) though I fear this may all end up as more of a crib sheet for next time we do GSOC, time will tell.

Vincent Sanders: The way to get started is to quit talking and begin doing

When Walt Disney said that he almost certainly did not have software developers in mind. However it is still good advice, especially if you have no experience with a piece of software you want to change.

Others have written extensively on the topic of software as more than engineering and the creative aspects, comparing it to a craft, which is a description I am personally most comfortable with. As with any craft though you have to understand the material you have to work with and an existing codebase is often a huge amount of material.

While developing NetSurf we get a lot of people who talk a lot about what we "ought" or "should" do to improve the browser but few who actually get off their collective posteriors and contribute. In fact according to the ohloh analysis of the revision control system there are roughly six of us who contribute significantly on a regular basis and of those only four who have interests beyond a specific frontend.

It has been mentioned by a couple of new people who have recently compiled the browser from source that it is somewhat challenging to get started. To address this criticism I intend to give a whirlwind introduction to getting started with the NetSurf codebase, perhaps we can even get some more contributors!

This first post will cover the mechanics of acquiring and building the source and the next will look at working with the codebase and the Netsurf community.

This is my personal blog, the other developers might disagree with my approach, which is why this is in my blog and not on the NetSurf website. That being said comments are enabled and I am sure they will correct anything I get wrong.
ResourcesNetSurf has a selection of resources which are useful to a new developer:
Build environmentThe first thing a new developer has to consider is their build environment. NetSurf supports nine frontends on several Operating Systems (OS) but is limited on the build environment that can be used.

The developer will require a Unix like system but let's be honest, we have not tried with anything other than Linux distributions in some time or MAC OS X for the cocoa frontend because its a special snowflake.

Traditionally at this point in this kind of introduction it would be traditional to provide the command line for various packaging systems to install the build environment and external libraries. We do have documentation that does this but no one reads it, or at least it feels like that. Instead we have chosen to provide a shell fragment that encodes all the bootstrap knowledge in one place, its kept in the revision control system so it can be updated.

To use: download it, read it (hey running random shell code is always a bad idea), source it into your environment and run ns-apt-get-install on a Debain based system or ns-yum-install on Fedora. The rest of this posting will assume the functionality of this script is available, if you want to do it the hard way please refer to the script for the relevant commands and locations.

For Example:
$ wget http://git.netsurf-browser.org/netsurf.git/plain/Docs/env.sh
$ less env.sh
$ source env.sh
$ ns-apt-get-install

Historically NetSurf built on more platforms natively but the effort to keep these build environments working was extensive and no one was prepared to do the necessary maintenance work. This is strictly a build setup decision and does not impact the supported platforms.

Since the last release NetSurf has moved to the git version control system from SVN. This has greatly improved our development process and allows for proper branching and merging we previously struggled to implement.

In addition to the core requirements external libraries NetSurf depends on will need to be installed. Native frontends where the compiled output is run on the same system it was built on are pretty straightforward in that the native package management system can be used to install the libraries for that system.

For cross building to the less common frontends we provide a toolchain repository which will build the entire cross toolchain and library set (we call this the SDK) direct from source. This is what the CI system uses to generate its output so is well tested.
External LibrariesNetSurf depends upon several external development libraries for image handling, network fetching etc. The libraries for the GTK frontend are installed by default if using the development script previously mentioned.

Generally a minimum of libcurl, libjpeg and libpng are necessary along with whatever libraries are required for the toolkit.
Project Source and Internal Libraries
One important feature of NetSurf is that a lot of functionality is split out into libraries. These are internal libraries and although technically separate projects, releases bundle them all together and for development we assume they will all be built together.

The development script provides the ns-clone function which clones all the project sources directly from their various git repositories. Once cloned the ns-make script can be used to build and install all the internal libraries into a local target ready for building the browser.

For Example:

$ source env.sh
$ ns-clone
$ ns-make-libs install
Frontend selectionAs I have mentioned NetSurf supports several windowing environments (toolkits if you like) however on some OS there is only one toolkit so the two get conflated together.

NetSurf currently has nine frontends to consider:
Building and running NetSurfFor a developer new to the project I recommend that the gtk version be built natively which is what I describe here.

Once the internal libraries have been installed, building NetSurf itself is as simple as running make.

For Example:
$ source env.sh
$ ns-make -C $ TARGET_WORKSPACE /$ NS_BROWSER TARGET=gtk

Though generally most developers would change into the netsurf source directory and run make there. The target (frontend) selection defaults to gtk on Linux systems so that can also be omitted Once the browser is built it can be run from the source tree to test.

For Example:
$ source env.sh
$ cd $ TARGET_WORKSPACE /$ NS_BROWSER
$ ns-make
$ ./nsgtk

The build can be configured by editing a Makefile.config file. An example Makefile.config.example can be copied into place and the configuration settings overridden as required. The default values can be found in Makefile.defaults which should not be edited directly.

Logging is enabled with the command line switch -v and user options can be specified on the command line, options on the command line will override those sourced from a users personal configuration, generally found in ~/.netsurf/Choices although this can be compile time configured.

6 March 2013

Keith Packard: composite-swap

Composite and Swap Getting it Right Where the author tries to make sure DRI3000 is going to do what we want now and in the future DRI3000 The basic DRI3000 plan seems pretty straightforward:
  1. Have applications allocate buffers full of new window contents, attach pixmap IDs to those buffers and pass them to the X server to get them onto the screen.
  2. Provide a mechanism to let applications know when those pixmaps are idle so that they can reuse them instead of creating new ones for every frame.
  3. Finally, allow the actual presentation of the contents to be scheduled for a suitable time in the future, generally synchronized with the monitor. Let the client know when this has happened in case they want to synchronize themselves to vblank.
The DRI3 extension provides a way to associate pixmap IDs and buffers, and given the MIT-SHM prototype I ve already implemented, I think we can safely mark this part as demonstrably implementable. That leaves us with a smaller problem, that of taking pixmap contents and presenting them on the screen at a suitable time and telling applications about the progress of that activity. In the absence of compositing, I m pretty sure the initial Swap extension design would do this job just fine, and should resolve some of the known DRI2 limitations related to buffer management. And, I think that goal is sufficient motivation to go and implement that. However, I wanted to write up some further ideas to see if the DRI3000 plan can be made to do precisely what we want in a composited world. The Composited Goal To make sure we re all on the same page, here s what I expect from the Swap extension in a composited world:
  1. Application calls Swap with new window pixmap
  2. Compositor hears about the new pixmap and uses that to construct a new screen pixmap
  3. Compositor calls Swap with new screen pixmap
  4. Vertical retrace happens, executing the pending swap operation
  5. Compositor hears about the swap completion for the screen
  6. Application hears about the swap completion for its window
In particular, applications should not hear that their swap operations are complete until the contents appear on the screen. This allows for applications to throttle themselves to the screen rate, either doing double or triple buffering as they choose. I didn t add steps here indicating buffers going idle or being allocated, because I think that should all happen behind the scenes from the application s perspective. Many applications won t care about the swap completion notification either, but some will and so that needs to be visible. Redirected Swaps? Owen Taylor suggested that one way of getting the compositor involved would be to have it somehow redirect Swap operations, much like we do with window management operations today. I think that idea may be a good direction to try:
  1. Application calls Swap with new window pixmap
  2. Swap is redirected to compositor, passing along the new window pixmap
  3. Compositor constructs a new screen pixmap using the new window pixmap
  4. Compositor calls Swap on the screen and the window, passing the new screen pixmap and the new window pixmap. When the screen update occurs, the screen and the window both receive swap completion events.
This has the added benefit that the X server knows when the compositor is expecting window pixmaps to change like this the compositor has to explicitly request Swap redirection. Window Pixmap Names and GEM Buffer Handles One issue that swapping window pixmaps around like this brings up is how to manage existing names for the window pixmap. Right now, applications expect that window pixmaps will only change when the window is resized. If the Swap extension is going to actually replace the window pixmap when running with a suitable compositor, then we need to figure out what the old names will reference. Are there non-compositor applications using NameWindowPixmap that matter to us? How about non-compositor applications using TextureFromPixmap to get a GEM handle for a window pixmap? For now, I m very tempted to just break stuff and see who complains, but knowing what we re breaking might be nice beforehand. Idling Pixmaps When an application is done drawing to a window pixmap and has passed it off to the X server for presentation, we d like for that pixmap to be automatically marked as discardable as soon as possible. This way, when memory is tight, the kernel can come steal those pages for something critical. Of course, applications may not want to let the server mark the pixmap as idle after being used, so a flag to the Swap call would be needed. Ideally, the pixmap would become idle immediately after the pixmap contents have been extracted. In the absence of a compositor, that would probably be when the Swap operation completes. With a compositor running, we d need explicit instruction from the compositor telling us that the window pixmap was now idle :
 
    SwapIdle
    drawable: Drawable
    pixmap: Pixmap
       
 
Furthermore, the application needs to know that the pixmap is in fact idle. I think that we ll need a synchronous X request that marks a buffer as no longer idle and have that return whether the buffer was discarded while idle. It doesn t seem sufficient to use events here as the application will need to completely reconstruct the pixmap contents in this case. This reply could also contain information about precisely what contents the pixmap does contain.
 
    SwapReuse
    drawable: Drawable
    pixmap: Pixmap
       
    valid: BOOL
    swap-hi: CARD32
    swap-lo: CARD32
 
Pixmap Lifetimes and Triple Buffered Applications If we redirect the Swap operation and send the original application window pixmap ID to the compositor, what happens when the application frees that pixmap before the compositor gets around to using the contents? Surely the Compositor must handle such cases, and not just crash. However, I m fine with requiring that the application not free the pixmap until told by the compositor.

Next.

Previous.