Search Results: "pm"

24 April 2008

Theodore Ts'o: Organic vs. Non-organic Open Source

Brian Aker dropped by and replied to my previous essay by making the following comment:
I believe you are hitting the nail on the organic vs nonorganic open source. I do not believe we have a model for going from one to the other. Linux and Apache both have very different models for contribution but I don t believe either are really optimized at this point. Optimization to me would lead to a system of less priests and more inclusion.
I made an initial reply as comment, and then decided it was so long that I should promote it to a top-level post. I assume that when Brian talks about “organic open source” what he means is what I was calling an “open source development community”. Some googling turned up the following definition from Mozilla Firefox’s organic software page: “Our most well-known product, Firefox, is created by an international movement of thousands, only a small percentage of whom are actual employees.” This puts it in contrast with “non-organic” software, where all or nearly all of the developers are employed by one company. (And anyone who proves talented at adding features to that source base soon gets a job offer by that one company. :-) By that definition we can certainly see projects like Wine, Mysql, Ghostscript (at one time), and others as fitting into that model, and being quite successful. There’s nothing really wrong with the non-organic software model, although many of them have struggled to make enough money when competing with pure proprietary softare competitors, with MySQL perhaps being the exception which proves the rule. In most of these cases, though, the project started more as an organic open source, and then transitioned into the non-organic model when there was a desire to monetize the project and/or when the open source programmers decided that it would be nice if they could turn their avocation into a vocation, and let their hobby put food on the family table. Solaris, of course, is doing something else quite different, though. They are trying to make the transition from a proprietary customer/supplier relationship to trying to develop an Open Source community and what John s candidate statement pointed out is that they weren t really interested in creating an organic open source developer community at all, but they wanted the fruits of an open source community with plenty of application developers, end-users, etc., all participating in that community. We don t have a lot of precedent for projects who try to go in this direction, but I suspect they are skipping a step when they try to go to the end step without bothering to try to make themselves open to outside developers. And by continuing to act like a corporation, they end up shooting themselves in the foot. For example, the OpenSolaris license still prohibits people from publishing benchmarks or comparisons with other operating systems. Very common in closed-source operating systems and databases, but it discourages people from even trying to make things better, both within and outside of the Open Solaris core team. Instead, they respond to posts like David Miller s with Have you ever kissed a girl? . (Thanks, Simon, for that quote; I had seen it before, but not for a while, and it pretty well sums up the sheer arrogance of the Open Solaris development team.) So while Linux may not be completely optimized in terms of less priests and more inclusion, at least over 1200 developers contributed to 2.6.25 during its development cycle. Compared to that, Open Solaris is positively dominated by high priests and with a you may not touch the holy-of-holies attitude; heck, they won t even allow you to compare them to other religions without branding you a heretic and suing you for licensing violations!

Theodore Ts'o: Organic vs. Non-organic Open Source

Brian Aker dropped by and replied to my previous essay by making the following comment:
I believe you are hitting the nail on the organic vs nonorganic open source. I do not believe we have a model for going from one to the other. Linux and Apache both have very different models for contribution but I don t believe either are really optimized at this point. Optimization to me would lead to a system of less priests and more inclusion.
I made an initial reply as comment, and then decided it was so long that I should promote it to a top-level post. I assume that when Brian talks about “organic open source” what he means is what I was calling an “open source development community”. Some googling turned up the following definition from Mozilla Firefox’s organic software page: “Our most well-known product, Firefox, is created by an international movement of thousands, only a small percentage of whom are actual employees.” This puts it in contrast with “non-organic” software, where all or nearly all of the developers are employed by one company. (And anyone who proves talented at adding features to that source base soon gets a job offer by that one company. :-) By that definition we can certainly see projects like Wine, Mysql, Ghostscript (at one time), and others as fitting into that model, and being quite successful. There’s nothing really wrong with the non-organic software model, although many of them have struggled to make enough money when competing with pure proprietary softare competitors, with MySQL perhaps being the exception which proves the rule. In most of these cases, though, the project started more as an organic open source, and then transitioned into the non-organic model when there was a desire to monetize the project and/or when the open source programmers decided that it would be nice if they could turn their avocation into a vocation, and let their hobby put food on the family table. Solaris, of course, is doing something else quite different, though. They are trying to make the transition from a proprietary customer/supplier relationship to trying to develop an Open Source community and what John s candidate statement pointed out is that they weren t really interested in creating an organic open source developer community at all, but they wanted the fruits of an open source community with plenty of application developers, end-users, etc., all participating in that community. We don t have a lot of precedent for projects who try to go in this direction, but I suspect they are skipping a step when they try to go to the end step without bothering to try to make themselves open to outside developers. And by continuing to act like a corporation, they end up shooting themselves in the foot. For example, the OpenSolaris license still prohibits people from publishing benchmarks or comparisons with other operating systems. Very common in closed-source operating systems and databases, but it discourages people from even trying to make things better, both within and outside of the Open Solaris core team. Instead, they respond to posts like David Miller s with Have you ever kissed a girl? . (Thanks, Simon, for that quote; I had seen it before, but not for a while, and it pretty well sums up the sheer arrogance of the Open Solaris development team.) So while Linux may not be completely optimized in terms of less priests and more inclusion, at least over 1200 developers contributed to 2.6.25 during its development cycle. Compared to that, Open Solaris is positively dominated by high priests and with a you may not touch the holy-of-holies attitude; heck, they won t even allow you to compare them to other religions without branding you a heretic and suing you for licensing violations!

23 April 2008

Rob Taylor: Nokia funds D-Bus based Accessibility

Nokia will be funding Codethink in the development of a replacement for AT-SPI based on D-Bus. The aim will be to make a cross-desktop accessibility framework that can be used by both Qt/KDE and GTK+/GNOME applications and other applications that currently make use of ATK. Mark Doffman will be heading up the work and is collaborating with Mike Gorse at Novell who has already started working on this, thanks to Novell’s UIA effort. The main architectural aims will be to cut drastically the amount of of IPC used and generally be lighter than AT-SPI is at the moment. We’ll be working to support both ATK and QAccessible. I’ve applied for a project on freedesktop.org to house this, so when those are up and running that’ll be the main place for discussion. In the meantime, feel free to ask questions on the accessibility-atspi mailing list at the linux foundation or for IRC, #a11y on GIMPNet. Thanks for this goes to Nokia and all in the accessibility community that have supported us in making this happen! Update:
I should also note we will also have a very strong focus on supporting automated testing tools, so anyone out there working in that area is more then welcome to join in and tell us what needs fixing with AT-SPI.

22 April 2008

Michal Čihař: Two phpMyAdmin projects in GSoC 2008

Well I did not expect that much when MySQL offered us to piggyback on their participation in GSoC 2008, but at the end we have two projects. One is about blobstreaming which I absolutely don't know and is probably some amazing new technology :-). Marc will act as a sponsor on this together with some guys from http://www.blobstreaming.org. The second project is setup rewrite where I will act as a mentor and Piotr Przybylski will implement it. Hopefully we will bring this to successful end and soon phpMyAdmin will bring you easier way to set up and configure it's options.

Joerg Jaspert: FTPMaster

[Warning, long post ahead. If you aren’t interested in Debian or it’s internals or my work - skip it. :) ] So as you might have read here, I got the delegation of the debadmin AKA FTPMaster group lately. I’ve got some pressure to not accept this delegation, but the response to my blog post about it had been extremely positive, everyone said I should not deny it. Knowing to have such a support within Debian is something I really appreciate, making it worth to spent time on my various jobs. Now, lets take this and write a little about what happened since I gained that extra group: Work I did since then: The above should nicely sum up my work in the past few days. Most of it pretty trivial, but still things that have to be done. I plan to keep doing things, slowly going from the trivial stuff to more complicated ones, but there is lots of stuff ahead. Some ideas, and this is a very incomplete list just from my head, there are tons of things that one could do, include Now, do you have any time to kill? Do you want to help? Some things you could do:

21 April 2008

Neil Williams: estron - data-centric development interpreter for GNOME

My previous post has led to a new name and a lot of improvements in the codebase. estron is the new name for DWI. With help from ikiwiki and Alioth, estron has a new home, a new wiki and a fresh start. I've implemented -Werror fixes across the codebase (with the sole exception of some old Gtk2 code that is now deprecated).

As is my regular practice, make distcheck is now working (although some actual tests need to be added) and I've added versioned symbol support. estron can work with libqof1 but it will be better served with libqof2 and that's probably for the best as that means I have time to finish estron to coincide with the libqof2 transition after Lenny. As mentioned in the ITP, if anyone wants to play with the code before then, I'll make an upload to experimental if you just ask (or you can just build estron from Debian SVN).

Thijs Kinkhorst: The problem with Planet Debian

There is some discussion again about what should be on topic for Planet Debian, whether it should be used for Debian-related announcements and what the archival policy should be. I believe that this discussion is rooted in two circulating views of what Planet Debian should be:
  1. A forum for DD's to post Debian-related thoughts and ideas, to be used for those cases where they think a mailing list post may be less appropriate;
  2. A way to collect stories about what DD's do and think in any context, the human interest angle: get to know your Debian peers more than just in technical matters.
Although I always regarded Planet Debian to be the latter and I appreciate reading posts telling me about someone's job or other interests, either one isn't an invalid approach per se. They do however prompt different choices about listing policy (Debian-only feeds vs. everything goes), how much effort it is to keep up with them and perhaps archival. The best way to match the expectations of those two groups would be to create two "planets", one for each group. Create an aggregator where any DD can add any of their feeds, and another one with a strict listing policy: only feeds that carry posts actually related to Debian development matters (probably a subset of the first). Then only the question remains which one will get to carry the Planet Debian name...

19 April 2008

Jeff Bailey: Physio, Illness

I started physiotherapy this past week. The first clinic I went to on Wednesday (Mountain View Physical Therapy) turned into an interesting disaster. When we walked into the place, the first impressions were that it was noisy and dirty. There was someone sitting upright in the waiting area asleep. The place is one big room, with the massage tables in the middle of the room and exercise equipment all around the outside. None of the staff looked really happy to be there. I met my therapist who, it turns out, is the director of the clinic. She was truly excellent - good enough that I would've overlooked everything else, except for the fact that she doesn't have 3 days a week free to see me. More weeks, I would've been lucky to get her more than once per week.

So for Friday, we found a different clinic. Orthosport. It was such a difference from the previous place, there's no good comparison. The receptionist was friendly, the therapist very professional, and the place really tidy. Even the assistants were very friendly and checked in on a regular basis.

But in between those two days, I got sick again. I had some sort of stomach flu or drug reaction the previous week, and it seemed to be coming back. After two days of throwing up, I made an appointment with the after-hours clinic today to get more anti-nausea drugs. They've also taken some more blood and want to do some tests next week to make sure that everything's still doing okay.

Another bit of love and affection goes to my new bank, RBC Centura. Not only have they done lovely things for bill payments, and sending money for free between my Canadian and American accounts, they also have a "Bank by Mail" program, where I can just stick cheques in an envelope that's already postage paid, and they deposit them for me. No need to find a machine of the right persuasion to do deposits. We still have a couple tasks to do before we can close out the HSBC account, but we're almost there.

Leif is pushing out new teeth. He'll be up to 9 when he's done.

Theodore Ts'o: What Sun was trying to do with Open Solaris

I was recently checking to see what, if any follow-up there had been from Sun’s ham-handed handling of the Open Solaris Trademark, and I ran across this very interesting comment from Jon Plocher’s Candidate Statement for the Open Solaris Governing Board:
“I also think there was a misunderstanding about what Sun desired when it launched the community (in part) to encourage developers to adopt and use Solaris. My take is that, while there *is* value in getting more kernel, driver and utility developers contributing to and porting the (open) Solaris operating system, there is significantly *more* value in having a whole undivided ecosystem based on a compatible set of distributions, where application developers, university students, custom distro builders and users are all able to take advantage of each other’s work. Put these two things together, and you can see Sun’s predicament. Sun *wanted* a community that empowered application developers, but *got* a community aimed squarely at kernel hackers. Whether you see this as the “kernel.org -vs- Ubuntu” fight, or the “fully open -vs- MySQL model” argument, in my opinion, it all is simply a reflection of the above mismatched expectations.”
So that explains why it’s take three long years to try to get basic open source development tools (such as putting Open Solaris source code in a distributed SCM located outside of the Sun firewall) for Open Solaris. It was never was Sun’s intention to try to promote a kernel engineering community, or at least, it was certainly not a high priority for them to do so. This can be shown by the fact that as of this writing they still are using the incredibly clunky requester/sponsor system for getting patches into Solaris; setting up a git or mercurial server is not rocket science. This lack explains why Linus gets more contributions while brushing his teeth than Open Solaris gets in a week. So if you run into a Sun salescritter or a Sun CEO claiming that OpenSolaris is just like Linux, it’s not. Fundamentally, Open Solaris has been released under a Open Source license, but it is not an Open Source development community. Maybe it will be someday, as some Sun executives have claimed, but it’s definitely not a priority by Sun; if it was, it would have been done before now. And why not? After all, they are getting all of the marketing benefit of claiming that Solaris is “just like Linux”, without having to deal with any of the messy costs of working with an outside community. As a tactical measure, astroturfing is certainly a valid marketing trick. But after three years, the excuse of “just you wait a little longer, we’re just trying to figure this open source community stuff out”, is starting to wear a little thin. Furthermore, if (as Jon Ploncher claims) this was about “empowering application programmers”, why was it that Sun’s first act was to trumpet how wonderful it was to release the Solaris source code under a Open Source license? This only seems to make sense if the Open Solaris initiative was really a cynical marketing tactic to try to save Solaris from being viewed as irrelevant. If that was Sun’s intention, I think it is fair to say that from a marketing point of view, the tactic has been at least partially successful — although as Jon has admitted, the goal of creating a full community with application developers, university students, and so on, hasn’t materialized for Open Solaris. Sun has the dream; the Linux community is living it. However, from business standpoint, I wonder if Sun will really be able to sustain their Solaris engineering team if they will really be doing all of the work themselves, and outside contributions continue at the rate of 0.6 patches per day. After all, the margins when you are selling low-cost AMD servers are much lower than when you are selling ber-expensive SPARC servers. With Linux, we have a major advantage in that kernel improvements are coming from multiple companies in the ecosystem, instead of being paid for by a single company. And given that 70-80% of Sun’s AMD servers are running Linux, not Solaris, it’s not clear how Sun justifies their Solaris engineering costs to their shareholders. Furthermore, if Solaris on x86_64 were to actually take off, there’s nothing to stop competitors from selling Solaris support — except the competitors won’t have to pay the engineering costs to maintain and improve Solaris, so they would be able to provide the support much more cheaply than Sun could. So while Sun’s marketing tactics have kept Solaris alive in some verticals, I have to question how successful Sun will be in the long-term.

Christian Perrier: Samba week

I spent the entire week at the SambaXP conference in G ttingen, Germany. The conference is the annual conference of the community of developers, contributors and users of Samba. I general call it "my annual german pilgrimage" as I attended all seven editions of the conference, the only FLOSS conference I attend on my work time, as my daily work involves quite a lot of uses of samba. This year featured a workshop or training session held by John Terpstra, longstanding FLOSS evangelist and member of the Samba Team for over 10 years, and Karolin Seeger, the brand new release manager of Samba. This has indeed been an incredible opportunity to have discussions with them about the packaging work for Samba. Actually, the work we did in the last 3 years to bring Samba Debian packages in a very good shape (when I don't screw up) is much appreciated. I think these days have been another opportunity to keep that link very closed. I have been impressed by the promising work of Karolin with respect to the preparation of the release and the very serious way she has to do that work....as well as her very friendly, while still discreet approach to technical discussion. Some work was done on Samba package bugs, though less than usually (the remaining ones are harder and harder to tackle!). I mostly work on Debian packages as well as .deb packages prepared by Sernet (the services company that organises the conference and employer of some Samba Team members), as these could some day become the packages provided by the Samba Team. Talks at the conference were pretty interesting, to keep connected with Samba 3.2 and 4.0 development. News from Andrew Tridgell about exchanges with Microsoft (yes, The Evil) and access to MS documentation, are very promising. The collaboration between Samba developers and Microsoft enginneers is now working well again, at the engineers level (as Tridge says: "lawyers are away, now, we can talk at the engineers level and restore the link that existed in the early 90's"). I also could measure the progress of the Openchange project whose ultimate aim is to provide a complete Microsoft Exchange replacement solution. They currently have working MAPI libraries and an Evolution plugin for Exchange is under development, while the bricks to build a server are patiently being put together. Good discussions with Julien Kerihuel, the lead (and French) developer and manager of the project. Jelmer Vernooij and I also settled the final plans to get Samba4 packages in Debian. I proposed to make a first upload to experimental, but quite soon to upload to unstable, the point being a largest as possible exposure of that code, so that upstream developers (Jelmer himself, Andrew Bartlett and a few others...) get as much feedback as possible. That upload will not be targeted for lenny (we'll block it from entering testing). For that reason and also because Samba 3 and 4 will certainly coexist for years, the source package will be named "samba4". Expect more news quite soon. And final conclusion of that week: I've also been delighted to be able to visit Alex and Meike in Hildesheim, as well as Andreas in Wernigerode. It's always a great pleasure to see longstanding Debian friends. At the beginning of the week, I planned to see *two DD and finally, at the end of the week, I can tell I've seen *three* Debian developers, finally!

18 April 2008

Axel Beckert: Harddisk prices gone mad

Cut and paste from Brack’s SATA harddisk pricelist: 
Samsung SpinPoint S166,  HDD,  80GB, 7200rpm, 8.9ms,  8MB Cache, SATA II NCQ, OEM, 3.5'', SAH-HD082GJ   CHF  53.00
Samsung SpinPoint S250,  HDD, 250GB, 7200rpm, 8.9ms,  8MB Cache, SATA-II NCQ, OEM, 3.5'', SAH-HD250HJ   CHF  64.00
[...]
Samsung SpinPoint T166s, HDD, 400GB, 7200rpm, 8.9ms, 16MB Cache, SATA-II NCQ, OEM, 3.5'', SAH-HD403LJ   CHF 109.00
Samsung SpinPoint T166s, HDD, 500GB, 7200rpm, 8.9ms, 16MB Cache, SATA-II NCQ, OEM, 3.5'', SAH-HD501LJ   CHF 101.00
So from 80 GB to 250 GB the price difference is less than CHF 10 and 500 GB harddisks are cheaper than 400 GB harddisks of the same type? We live in a strange world. And no, none of this harddisks is marked as special price or promotion. Oh and for all those not having CHF as your daily currency:
CHFEURUSD
1.00€0.626417$0.997506
1.59638€1.00$1.5924
1.0025€0.627983$1.00
(Rates from x-rates.com.)

17 April 2008

Neil Williams: Data Centric programming - a new start at Alioth.

It's time for me to stop complaining about the commercialisation of SourceForge and vote with my code. I'm leaving for brighter, bluer pastures at Alioth.

Query Object Framework is first (libqof1). The others will follow. I'm leaving because of unusable mailing list archives, impossibly slow web pages and wiki and the general amount of work involved in the SF interface. I'm gaining sane mailing lists, a friendly and usable ikiwiki and a simpler (although very similar) interface.

I'm also migrating most of those packages to subversion. With so many other changes, it's not exactly a problem to add cvs2svn into the mix.

Now, of course, this does not mean that these upstream packages suddenly become Debian-only or Debian-native. It does mean that I have a chance of encouraging new developers to join the team by providing the kinds of services and support that people need when starting off with a new project. (I suppose the one thing I don't yet have is an IRC channel. Not sure if it is worthwhile but I'll see what happens.)

QOF will form the basis of the support available for each of the other projects - all will share the same low-volume QOF-devel mailing list and QOF Wiki as provided by Alioth. I'll also migrate QuickList because I'm looking at restarting development on a stalled project allied to QOF and which could provide missing functionality in QuickList. I did look at Gaby in this mix but a botched Gtk1->Gtk2 transition that ended up with two incompatible forks running different parts of the original codebase put me off completely (and neither of the forks actually worked). (Some packages deserve to be dead upstream.)

I'm looking for new blood, fresh faces and a new desire to take these projects forwards. The driving force is data freedom - creating data-driven applications that are completely agnostic to where and how the data happens to be stored. It is user data, it should be as free as the source code that wrote it. Who cares what format it is in? Let's get it into a format that some other program can use and make use of the existing data instead of duplicating it all the time. (The observant will note that my old haunt of GnuCash is stubbornly ignorant of the merits of a decent import/export implementation that actually covers more than 50% of the data in the application. No wonder I took the chance to sponsor homebank.)

DWI Data With Interaction (aka DUI -- Data Under the Interface) is what can provide the missing interfaces. It'll take a while to get it back up to date and available in Debian but it can provide the link between data and interfaces. GNU/Linux users should be able to design an interface purely on the basis of the data that it needs to handle and not care one jot about how that data comes from one free software app and goes on to the next free software app. QOF is gaining a connection to libgda which means that data can go to and fro between a whole range of databases and file formats. Homebank and pilot-qof can gain new data inputs (and possibly outputs) by utilising the existing import/export support in new ways via DWI/QOF (as libqof2). That can give a complete interface for all PIM and financial data for a typical SOHO user and all based on SQL-type queries so that every stage of the process can be tweaked and optimised according to user requirements at runtime.

(Where does Emdebian come into this? Well, many people like me need to record and process PIM/financial data on the move so Emdebian means that those users can have the same data processing on their handhelds as on the desktop. Smaller apps to capture the data quickly and efficiently whilst on the move; larger apps to refine, optimise and filter the data on the desktop entirely under user control but with a simple, scriptable, interface based on runtime queries so that the user can do exactly the same conversions tomorrow with new data but none of the configuration. Easy!)

Is this groupware ? No - it is personal, ultimately flexible and entirely customised for particular users by the users themselves. It's about small niche markets, about independent business people with bespoke and highly specialised requirements. No one is going to spend time writing their reports in Scheme or Haskell, because they are the only people who need that particular report. Instead, that individual user needs to be empowered through a simple and intuitive interface to take a piece of data from one source, correlate it with a static piece of data from individual configuration sources, calculate a result according to a third piece of data from a completely separate source and combine them all into a new set of objects that can be used in other queries. It's about preparing invoices from Palm data, it's about handling customer accounts for a one-man company or handling a stocks and shares portfolio for a partnership. Ultimately customised, ultimately flexible and completely format-agnostic.

These are the kind of applications that hold people back from GNU/Linux migrations - bespoke applications that are developed for just a few small companies or just a few situations. Free software finds it hard to provide those apps because free software depends (currently) on someone else being motivated to write a bespoke program and that person rarely understands the niche. This is a chance to put the development into the hands of those with the motivation to do the work, by making the development process more WYSIWYG.

The question is: Can we come up with a better name for DWI, dear lazyweb?

"DWI is a fairly simple environment for quickly creating data-driven applications, that is, graphical applications that manipulate and show info from a database. This environment differs from others in that it is focused on native GTK/Gnome support through the Glade GUI designer, and thus allows you to build user interfaces as elegant as you can make them in Glade."
http://dwi.sourceforge.net/

The big appeal is that DWI is database-agnostic - it only cares about the data structure, not the file format. Ally that with QOF which provides data-centric SQL-based queries and conversions between objects (including combining elements from one object into one or more other objects without having to hardcode the connection between the objects at compile time) and we have an interesting and powerful combination of engines, frontends and backends that are all data-centric.

Please, if this sparks your interest, join the QOF-devel mailing list, have a play with the ikiwiki at http://qof.alioth.debian.org/ and have a go with the code. This whole area is begging for development but it is just so far from being usable by the target audience at this stage. There is a lot of programming to be done by current developers before the principles can be let loose on the masses.
:-)

This is about working between existing projects and between distributions - no longer dedicated to a single application, platform or file format, no more tunnel vision. Work on the edges of existing applications (and indeed projects), remove the sharp edges of applications that should work together but can't. The majority of the code in these libraries and applications is already stable and mature. Like so many parts of free software, it is time, IMHO, to put aside our egotistical devotion to the one-true-executable and come up with an actual solution.

Bah! I'm beginning to sound like a PR exec instead of a dedicated geek. What does it take to motivate people and get things done around here?
:-)

Matthew Garrett

This week I'm in Mountain View for the X Developers' Conference, where I gave a presentation on things graphics drivers have to care about in order to make suspend/resume work reliably. Various people suggested that I write up an explanation of the current situation, along with some best practices for manufacturers.

One of the major differences between the ACPI specification and the older APM specification is that saving and restoring hardware state is generally left up to the operating system, with the firmware playing a much more passive role. Linux is now able to handle this state restoration for the majority of hardware, but we still have issues with graphics. Modern graphics hardware can be hugely complicated, which to a large extent we've been able to ignore because the platform will initialise it on boot. Many X drivers know how to switch a card from text mode to graphics mode (and vice-versa), but that's not adequate.

What can make this situation even more awkward is that the ACPI specification makes no guarantees about what state the hardware comes up in. It may still be in PCI D3. It may be powered up, but the registers may all be zeroed. It's even valid for the hardware to come up in text mode. A driver has to be able to deal with all of these situations.

So far, we've mostly tried to deal with this situation by running chunks of the video BIOS with applications like vbetool. This is hacky and often works badly - in some cases, it won't work at all (nvidia rewrite the entry point to their video BIOS after boot, in order to prevent situations where someone attempts to POST the card and it tries to jump to bits of the system BIOS that no longer exist). The only long-term option is for the kernel to handle the graphics suspend/resume. There's several levels at which this can be supported:
Since 2.6.25, i915 and later Intel hardware can manage basic mode reprogramming without requiring any external assistance. The radeon kernel modesetting code is also heading in this direction, and we ought to be able to manage something there before too long. The nouveau developers are heading towards the stage of being able to do native mode programming, so there's some hope of progress there as well.

It's not all plain sailing, though. Vendors can wire up hardware in different ways. AMD and Nvidia hardware provides scripts in the ROM that, in theory, abstract this away. The downside to this is that we'll have to put the interpreters for these scripts in kernel space, though this is probably needed for kernel modesetting work anyway. Even then, it's possible that we'll need workarounds for specific pieces of hardware that don't quite behave as we expect them to.

As far as making graphics work over suspend/resume, we have a pretty good idea where we are now (failing on anything other than modern Intel hardware), where we need to be in the future (triumph, huge success, cake) and how to get there (?). However, hardware manufacturers can still help.
Chipset vendors:Tell us how to reinitialise your hardware. If that involves parsing BIOS tables, then please let us know how to do that. Expecting us to jump into your ROM and execute x86 code is ok if that's genuinely the only way to do it, but for any modern hardware that's clearly not the case.
System vendors:Try not to alter your system behaviour too wildly. Requiring magic register writes in order to light up the VGA output does not classify as value add. Please don't have your firmware program text mode directly - we're trying to get away from that. And do not attempt to detect that you're running on Linux and alter your system behaviour. If it's possible for you to tell that you're not running on Windows, that's a bug in Linux. If you rely on that bug, you will break. I'm not joking here in the slightest. The closest thing to a "standard" interface between an OS and the hardware is whatever Windows does, and we're basically aiming for bug-for-bug compatibility there.

David Pashley: Using In-memory tarballs with Archive::Tar

Archive::Tar is a useful library for working with tar archives from Perl. Unfortunately, one thing it doesn't allow is using data from memory as the archive. From the TODO section:
Allow archives to be passed in as string Currently, we only allow opened filehandles or filenames, but not strings. The internals would need some reworking to facilitate stringified archives.
Fortunately, it does allow you to use a filehandle. I've previously mentioned about how useful the IO::Handle subsystem in perl is. And we should be able to use it in this case. The module we'll want is IO::String, which is a IO::Handle over a perl scalar. We can use it: 
my $tar = new Archive::Tar(new IO::String($data));
Unfortunately when we run this now we get: 
Cannot read compressed format in tar-mode at Foo.pm line 41
No data could be read from file at Foo.pm line 41
It turns out that this is because Archive::Tar uses IO::Zlib internally if the file isn't uncompressed, but this doesn't provide the ability to uncompress from a filehandle. The answer is to uncompress the data before passing it to Archive::Tar and the easiest way to do this is to use the IO::Uncompress::Gunzip module, so we can rewrite our code to: 
my $tar = new Archive::Tar(new IO::Uncompress::Gunzip(new IO::String($data)));
Now when you run the script, Archive::Tar has an uncompressed tar stream. Yet another situation where IO::Handles comes to the rescue.
Read Comments (1)

Joerg Jaspert: debadmin delegation - should I accept or deny it?

As you might have read on debian-devel-announce lately, Sam delegated the ftpmaster rights to me. Steve, our new DPL, did state his full support in an IRC query minutes later. Now I got a query from someone who is, pretty strongly, argumenting that I should decline this delegation. Mostly on the grounds that it is not nice to force new team members into an existing team, and that it is not a good environment to work in, if a DPL can do that if he feels like proving a point. Leaving alone the fact that delegations are one (central) part of the DPL job, and that the constitution leaves a way if you do not like those delegations (GR), I’m now unsure what to do. (Just for the record - i once was bitten by a DPL using his powers in a way I absolutely did hate (and still do). Some of you might remember the fun around the DebConf7 venue decision). I could do
  1. decline the delegation, stay ftp assistant only. Which I am for a long time now. Possibly see some flame on the lists when I do this.
  2. do nothing, ie. accept the delegation. That will get at least one of the long time ftpmasters upset, possibly more of them. Maybe with a resign from that job. Or Debian. But it would enable me to do the work only ftpmasters can do, like changes to the code, changes to the archive directly, etc. Without delays while waiting for a master to comment on something I prepared. Good example here is the current situation with the release teams transitions: They are 90% done and merged, just one little more merge is waiting to happen before they are ready to use… And whenever I tried to get a master (mostly Aj, as he was working with me on this) to do the final steps I got nothing but silence back, even when he was otherwise active…
Bah. I hate this. The thing is - I really do want to have debadmin rights, and am working in that direction for a long time now. Not publically, but by talking with James in the past months, … As Sams mail showed - that did go nowhere, unfortunately. Now, could readers of my blog (and I know there are people reading this, my traffic stats tell me so :) ) please comment on this and give me advice? I just want some opinions about this. (Yes, my blog does understand comments since some time :) ) UPDATE: Thanks to everyone who commented on this, either by using the comment form here, by mail or via IRC. I really appreciate your input. I’ve decided to not decline the delegation and one DSA member already added my account to the debadmin group. (Thanks).

16 April 2008

Simon McVittie: Implementing D-Bus services with telepathy-glib

16:46 < epmf> tp-glib is made of magic

—#telepathy, 2008-04-15

Rob pointed out today that I'd been promising to blog about telepathy-glib for several months and still haven't done so. I think there's too much to cover in one blog post, so I'll start with the oldest part - implementing a service (typically a Telepathy connection manager).

While telepathy-glib is (obviously) intended for Telepathy implementors, I think the ideas we've been implementing are likely to be useful for any D-Bus API, particularly flexible/complex APIs where an object implements many interfaces.

Introduction to telepathy-glib telepathy-glib started out as a collection of code extracted from our XMPP connection manager, telepathy-gabble, but at some point it grew beyond that into a wrapper around/partial replacement for dbus-glib. We've been quite pleased with it, really :-) My first major round of work on telepathy-glib, during the first half of 2007, was to split it out from telepathy-gabble (versions up to 0.5.9 were released as part of Gabble 0.5.x tarballs, in fact), leading to the release of the 0.6.x stable branch in late September. This was very helpful for implementing connection managers - Salut, Idle and telepathy-sofiasip (our connection managers for link-local XMPP, IRC and SIP) all started off by forking/cargo-culting from Gabble, and achieved huge code reductions by porting to telepathy-glib. Also, Will Thompson used it in his Summer of Code project, telepathy-haze, to go from nothing to a working and useful Telepathy implementation based on libpurple in a matter of a few months. The second major round of work, which I'll discuss in a later article, added client code to telepathy-glib, obsoleting the older/worse libtelepathy and allowing client programs like Empathy and telepathy-stream-engine to be written using telepathy-glib. In the process, I accidentally reinvented DBusGProxy :-) More details to come later!
Some background: service-side code generation
The secret Collabora code-generation process

(credit: daf)

Supporting D-Bus in the GObject world has always involved quite a lot of code generation. The core API of dbus-glib is heavily reliant on varargs functions, which aren't type-safe and are easy to get wrong. dbus-glib contains a program called dbus-binding-tool, which is meant to generate reasonably sane GObject APIs for D-Bus objects. Unfortunately, it seems to be intended to generate a whole GObject at a time. Early versions of telepathy-gabble used dbus-binding-tool plus a script called gengobject.py to generate API stubs for the exported objects; developers then filled in the blanks, and hey presto, we had a GObject on D-Bus. This was fine up to a point, but had a couple of major drawbacks. Whenever we changed the D-Bus API (quite common during the early development of Telepathy), there was a very laborious and error-prone merging process. We ended up with the following process:
  • copy the D-Bus introspection XML from the telepathy-spec repository into a directory xml/pristine (actually, the canonical form for the spec was Python for a while, and we had a script that exported a contrived object onto D-Bus and introspected itself to get the XML - but that's another story!)
  • preprocess the introspection XML to add the dbus-glib CFunctionName annotation, resulting in a directory xml/modified
  • run dbus-binding-tool and a Python script gengobject.py to generate C source in xml/src
  • three-way-diff the old version of xml/src, the new version of xml/src, and the real C code in src to create a new version of src
  • pray that the diff process hadn't randomly exchanged functions' implementations
  • update the resulting code in src so it worked
  • check the whole mess back into darcs
  • hope that nobody else had made changes that would result in darcs conflicts
If you haven't yet gathered, this was a bit of a nightmare. Also, it was very easy to return the wrong thing from a method without noticing - because all the APIs were varargs, the compiler didn't notice, and the first we'd know about it was a mysterious segfault under certain circumstances.
How telepathy-glib fixes code generation telepathy-glib improves on this by taking advantage of this innocuous-looking feature in dbus-glib: 
commit 355ef78d98d6fc65715845d56232199162cab12a
Author: Steve Frécinaux <steve istique net>
Date:   Thu Aug 17 11:48:20 2006 +0200
    Interface support for bindings.
Instead of running dbus-binding-tool or creating GObjects with D-Bus interfaces, we put the entire Telepathy spec through glib-ginterface-gen.py, a distant descendant of Gabble's gengobject.py. For each interface in the spec, we generate a GInterface that mirrors the D-Bus interface. Any GObject that implements the GInterface automatically gets the D-Bus interface if it's exported onto D-Bus. There are a few non-obvious refinements, though:

  • The signature of the method implementation is always in the mode that dbus-glib calls "asynchronous", where the method implementation can either send a reply message before or after it returns. For "slow" methods, this is the only thing you can do. For instance, many Telepathy D-Bus methods can't return anything until some TCP round-trips to the server have happened. This is also the only thing you can do if you want to extract extra information, like the sender's unique name, from the method-call message (dbus-glib's "synchronous" API doesn't allow this to be done). For "fast" methods, sending a reply message before returning is just as easy as using the "synchronous" API (it's just a difference of syntax), and has an API consistent with that of the "slow" methods, making it easier for service authors.

  • The layout of the GInterface vtable is private, and (auto-generated) accessor functions are used to fill in implementations. This means our ABI doesn't change just because we re-ordered functions in the spec.

  • If no implementation is provided for a method, we just raise an appropriate error (org.freedesktop.Telepathy.Errors.NotImplemented), rather than suffering an assertion failure. This means we can safely add methods to an interface.

  • While we're generating code anyway, we generate some static inline wrapper functions which wrap dbus_g_method_return(), to have type-safe method returns. You can easily check that a method replies with correct types, by checking that the implementation of Foo() replies by calling tp_svc_some_interface_return_from_foo().

  • Similarly, we generate wrapper functions to emit signals, to get type-safe signal emission.

Not all interfaces are stable enough to be included in telepathy-glib's stable API and ABI, so some of our other projects include a copy of the telepathy-glib code-generation tools, and generate their own "mini-telepathy-glib" (traditionally in a /extensions/ directory) for their implementation-specific (drafts of) interfaces. This isn't yet as polished as it ought to be (mainly because we don't want to freeze the augmented-introspection-XML format that we write the Telepathy spec in, because it's rather ad-hoc and hackish in places) but it works quite well in practice. To see what this looks like in practice, have a look at the examples in telepathy-glib, or at a Telepathy connection manager like telepathy-gabble.
Mixins and base classes Another feature of telepathy-glib which makes life easier for connection manager authors is that it provides "mixins" and base classes which implement some of the GInterfaces. There's absolutely nothing magic about the base classes - they don't have any access to private implementation details of the GInterfaces, they just implement them like everyone else does. The "mixins" are only slightly more magical - they store some extra state inside the structures representing GObjects and their classes, and use it to provide default implementations of some or all of the methods on a particular interface (or two interfaces, in the case of the new TpMessageMixin). I'll leave it up to Rob to explain those in greater detail, since I seem to remember that he invented them :-P The examples and regression tests in telepathy-glib are quite good examples of how these work in practice.

Russell Coker: Motivation and Perspective

Patrick Winnertz writes about the demotivating effect of unreasonable delays on joining the Debian project [1]. While I agree that things need to be improved in terms of getting people in the project in a timely manner (the suggestion of providing assistants seems good), I don’t think that anyone has a good reason for being demotivated because of this. I first applied to join Debian in late 1998 or some time in 1999. At the time part of the process of joining was to receive a phone call. At the time I was living in a hotel and they refused to call me on such a line. I could have easily camped out in the hallway of a hotel (the cheap London hotels often had a pay-phone in the hall and no phones in the rooms) and pretended that it was my own phone with an unlisted number. Unless they refused to allow people with unlisted numbers to join (which seems unlikely) then I could have joined then. So it seems that at the time I could only join Debian if I was prepared to lie about my ownership of a phone line. I wasn’t overly bothered by this - there has never been a shortage of free software projects that need contributions of code. By late 2000 the rules had changed and I joined without needing a phone call. In the mean time I had forked the Bonnie storage benchmark program to form my own project Bonnie++ [2], created Postal - mail server benchmark suite [3] and worked on many other things as well. I have sympathy for the people who apply to become Debian Developers and who have to wait a long time, I’ve been in the same situation myself. But there are plenty of things that you can do in the mean time. Some of the things that you can do are upstream development work, filing bug reports, submiting patches that fix bugs, and writing documentation (all forms including blog posts). Also when projects aren’t yet in Debian it often happens that someone creates unofficial packages, the person who does this doesn’t need to be a DD. Producing back-ported packages for new versions of programs that are in a stable release can also be done by people who are not DDs. Unofficial and back-ported packages provide less benefit for the project as a whole but considerable benefit for the people who want to use them. There is a lot of work that can be done to fulfill clause 4 of the Debian Social Contract [4] (Our priorities are our users and free software) which doesn’t require being a Debian developer. It seems to me that if you have the right approach to this and maintain the perspective that Debian is one part of the free software community (and not necessarily the biggest or most significant) then a delay in your application to become a DD won’t be particularly demotivating.

Share This

MJ Ray: Better Free Software Organisations?

Another zero-day announcement of a Software in the Public Interest online meeting has been posted. The announcement mentions "one motion has been raised" but the agenda doesn't include any motions, so I'm not sure what. Also, the agenda lists "Debian logo licence" as up for discussion but I thought last month's meeting resolved that. I watch SPI fairly closely and I'm confused. How about the rest of you? SPI isn't alone in this: many software organisations seem to suffer from similar problems. For example, I ranted in passing about Free Software Manchester yesterday, who just posted the notes from their own zero-day meeting and I've suggested possible ways of reforming debian's currently-stalled New Maintainer process more than once. Are free software users particularly bad at the basics of running an interest society (like welcoming and expiring members, calling meetings, publishing routine communications, and so on), have I been spoiled by cooperatives with their friendly Member Services departments or secretariats, or what? Is this why so many free software orgs seem to include self-perpetuating leadership groups? Is this a serious problem if, as reported, Software Development is a Team Sport [etbe]? Are there fully-working free software mass participation groups out there? I feel a lot of these problems are caused by attempting to order our inherently entropy-filled world completely and insisting everything follows petty rules, such as refusing to answer a question because the "wrong" member asked it. The world will not become less random just because hackers try to impose arbitrary rules. Sometimes it's good to put down minimum standards (because calling zero-day meetings is a mostly-avoidable way of excluding some members) but it will always be a poor alternative to trying to do the best you can for others. How do we get past this? My pro-cooperation-and-better-business platform for SPI board went pretty badly and I've had some anti-cooperative flames back from someone starting another free software group this month, so I don't think I can fix these existing organisations any time soon. About 1 in 6 people in the UK are members of a cooperative, so even if that is reflected among hackers (and I think it's lower), all of them would not be enough to reform much. One of the most common memes in free software is "show us the code" and the few other free software cooperatives I've seen have mostly failed, with a few surviving but hitting a size limit. As a result, I'm currently negotiating the start of a new free software cooperative. So far, I'm really happy with how that's going. Many cooperators learn at the feet of large consumer cooperatives like the Cooperative Group who run training courses for new members about putting cooperative values and principles into practice, which we smaller groups couldn't run ourselves. As a result, most cooperators already know how to work well together. Should large software societies like SPI try commissioning similar courses?

Miriam Ruiz: More on OOXML: ISO tries to recover lost credibility through press notes

After the unfortunate show that Microsoft and ISO provided us recently, ISO’s response is a FAQ note about ISO/IEC 29500 (OOXML, that’s it) in which they threaten Microsoft with the most severe of penalties if they’re tricking on as. Funny. In that statement, they try to answer all the issues that initially were concerns but that now has turned into proven facts about the OOXML standard and it’s approval, and to try to recover some of their lost credibility. It’s sad to be commenting on this, words don’t come easy, as if to say. Microsoft has already proven who’s the one in charge here, who owns the committees, who takes the decisions and in which terms, and nothing happens. It’s gonna take more than to publish a FAQ note saying how well you did it and to threaten Microsoft to behave or else. No one seriously believe you at the moment, you know? A funny thing is that ISO dismisses any responsibility about standardization, and relies on the market taking the decisions for them. Microsoft has said many times that they consider something positive and healthy to have multiple standards for the same, and let the market decide. Now ISO declares to think along the same lines: “After a period of co-existence, it is basically the market that decides which survives”. We don’t really need an organization for standardization for that, do we? We already have the market for that. Will ISO apply the same policy for every standard from now on? Like, say, having a couple of competing standards at least for every need, and let the market decide? ISO tries to recover from their credibility loss with this note (“the standards development process is credible, works well and is delivering the standards needed, and widely implemented, by the market“), without much success at least in what I’m concerned. If they keep trying to convince with press notes instead of with real acts, no one is gonna believe them ever again.

15 April 2008

Russell Coker: Software Development is a Team Sport

Albert writes about software development and how much teamwork is used [1]. He makes an interesting clash of analogies by suggesting that it’s not a “team sport” because “its not like commercial fishing where many hands are used to pull in the net at the same time“. I think that software development for any non-trivial project is a team sport. You don’t have the same level of direct coordination as required for pulling in a net (or the rugby scrum [2] to use a sporting analogy), but that doesn’t stop it being a team effort. Some parts of team development projects are like a relay event, in corporate environments the work is done in parallel simply because everyone is working the same hours but in free software development projects the work is often serialised. I think that it’s often more effective to serialise some work, if someone is actively working on one sections of code it may save time to avoid working in that area until they are finished. There is little benefit in writing code to old interfaces. Some parts of team projects have specialised skill areas (EG debugging, skills in particular programming languages, and graphical design). Soccer is one sport where different rules apply to different players (the goal keeper can use their hands). In ice-hockey the protective clothing used by the goal keeper is considerably different from that used by other players. In most team sports where the aim is to put a ball through a goal at one end (EG basketball and all versions of football) there seems to be some degree of specialisation, some players are dedicated to scoring goals while others are dedicated to defense. The fielding team in cricket has every player assigned to a different part of the field - with slight differences in the skills required. Then there is the issue of large projects such as Linux distributions. It seems to me that a Linux distribution will always comprise multiple teams as well as some individual projects. Maybe we could consider Linux distributions (and distributions of the other free OSs) to be similar to countries that compete in the Olympics. The culture of the Free Software (or Open Source if that’s your idea) community can be compared to the Olympic Spirit. Of course the Olympic idea that people should come together in peace for the Olympic Games and that it’s about honor not money is pretty much dead. Maybe the Free Software development processes should be compared to an ideal of what sporting contests would be if there weren’t unreasonable amounts of money (and therefore corruption) involved. Of course no analogy is perfect and there are many ways in which this one breaks down. One of which is the cooperation between distributions. There is a lot of private discussion between developers of various distributions and upstream developers about how to plan new features. It’s not uncommon for developers to announce certain development decisions as soon as they are made to help other distributions make decisions - for a developer in a distribution project if there is an issue which doesn’t matter much to you or your users then it’s often good to strive for compatibility with other distributions. When users advocate new features or changes they sometimes try multiple distributions. It’s not uncommon for a feature request to be rejected by one distribution and then accepted by another. Once a feature is included in a major distribution the upstream developer is more likely to accept it due to it’s wide testing. Then when the feature is in upstream it’s almost certain to be included in all other distributions. I often recommend that when someone disagrees with one of their bugs being closed as “not a bug” that they try reproducing it in another distribution and reporting it there. As a side note, the criteria for reporting a bug in any free software distribution is that you can describe it in a way that allows other people to reproduce it - whether it’s a bug that afflicts you every day or whether you installed the distribution for the sole purpose of reporting the bug in a new forum is not relevant. As a general rule I recommend that you not have the same bug report open in more than one distribution at any time (if you notice a bug reported in multiple distributions then please add a note to each bug report so that work can be coordinated). As a general rule the only situation where I will open the same bug in multiple forums is if I have been told that the responsible person or people in one forum are unwilling or unable to fix it. Finally, the people who consider that they don’t need to be a team player because they do their coding alone might want to consider Qmail. Dan Bernstein is a great coder and Qmail is by most metrics a fine piece of software, in terms of security Qmail is as good as it gets! If Dan was more of a team player then I believe that his mail server would have been much more successful (in terms of the number of sites using it). However I do understand his desire to have a great deal of control over his software.

Share This

Next.

Previous.