Search Results: "will"

20 December 2009

Wouter Verhelst: Good, not evil

There is a bit of a fluff online currently about the following clause in the "jsmin" code (whatever that is):
The software shall be used for good, not evil
This seems to have started when Google rejected a project based on that code due to its license being not free or open source according to their standards, and therefore not welcome anymore. The arguments then quickly degenerated into things like 'when did google stop being against evil'. But those are all besides the point. One of the most important properties of free and open source software is that anyone can use it for any purpose; there are no restrictions to using them. The DFSG (and hence, the OSD which was derived from the DFSG) encode this as follows:
No Discrimination Against Fields of Endeavor
The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.
You may think "how is 'evil' a field of endeavor", but that is the wrong question. To "use software for evil" can mean any of a number of things, including "nuclear research", "weapons development", "abortion", or, heck "the cash register in a butchery shop", depending on the ethical and moral framework through which the person writing the license sees the world. The ability to give someone a CD or DVD with a bunch of software on there, being able to tell them that they can just use this in any which way they see fit, is a very strong and important feature of the free and open source community. Every time someone comes up with a clause like the above, however, this ability is diluted somewhat; and if it is readily accepted within the greater free and open source movement, then eventually everyone interested in using a piece of software must first check whether they're not trying to use software that forbids someone's pet evil, and we lose one of the greatest strenghts that does exist for free software, but not for proprietary software. The sad thing is, the jsmin author seems to agree. From a video/transcript on which he talks about his absurd license clause is the following quote:
Also about once a year, I get a letter from a lawyer, every year a different lawyer, at a company I don't want to embarrass the company by saying their name, so I'll just say their initials IBM
[laughter]
saying that they want to use something I wrote. Because I put this on everything I write, now. They want to use something that I wrote in something that they wrote, and they were pretty sure they weren't going to use it for evil, but they couldn't say for sure about their customers. So could I give them a special license for that? Of course. So I wrote back this happened literally two weeks ago "I give permission for IBM, its customers, partners, and minions, to use JSLint for evil."
Or, in other words, all you have to do if you want to use this software for evil is set up a second company, tell Douglas that this second company wants to sell software that uses his software to people who might use it for "evil", even though the first company won't, and you're in business. Because Douglas doesn't really oppose evildoers. So the question is, why is that clause in there in the first place? There are only two possibilities; either Douglas didn't really think about those issues, in which case I hope he will one day see the light and remove the clause; or he did, and decided to go ahead and put that clause there anyway. And that would be evil.

Wouter Verhelst: QNAP NAS device

A customer called me a while back. They have a pretty high-end NAS (2U server with external disk enclosure) that they use for most of their needs: both document and other storage for management types, as well as storage for their EDA cluster. People familiar with EDA tools will know that they are pretty disk intensive. They tend to require shitloads of diskspace, preferably at high performance. The problem was that extending the storage space of this high-end NAS device is rather expensive. Supported disks only come at 300G units with prices that are higher than the cheap TB disks available today for the desktop market; and while the price and performance of those disks is worth it for the EDA requirements, the number of people storing documents and similar on the same device is not so high that performance would be an issue. Thus, what they wanted was a cheap solution to augment, rather than replace, their current storage solution, with the focus being on low cost at the possible expense of performance and, since bacula runs well there, reliability. They were quite surprised when I offered them a solution based around a QNAP device; they had expected a price, without hard disks, of about three to four times the price of the QNAP, and therefore were very happy with this suggestion. Unfortunately, the devices produced by this manufacturer that are supported by Lenny, the current Debian stable release, were EOL'd just before we placed the order; however, their replacements will be supported by Squeeze, the next release, and with a bit of help from Martin Michlmayr, we were able to install Lenny with a somewhat more recent kernel. This drove the price up somewhat, both because the newer devices are a few tens of euros more expensive, and because we agreed to pay Martin to prioritize work on the specific device that we'd need, so that we wouldn't have to wait for him to get around to it. As of this Thursday, the device, which had finally arrived after some initial delays, has been installed. Unfortunately, one of the disks that they'd bought was DOA; but they had the foresight to buy five, rather than four, disks, so that was not a major problem. In fact, the only real issues that I ran into were this one arm-specific bug, and the fact that the high-end NAS device is still running etch, which means that rsync --acl won't work. That means I'll have to go back there soon to do an upgrade of the high-end device (which had been planned for a while already); for the arm-specific bug, a workaround is already in place. All in all, a pretty good experience.

Robert Collins: Various releases


Recently I ve been working on the Python unittest API in my spare time, with a long term goal of making it possible to safely and sensibly glue many different plugins together into the core. Two important components of that goal are being able to extend the data included in a test result, and being able to change how a test is run (such as adding new exceptions that should be treated as specific outcomes python unittest uses exceptions to signal outcomes). In testtools 0.9.2 we have an answer to both those issues. I m really happy with the data included in outcomes API, TestCase.addDetail . The API for extending outcomes works, but only addresses part of that issue for now. Subunit 0.0.4, which is available for older Ubuntu releases in the Subunit releases PPA now, and mostly built on Debian (so it will propogate through to Lucid in due course) has support for the addDetail API. Subunit now depends on testtools, reducing the non-protocol related code and generally making things simpler. Using those two together, bzr s parallelised test suite has been improved as well, allowing it to include the log file for tests run in separate processes (previously it was silently discarded). The branch to do this will be merged soon, its just waiting on some sysadmin love to get these new versions into its merge-test environment. This change also provides complete capturing of the log when users want to supply a subunit log containing failed tests. The python code to do this is pretty simple:
def setUp(self):
    super(TestCase, self).setUp()
    self.addDetail("log", content.Content(content.ContentType("text", "plain",
         "charset": "utf8" ), lambda:[self._get_log(keep_log_file=True)]))
I ve made a couple of point releases to python-junitxml recently, fixing some minor bugs. I need to figure out how to add the extra data that addDetails permits to the xml output. I suspect its a strict superset and so I ll have to filter stuff down. If anyone knows about similar extensions done to junit s XML format before, please leave a comment :)

Christian Perrier: Font packages moved to source v3

Today, Hideki Yamane completed his work on the last four packages maintained by the Font packaging Team to make them switch to source v3. Along with these changes, we also moved all our packages to debhelper v7 and minimal rules files. We also dropped the dependency on the obsoleted defoma. The team maintains 77 source packages for fonts and has now built a quite solid experience in this. In the upcoming weeks, I'll try to get in touch with other people who maintain fonts, inviting them to join the team and maintain their packages in our SVN (we don't do very fancy things so SVN is very well suited for such work). The maingoals will be to continue dropping support for defoma (dropping defoma hints files: you can see good examples on how to do this properly in packages maintained by the team, such as ttf-dejavu). We also intend to drop useless Recommends or Suggests that appear in some font packages (often on "x-ttcidfontconf fontconfig" which is now completely useless) as well as harmonizing package naming schemes (with a "(ttf otf)-<foundry>-<fontname>" scheme). Feel free to join and share your advice and thoughts about The Right Way To Maintain Font Packages on our mailing list.

Julien Danjou: Teething troubles

It's not that often that I start something from scratch. It's an amazing feeling to start a new project, to start writing something new. I like that. It's creation, it's an artistic part of our computing stuff. I feel like a code artist. And what I like even more is that little feeling that you are going in an unknown land. Some area in this tech world where nobody ever came before you, or only a few pioneers. That the sensation I got starting to using Cython, Python 3 and various other tools. I just spent half of my time trying to fix problems, rather than working on *my* code. Problems in autoconf macro not knowing Python 2.6 or Python 3.1. Problems and limitations in Cython. And problem in Python. That last one was a hard one. I'm still a beginner in the Python world: I barely know anything. And I was trying to use something nobody never did: building an embedded Python with a set of built-in modules. I spent hours trying to find why one type of module importing was badly failing. I finally found the answer thanks to a guy. who has the same problem A guy ? No. A pioneer. What do I say? A hero. He's been my week-hero! Thank you Miguel Lobo because you found the bug I chased for hours and because you even reported it as issue 1644818, including a patch! How not damn wonderful is that? I will not bore you with the technical details of that bug, since nobody cares. Nobody cares, even the Python guys, since that bug has been opened for 3 years, and nobody even reviewed in that time. I found an old thread about that bug where some guys were wanking about how they should do the review, because Miguel pushed for several weeks to have a review, back in 2007. But that bug was in my way. I had to do something. So I prepared my mail reader, mounted my web browser and here I was for a uniq quest: getting a Python bug fixed. At that point, if you did not stop reading earlier, you might get very excited. Don't be, spoiler, it's still not fixed. You'll have to wait the end of the season and see all the episodes I'll have to write to get the end of the story! Let's continue. I had to create an account on the Python bug tracking system. That was a trivial task for a man like me (you bet). Then, I launched a verbal attack, something you rarely see in a bug tracking system. Something I knew would awake any developer caring about their software.
Julien Danjou: Is there any chance to see this *bug* fixed someday?
I had the deep feeling that my quest was starting here. How many days would I have to wait until I get an answer? Time was passing. Minutes were ticking while I was waiting, sat in a comfortable sofa in a softly lighted room. It seemed like all my life was shorter than the delay I had to wait to get an answer. After waiting for hours, suddenly, and only 15 minutes later, I got an answer:
Martin v. L wis: Please ask on python-dev. I may be willing to revive my five-for-one offer.
Martin? Don't know that guy. Who is he? Who is he like? Will he fix that bug? What is this offer? So many question without an answer. But he asked to ask on python-dev, and I said: challenged accepted! I will write a mail to python-dev to get that bug fixed. Which I did. I sent a short (but well written you know, I made efforts) "WTF?" to pyhon-dev. And then the guy asked me to review 5 bugs so he will review and fix this one. And this is how I said that he was pissing me off for blackmailing me to fix a bug that was its "duty". Therefore, this is the end of the story so far. Will that bug be fixed some day? There's a hope, because another guy jumped in and took the bug assignment. To be continued. My conclusion about all that story: that is a little rude to start something new, with new tools, and get quickly into teething troubles. It's even more harsh to enter a community because you just found bugs, and be not very well received when you ask to apply a 10 lines long fix somebody wrote 3 years ago to fix it. I'll probably still use Python :-), but I get a darker image of its community now.

Debian News: Updates on LXDE

Several LXDE components have been updated recently in sid. They bring some new features and improve flexibility of customization. Below you have a summary of the most important changes:

Andrew Lee

Jose Luis Rivas Contreras: Collecta, searching on real-time

Real-time search on Collecta of xscreensaver

I haven t read of it, and it seems it has been around since a good time. Collecta is a search engine for things in real-time. It searchs on Twitter, Identi.ca, Youtube, Flickr, Blogs, Ustream and shows you what people is saying. It s like a meta-searcher, I guess using API s from those services. But it has it s own API so it will be great for someone that s into web2.x and wants information on real-time from everywhere with just one API. There s a lot of people doing this already and there s a gallery about their development. Maybe tomorrow I will get a little into it, it looks really nice. BTW, it s what powers search.identi.ca

19 December 2009

Stephan Peijnik: Rest in peace Flo: 13.11.1986 16.12.2009

Today is a sad day. Everything feels like I am having a bad nightmare. That's because today I learnt from the too early death of my friend Florian Hufksy.

I am sitting here and do not really know what to write. I keep on thinking about the great times we spent together. The time we started programming when we were twelve. The time we spent learning BASIC. All the times you knew more than me and could teach me a thing or two. I remember our geek talks. How we would discuss latest games. How we lost contact and how we met again. I am thinking about how sorry I am for not having met you often enough. I keep on trying to understand what drove you that far. How you could just end it all. More and more memories come to my mind, like the moment when you showed me one of your projects, Super Mario War. The moments we had playing video games together. All those moments, all that time, I miss you my friend. You were a genius, always a step ahead, not only of me, but seemingly the whole world. I can't stop thinking about your brilliant ideas and how you always finished your projects. You were a real hacker, a real genius, a person trying to make the world a better place, a person who will be missed, not only by me.

You were a genius and I always respected you, not only as a hacker, but as a beloved friend. Why did we not spend more time together? Why did you have to go? Why do I have to write this now, sitting here in my chair with tears in my eyes? And all those memories come up again and again. There is so much more that comes to my mind, but I can't keep on writing, it just hurts too much.

The world is a sad place today. I am sad. I am mourning the too early death of my beloved friend, Florian. You will always have a special place in my heart.

Frans Pop: debmirror IV

The Debian FTP-masters recently changed the way gzipped meta files are compressed in order to make them more efficient to update using the rsync option. This was done by adding the --rsyncable option when calling gzip. Consequence was however that when debmirror compressed Packages, Sources and Contents files after updating them by applying diffs, the md5sum of the gzipped file created by debmirror no longer matched the md5sum listed in the Release file (because debmirror did not use --rsyncable). Result was that debmirror would also download the full gzipped Packages, Sources and Contents files from the parent mirror, something the diffs are meant to avoid. Not nice. Anyway, this has been fixed in debmirror 2.4 which now by default also uses --rsyncable when gzipping the updated meta files. I've also uploaded a fixed version for Lenny (20070123lenny1), which should soon be available from proposed-updates and will be included in the next stable point release. For archives that also provide diffs (most archives don't have them) but do not have rsyncable gzipped files, the default options used when calling gzip can be overruled using the new option --gzip-options (only in version 2.4). Tip: if you are using the rsync method to download files, using --diff=none may well be more efficient now that the archive has rsyncable gzipped meta files. Version 2.4 also has a few other improvements and fixes. If you're currently using version 2.3.x an update to the new version is probably a good idea.

Mike Hommey: Iceweasel bug triaging

I ve spent a few hours going through all the unclassified important bugs assigned to iceweasel. This resulted in In the closed bugs, there were several kind of bugs: It will feel good when it will be visible on the bug graph. Still 500+ to go *sigh* Who wants to jump on the bandwagon ? ;)

Dirk Eddelbuettel: Rcpp 0.7.0

A new release of Rcpp awaits inclusion on CRAN and has also been sent as a .deb to Debian. It will hit mirrors over the next few days, in the meantime you can get it here as well. This release has a couple new features : Fuller details are in the ChangeLog on Rcpp page.

Michael Banck: 19 Dec 2009

Tutorials at the Garching Debian Stammtisch Munich traditionally used to have a lot of Debian Developers, but over the last couple of years quite a few of us who used to be students graduated and moved elsewhere or became very busy with their day jobs. We still meet for having a beer and a chat, but not as much as some years ago. We used to meet about once a month, but in 2009 we only managed to meet four times (however, we organized a Bug Squashing Party in November and had a special meeting as Lenny Release Party in February) As the meetings are really rather informal and not necessarily very Debian related, it is difficult to attract new people this way. So Johannes Wiedersich and I decided to try a more hands-on approach by having a second meeting in Garching, in the student-run bar on the campus of the Technische Universit t M nchen (TUM). The idea was to get more of the local science, mathematics and computer science students (as well as possibly interested faculty members) involved. We tried a first time about a year ago, but after two or three meeting in late 2008 and early 2009, we lost momentum. However, we began organizing the meetings again with the start of the winter term, and had two rather successful meetings so far. For the first meeting, we basically handed out some information on how to get involved locally (the Debian-Munich list, its subscription address, the wiki etc.) and discussed Debian in general and Debconf11 in Munich in particular. About half a dozen people showed up, and two of these attended the Bug Squashing Party later that month, and another one (a faculty member) got very active in Debconf11 organization. Thus, I was quite happy with the outcame of that meeting. Some days ago, we had another meeting, and this time I was doing a live-tutorial on Debian package building. As Johannes was ill, we did not manage to announce or publicize the meeting well in advance, so only three people showed up. Still, I think it went rather OK, and we will be doing another Debian package-building tutorial for the next meeting, and possibly other turorials/workshops afterwards (ideas so far include library maintenance, Debconf, how the Debian community is organized and how to get involved in it). As doing a live-tutorial on one notebook is a bit difficult if you both have to type on it and people should see what happens, we will either use some extra hardware next time, or move to some nearby seminar room with a projector, this will be announced in advance. So if you are on Garching campus or nearby and interested in Debian development (and Debian package-building in particular), come to the next meeting on January 13th! We decided to meet on the second Wednesday of each month, at 18h. Subscribe to the Debian-Munich list to get the invitation or watch out for the flyers on the campus.

Russell Coker: Shared Objects and Big Applications

Some time ago I wrote a little utility named memlockd [1]. Memlockd will lock files into memory which allows significantly faster access when the system pages heavily, in my simulated tests I have found that having the programs and shared objects needed for logging in locked in memory can make it possible to login without a timeout when there is heavy paging, this can make the difference between recovering a system with some processes that are out of control and having to reboot it (often without discovering the root cause). As always happens some people use my software in ways that I never planned. One guy is using it to try and make OpenOffice.org load faster. I m not sure that this is a good idea. In a typical installation when configured for the purpose that I intended it (system recovery from a rabbit process) memlockd will take a bit less than 10M of RAM on an i386 platform (that is for bash, login, sshd, getty, busybox, and all necessary shared objects and a few data files. Since RHEL 4 Red Hat distributions have whinged at boot time if there was less than 256M of RAM available, installation of a Red Hat based system on anything less than 128M of RAM has been impossible for some years, and Debian systems perform very poorly with less than about 128M of RAM when you run apt-get. I initially designed memlockd to run on my SE Linux Play Machine which has 128M of RAM in it s current incarnation. Locking 7.5% of RAM on the system may impact performance, but as a large part of that RAM is used for things like libc and bash which tend to be partially paged in at all times this shouldn t be a noticeable impact. But locking 100M or more of OpenOffice seems more likely to have the potential to hurt performance, I often run OpenOffice on a machine with 512M of RAM and the biggest desktop machine I use has 1.5G of RAM for me it wouldn t make sense to lock OpenOffice into memory. But it could be that there is some unusual aspect of his system that makes running memlockd with OpenOffice likely to give worthy benefits in performance without significantly hurting performance for other programs, for example it could have 4G of RAM and a really slow disk. It is also a possibility that the usage of the system makes OpenOffice so much more important than other programs that any decrease in performance in other areas is not relevant. In any case I m happy to help people use my software to do unusual things so I ll support this use. I ve been asked why memlockd doesn t seem to give much benefit when starting up OpenOffice when run with +/opt/openoffice.org3/program/soffice.bin in the config file, where the + means to lock all shared objects that ldd reports that the binary needs. for n in ldd /usr/lib/openoffice/program/soffice.bin sed -e s/^.*=..// -e s/\ .*// sort -u ; do readlink -f $n ; done sort -u > ldd.txt I used the above command to get a list of all shared objects that ldd reports for the soffice.bin program. On my system (Debian/Lenny i386) it reports 77 shared objects loaded. When memlockd locks all those ps reports that the RSS is 51944K. cat /proc/1234/maps sed -e s/^.*\ // sort -u > /tmp/map.txt Then I used the above command to get a list of the files that are memory mapped by OpenOffice when running OpenOffice calc where 1234 is the PID of the soffice.bin process (I expect that the numbers will be similar for writer, impress, etc I just happened to have a spreadsheet open). It reports 172 memory mapped files which include 9 files related to fonts and 64 shared objects under /usr/lib/openoffice/program which are not found by ldd among other things. It s quite common for a large application to use dlopen(3) at run-time to map shared objects instead of linking against them. Running memlockd with this list gave an RSS of 118644K, which is more likely to give a useful performance boost to OpenOffice load times.

18 December 2009

Asheesh Laroia: Diversity in Free Software: South Asians as an example

As someone born in India, I sometimes look around and wonder, Where are the Indians (and other South Asians) in Free Software? (I don't mean to exclude South Asians from other countries, so I will lump us together. I believe that we are more similar than we are different, although I know more about India than about the rest of South Asia.) There is no shortage of Indians performing information technology jobs in the United States. The same is true in academia; the Computing Research Association uses National Science Foundation data to show about 15% of computer science bacholor's degrees are awarded to "Asians or Pacific Islanders." These are not precise numbers targeted at South Asians in particular, but they confirm a general feeling that plenty of technologists in the United States are from that part of the world. South Asia is quite a populous region, coming in at over one billion people. It, too, has plenty of technology workers. So much FLOSS conversation happens in English, and India is well-suited to handle this; English is an "official language". Indian academia reports that there are 350 million English users and about 90 million English speakers. So let's visually compare the Debian developers map for South Asia (over one billion people) and that of New Zealand, a country of four million. India: New Zealand: These two countries have about the same number of Debian developers (at least, who have marked their location in the Debian LDAP database). About four. South Asians comprise about one sixth of the world's population. There are about one thousand Debian developers; we represent at best 1% of that. These numbers are comparable to the under-representation of women in Free Software, especially when you compare the figure to South Asians' over-representation in the rest of information technology. That makes me sad. Take a look at the Debian developer map again. You'll see that Debian is certainly not an Americans-only project, or even an English-speakers-only project. South America has a respectable dotting of developers, and Western- to Central-Europe are packed. I have strong feelings about Free Software. It emerges from an ethos of personal empowerment, and with open source it has become a dominant force in computing. Yet there are plenty of sharp people -- at least women and South Asians -- who, somehow, become culturally excluded from participating.

Why care about diversity? Consider the diversity of contributors we already have. Some contribute to Free Software because of particular business needs, such as what caused Avi Kivity to write KVM, the new leader in Linux-based virtualization. Everaldo's art background gave us the "Crystal" icon set that set the standard for sharp-looking icons on the Free Desktop for years. Josh Coalson knew about compressing sound, and his Free Lossless Audio Codec is now the standard in high quality audio. We already have a great deal of diversity. We should be celebrating! Back in 2001, FLAC's users were celebrating. In that year, I decided to ditch proprietary operating systems because I felt I could achieve all my computing needs in the Free world. A happy user of FLAC myself, I lurked on the mailing list as I watched grateful people thank Josh for the great software he wrote. Different contributions will excite different sorts of users. The more different people we have improving FLOSS, the more happy users we can make. Happy users of FLOSS are Free users. Happy users can become contributors, putting forth code, documentation, translations, and word-of-mouth marketing. The first reason to improve diversity in FLOSS is to better suit our users' needs. The more diversity we have in our contributors, the more chance we have of tickling our users in the ways that please them the most. I wish to see an end to software that restricts users' freedom, so I want to see us build the tools that users want. One thing that pleases me is when I see other people contributing who seem similar to me. When I went to Debconf, I was thrilled to be surrounded by people who cared about software freedom and technical excellence. I had even more fun being social, chatting about rainforests, mutual friends, websites, and music. I might have had the most fun playing the card game Mao. A second reason, then, to improve diversity in FLOSS is to increase contributor retention by increasing joy. Mao was an example of a cultural bond I happened to share with a handful of Debianites. The more diversity we have, the more frequent these sorts of coincidences will be. The final, most obvious, reason to reach out to groups of people who do not typically contribute is that we can increase our numbers. That by itself is so valuable. Ubuntu sees 100 new bugs per week, even after the bug squad's efforts. If we can do a better job of recruiting new contributors, the raw numbers give us more strength in creating and maintaining world-class software as well as letting the world know about it.

Changing the balance I believe that there are plenty of South Asians quite capable of contributing to FLOSS. I believe the same of women. I believe the same of men. Back to the topic at hand. Why do the South Asians vanish when we look at Free Software, not tech in general? There are plenty of reasons I can dream up, based on my experience with Indians.
  • Plenty of South Asian parents urge their children to make low-risk career choices.
  • My mother reports that schools in India focus on memorization instead of creativity. This can leave little room for extracurricular pursuits.
It's tough for FLOSS advocates to work directly on these distant issues. But I think we can focus some problems we can help solve. Crucially, awareness of Free Software spreads best by social circles. I learned about Linux from a friend at a summer camp. I'll repeat that:
  • Awareness of Free Software spreads best by social networks.
So if you want to spread that awareness, try to be a bridge. If you meet someone from an unusual background for open source who needs support or mentorship, try to help. That is an investment in the diversity and growth of Free Software. Those people can now unlock more "open source minorities."

What success looks like Google Summer of Code helps some new contributors get started and provides that mentorship. Rachel McCreary was invited to the SciPy conference after a successful summer. Her father left a comment explaining how her sisters participated in FLOSS via Google's Highly Open Participation (GHOP) Contest:
Rachel was inspired and motivated by BOTH of her little sisters, each completing six GHOP tasks (if memory serves).
GHOP and GSOC has been a game-changer for these girls. Rachel's younger sister is applying to schools such as MIT with an interest in a science major. The youngest daughter now has a Caltech poster on her wall with the intent to eventually attend.
Their proud Dad
Soon, these stories will be commonplace. Until then, we have work to do. (I'm still researching these topics. If you can help me find any sort of data to help me learn more about diversity in FLOSS, even if it seems like I wouldn't like it, leave a comment.)

Joey Hess: upstreams and packaging

As distributors we should not discourage upstreams that wish to generate binary packages themselves, rather we should cooperate with them, and ideally they will end up maintaining their stable release packages in our distributions.
-- Robert Collins I agree. Robert goes on to talk about the tendancy Debian (and apparently also Ubuntu) have to dislike upstream providing a debian directory. Funny thing is that we also like to say team maintenance is a good idea. An upstream who is doing packaging work is a readymade half of a team; if you write them telling them to rm -rf debian, you are turning away a team member. With a slap in the face. Worse, it's a team member who has demonstrated that they are capable of working in far more complex systems than a debian directory, since they wrote/maintain the software itself. BTW, for those who are skeptical of teams, on the basis that they dilute responsibility: A team consisting of an upstream developer and a distribution maintainer is inherently the more healthy sort of team, where each member has a well-defined area of expertise, but can also venture outside their area when needed. When I packaged FBReader for Debian, upstream already had their own packaging, and I worked with them to fix problems with it and make it something I could be happy maintaining. This was sometimes tricky, since upstream was also maintaining packages for maemo. Sometimes the tools were not ideal. It was still worth it. On the other side of the coin, d-i is an upstream for several distributions. If they told us we needed to rm -rf debian, we'd not have a lot of d-i left. tools I also agree with Robert that most of the trouble comes down to problems with tools. BTW, RPM does not have these problems, and in that world it's typical for upstream to provide a spec file. Much of the problem comes down to the crummyness of dpkg's source format, which cannot rename files, delete files, etc. That the source format directs us to 1970's source management (ie, tarballs and patches), instead of 21st century source management (ie, DVCS), doesn't help either. dpkg's 3.0 quilt source format tries to address the issue by removing any debian directory in upstream's tarball. I am not sure that this is at all the right approach; it makes it harder, not easier to work with upstream as a team. A better approach might be to consider anything that hardcodes the debian directory as having a bug. If upstream can easily package debs using a deb, or maemo, or ubuntu directory, you sidestep any potential conflict while still being able to work with them via a symlink. To put my time where my mouth is: Anytime debhelper prevents working with an upstream who wants to ship a debian directory; that is a bug. I will fix them. (Note that debhelper already provides a --ignore that can be used if upstream has provided a dehelper control file that you can't delete and don't want used.) Previously: Look who's packaging

Robert Collins: Why upstreams should do distribution packaging


Software comes in many shapes and styles. One of the problems the author of software faces is distributing it to their users. As distributors we should not discourage upstreams that wish to generate binary packages themselves, rather we should cooperate with them, and ideally they will end up maintaining their stable release packages in our distributions. Currently the Debian and Ubuntu communities have a tendancy to actively discourage this by objecting when an upstream software author includes a debian/ directory in their shipped code. I don t know if Redhat or Suse have similar concerns, but for the dpkg toolchain, the presence of an upstream debian directory can cause toolchain issues. In this blog post, I hope to make a case that we should consider the toolchain issues bugs rather than just-the-way-it-is, or even features. To start at the beginning, consider the difficulty of installing software: the harder it is to install a piece of software, the more important having it has to be for a user to jump through hoops to install it. Thus projects which care about users will make it easy to install and there is a spectrum of ease. At one end,
checkout from version control, install various build dependencies like autoconf, gcc and so on
through to
download and run this installer
Now, where some software authors get lucky, is when someone else makes it easy to install their software, they make binary packages, so that users can simply do
apt-get install product
Now some platforms like MacOSX and Microsoft Windows really do need an installer, but in the Unix world we generally have packaging systems that can track interdependencies between libraries, download needed dependencies automatically, perform uninstalls and so on. Binary packaging in a Linux distribution has numerous benefits including better management of security updates (because a binary package can sensibly use shared libraries that are not part of the LSB). So given the above, its no surprise to me to see the following sort of discussion on #ubuntu-motu:
  1. upstream> Hi, I want to package product.
  2. developer> Hi, you should start by reading the packaging guide
  3. (upstream is understandably daunted the packaging guide is a substantial amount of information, but only a small fraction is needed to package any one product.)
or (less usefully)
  1. upstream> Hi, I want to package product.
  2. developer> If you want to contribute, you should start with existing bugs
  3. upstream> But I want to package product.
Another conversation, which I think is very closely related is
  1. developer> Argh, product has a debian dir, why do they do this to me?!
The reasons for this should be pretty obvious at this point:
  • Folk want to make their product easy to install and are not themselves DD s, DM s or MOTU s.
  • So they package it privately such as in a PPA, or their own archive.
  • When they package it, they naturally put the packaging rules in their source tree.
Now, why should we encourage this, rather than ask the upstream to delete their debian directory? Because it lets us, distributors, share the packaging effort with the upstream. Upstreams that are making packages will likely be doing this for betas, or even daily builds. As such they will find issues related to new binaries, libraries and so on well in advance of their actual release. And if we are building on their efforts, rather than discarding them, we can spend less time repeating what they did and more packaging other things. We can also encourage the upstream to become a maintainer in the distro and do their own uploads: many upstreams will come to this on their own, but by working with them as they take their early steps we can make this more likely and an easier path.

Julian Andres Klode: The previous post


Well, it seems that several news sites (golem.de, pro-linux.de, linux-magazin.de, linux-magazine.com, ubuntu-user.de [the last ones all from the same publishing house]), especially German ones have picked up the last blog post with same false impressions. First, they stated that I am planning an APT2 release for Christmas. They took the statement
[...] the internal branch has seen a lot of new code[...]. Most of the code will need to be reworked before it will be published, but I hope to have this completed until Christmas.
as a proof for this. But in the context of this paragraph, publish was not meant in the term of publishing a release, but in the term of publishing the code (of the internal branch) in the public repository. The code is public now and lives in a temp branch and will be reworked there for inclusion in the master branch. Secondly, those news sites called me an Ubuntu developer. While I do contribute to Ubuntu, and am an Ubuntu Member, I am NOT an Ubuntu Developer, as I am NOT a member of the relevant ubuntu-dev team at launchpad. Thirdly, those who stated that speed is a goal (mostly pro-linux, at least from my understanding): It is not, at least not now. It is just a coincidence caused by using a relational database. Furthermore, the test was not really fair, since the other package managers both provide more information than capt; information which has yet to be made accessible in APT2. It was just an initial conclusion that SQLite is quite fast. Please note that APT2 is a free time project in development, and the programming language used is still in development as well; as well as some other reverse dependencies. I should also state that neither Debian nor Ubuntu have any plans to drop apt at the moment; and APT is still actively developed. Posted in APT2

17 December 2009

C.J. Adams-Collier: IRC logs for #ubuntu-us-wa

Hello, google. I would like to introduce you to our chat logs. Chat logs, google. Google, chat logs. We will discuss things here such as Mono, GNOME and Debian. We may even use it to talk about work on the DLR project stuff.
<script>//<script language="javascript" src="http://reddit.com/button.js?t=1"></script>

Eugene V. Lyubimkin: a small script to satisfy build-dependencies from .dsc

Here it is: http://paste.debian.net/54275/. It needs /usr/bin/cupt from testing or unstable to work. Please report bugs or suggestions in comments to this post or anywhere where you can reach me.

Typical usage:

1) './script package.dsc -s' - preview
2) 'sudo ./script package.dsc' - do the work

Type './script' to see usage.

I will move it to more permanent storage soon.
UPD: Moved. http://people.debian.org/~jackyf/satisfy-dsc-deps.

Vincent Sanders: What is it with embedded computing?

Last week I was given a development board for a system on chip (SoC) we were looking at using. All i had to do was ensure I could get a reasonable, supportable, bootloader and Linux kernel running on it.

Now I am a fairly competent engineer, I have worked with the ARM processors and SoC for many years. It should not therefor take me four days of almost continuous frustration to get a bootloader and kernel onto a system, but it has.

Firstly the system arrives and we look it over, the hardware design is not exactly elegant but seems like it will get the job done. That is at least a step up from several previous offerings which have been just plain broken. So I power it up and it boots into Fedora core 8, possibly not my first OS choice for a diskless system, but it does work.

Perhaps its worth mentioning that my experience with doing any Open Source development has caused me to have a severe aversion to software which is not upstream, especially kernels. The effort required to run your own patch series and keep it up to date is nothing short of Herculean.

Let us be honest, how many developers do you know who are talented enough to generate good code and patient enough to separately maintain it over a long period of time? I shall not belabor this point as others have repeatedly made it for me in much more eloquent ways. Lets us just say from my point of view. Upstream - good. Long term forks - bad.

With this in mind, the solutions Simtec develop are always with a view to our customers being able to simply download and use the standard upstream software wherever possible. Here is where the story stops being smooth and becomes a rant.

The system was running 2.6.22, one that was patched all which ways. "maybe they just shipped an old kernel" I thought. I opened the CD they provided and...oh, OK its just zips and tarballs of the binaries and source they used (actually that in itself is a minor miracle with a lot of vendors right now - see the almost continuous stream of copyright infringement litigation from the FSF) but sill stuck back in July 2007.

So i did teh normal thing, entered the URL printed on the outside of the box and went looking for the updates. No dice, just the same old stuff. So I contacted the company and recived the news that no there were no updates, that was what was published for the device, thats what I could have.

My initial reaction was of tired resignation...this would not be my first SoC port (probably around the fifth or sixth!) So I responded with cd linux-2.6;git checkout master;git pull and went to see what I could see. and lo and behold, there in the kernel, was full support for not only the SoC but for the other version of the very board I had.

Fortunately after mentioning this on IRC my giraffe loving friend. pointed me at the correct community git tree to find the tiny patch required to support my board.

So the vendor was not only shipping old code but actively dissuading customers from even looking for new stuff, which it turns out was mainline anyway! To add to this somewhere along the lines the company name on the box and contact information has changed and...well confusion reigned.

Right Plain sailing from here on? not quite...that would have been too easy. So you can build a mainline kernel...but you cannot boot it. you guessed it the bootloader they ship on the board is ancient, specifically a u-boot fork from December 2005. Having finally figured out how this was going to go, I went to the uboot repo, pulled the latest and built it for my target. And then I needed to upgrade the u-boot and joy of joys bricked the system.

Fortunately the system has a integrated FDTI serial/JTAG port so re-writing the flash should be as simple as running openOCD configured correctly. Of course this did not work...in the end I resorted to running their binary built versions from the CD (tarfile inside a zip inside a tarfile...whatever) and wrote their original image back to the board...which worked.

Long story short, next day I discover that i need a magic different uboot target (make u-boot.kwb for those that care) write this with the binary openocd build and...success
From then on its all been a bit anticlimactic and straightforward and I now have a system capable of starting a kernel over the network and a root fs on network, disc or flash.

My main complaint from all of this is what the hell are vendors doing shipping crap like this? Why do they all insist on taking a dreadful code drop from the SoC manufacturer when they first brought the part up and shipping that prototype junk until the part reaches the end of its life?!?

All the usual excuses about support etc. will come out, but seriously, you recive *no* support from vendors unless you pay for it and even then they will simply tell you whats on the CD is "it". Btw the diff for u-boot to support this board is larger than the original sources and the 2.6.22 is attacked with a 65MB diff which cannot be good.

The Open Source option is continuously innovating and improving, producing ever better software...but the embedded world seems to be forever stuck shipping crap. Is this just me? Does anyone have any solutions? Or am I stuck with this until I change profession?

Next.

Previous.