Search Results: "haas"

2 July 2024

Ben Hutchings: FOSS activity in June 2024

Ben Hutchings: FOSS activity in May 2024

13 March 2023

Russell Coker: Firebuild

After reading B lint s blog post about Firebuild (a compile cache) [1] I decided to give it a go. It s non-free, the project web site [2] says that it s free for non-commercial use or commercial trials. My first attempt at building a Debian package failed due to man-recode using a seccomp() sandbox, I filed Debian bug #1032619 [3] about this (thanks for the quick response B lint). The solution for me was to edit /etc/firebuild.conf and add man-recode to the dont_intercept list. The new version that s just been uploaded to Debian fixes it by disabling seccomp() and will presumably allow slightly better performance. Here are the results of building the refpolicy package with Firebuild, a regular build, the first build with Firebuild (30% slower) and a rebuild with Firebuild that reduced the time by almost 42%. 
real    1m32.026s
user    4m20.200s
sys     2m33.324s
real    2m4.111s
user    6m31.769s
sys     3m53.681s
real    0m53.632s
user    1m41.334s
sys     3m36.227s
Next I did a test of building a Linux 6.1.10 kernel with make bzImage -j18 , here are the results from a normal build, first build with firebuild, and second build. The real time is worse with firebuild for this on my machine. I think that the relative speeds of my CPU (reasonably fast 18 core) and storage (two of the slower NVMe devices in a BTRFS RAID-1) is the cause of the first build being relatively so much slower for make bzImage than for building the refpolicy, as the kernel build process involves a lot more data. For the final build I moved ~/.cache/firebuild to a tmpfs (I have 128G of RAM and not much running on my machine at the time of the tests), even then building with firebuild was slightly slower in real time but took significantly less CPU time (user+real being 20mins instead of 36m). I also ran several tests with the kernel source tree on a tmpfs but for unknown reasons those tests each took about 6 minutes. Does firebuild or the Linux kernel build process dislike tmpfs for some reason? 
real    2m43.020s
user    31m30.551s
sys     5m15.279s
real    8m49.675s
user    64m11.258s
sys     19m39.016s
real    3m6.858s
user    7m47.556s
sys     9m22.513s
real    2m51.910s
user    10m53.870s
sys     9m21.307s
One thing I noticed from the kernel build tests is that the total CPU time taken by the firebuild process (as reported by ps) was more than 2/3 of the run time and top usually reported it as taking around 75% of a CPU core. It seems to me that the firebuild process itself is a bottleneck on build speed. Building refpolicy without firebuild has an average of 4.5 cores in use while building the kernel haas 13.5. Unless they make a multi-threaded version of firebuild it seems that it won t give the performance one would hope for from a CPU with 18+ cores. I presume that if I had been running with hyper-threading enabled then firebuild would have been even worse for kernel builds as it would sometimes get on the second thread of a core. It looks like firebuild would perform better on AMD CPUs as they tend to have fewer CPU cores with greater average performance per core so a single CPU core for firebuild will be less limited. I presume that the firebuild developers will make it perform better with large numbers of cores in future, the latest Intel laptop CPUs have 16+ cores and servers with 2*40core CPUs are common. The performance improvement for refpolicy is significant as a portion of build time, but insignificant in terms of real time. A full build of refpolicy doesn t take enough time to get a Coke and reducing it doesn t offer a huge benefit, if Firebuild was available in past years when refpolicy took 20 minutes to build (when DDR2 was the best RAM available) then it would be a different story. There is some potential to optimise the build of refpolicy for the non-firebuild case. Getting it to average more than 4.5 cores in use when there s 18 available should be possible, there are a number of shell for loops in the main Makefile and maybe some of them can be replaced by make constructs to allow running in parallel. If it used 7 cores on average then it would be faster in a regular build than it currently is with firebuild and a hot cache. Any advice from make experts would be appreciated.
Related posts:
  1. SE Linux Status in Debian 2012-03 I have just finished updating the user-space SE Linux code...
  2. CPU vs RAM When configuring servers the trade-offs between RAM and disk are...
  3. Load Average Other Unix systems apparently calculate the load average differently to...

7 March 2021

Louis-Philippe V ronneau: New Year, New OpenPGP Key

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Sun, 07 Mar 2021 13:00:17 -0500
I've recently set up a new OpenPGP key and will be transitioning away from my
old one.
It is a chance for me to start using a OpenPGP hardware token and to transition
to a new personal email address (my main public contact is still my
 @debian.org  address).
Please note that I've partially redacted some email addresses from this
statement to minimise the amount of spam I receive. It shouldn't be hard for
actual humans to follow the instructions below to find the complete addresses.
The old key will continue to be valid for a few months, but will eventually be
revoked.
You might know my old OpenPGP certificate as:
pub   rsa4096/0x7AEAC4EC6AAA0A97 2014-12-22 [expires: 2021-06-02]
      Key fingerprint = 677F 54F1 FA86 81AD 8EC0  BCE6 7AEA C4EC 6AAA 0A97
uid       Louis-Philippe V ronneau <REDACTED@riseup.net>
uid       Louis-Philippe V ronneau (alias) <REDACTED@riseup.net>
uid       Louis-Philippe V ronneau (debian) <REDACTED@debian.org>
My new OpenPGP certificate is:
pub   ed25519/0xE1E5457C8BAD4113 2021-03-06 [expires: 2022-03-06]
      Key fingerprint = F64D 61D3 21F3 CB48 9156  753D E1E5 457C 8BAD 4113
uid       Louis-Philippe V ronneau <REDACTED@veronneau.org>
uid       Louis-Philippe V ronneau <REDACTED@debian.org>
These days, I mostly use my key for Debian and to sign git commit. I don't
really expect you to sign my new key if you had signed my old one.
I've published the new certificate on keys.openpgp.org as well as on my
personal website. You can fetch it like this:
    $ wget -O- https://veronneau.org/media/openpgp.key   gpg --import
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEZ39U8fqGga2OwLzmeurE7GqqCpcFAmBFFM8ACgkQeurE7Gqq
CpcuchAAscAeszdtA+TlCI4YvK5nlk+nJnCnNBSnl7Et+jiNjq8kB/Fud+dWMTXC
Zag8oJkalbbxub0BT0bEAn+BiBunu58E0gd0Xq4syTbqZ5o5IN17S/tfxCD0k1hf
ewrnYZ2l0i5g4YvHGKC+Xv4D+Z84BylnIRaXHqlUdluOVfVYDfLybOAqoktO/KUH
I+vQBwXj0Fr/QAtgiz5Nwh/YHFiU9xMSvr5ozRwAFs6+xfIqFHuVPRRkEN5iVo4D
kkMIz+kFfkoh4aWIP4dgAu39XnEgxwTR9J+4yE8TzCCMzO7xCK0X6vqgPAxYMPvb
RuP4FnGWOnGnlcudCUAUkOaryrwRi+dPQTnNICHTYsvVc7dg+W0EhVUkwEuuEwpI
qtcB/Y5AGhqK0Cc11uXiFjIQwLTgwcUez4F0xrGeqsTtAM5gyRup2w0jbocTuYSh
ZRv/2zwrq/S3xVrUYGqdT+L5odmkBzz9zOwY5WlU2H9CMFOdh71XOv9wWQXan9ou
hLRodeOQ8MinIBP+sX36ol1zg/aP7mCHvRRSBzWt7l3WhVxgZFpNwIfp/RZqU0R4
IEq48mntFhPvHJjFmAKLKK/ckzNMtSn+HWQPJV3HTInKCTu5PTNMU3SAvPHOHEps
V6WWSOPB+1Lm/tlIULDc+0SopWoiWO4NObCSs8zMZHlYPBk5x/KIdQQBFgoAHRYh
BMqnQAcHqBawIC/DzfQlelCyHPqFBQJgRRTPAAoJEPQlelCyHPqFFVEA/1qScaAk
O+eBEE4q0BaJDsqweCS1XCcuQGkQCKi5Zv6kAQChQ96Ve7cKbN/wRkT9pdIhmx01
+CmIsnp3k6N0ZYLLCg==
=onl0
-----END PGP SIGNATURE-----

1 April 2017

Paul Wise: FLOSS Activities March 2017

Changes

Issues

Review

Administration
  • Debian systems: apply a patch to userdir-ldap, ask a local admin to reset a dead powerpc buildd, remove dead SH4 porterboxen from LDAP, fix perms on www.d.o OC static mirror, report false positives in an an automated abuse report, redirect 1 student to FAQs/support/DebianEdu, redirect 1 event organiser to partners/trademark/merchandise/DPL, redirect 1 guest account seeker to NM, redirect 1 @debian.org desirer to NM, redirect 1 email bounce to a changes@db.d.o user, redirect 2 people to the listmasters, redirect 1 person to Debian Pure Blends, redirect 1 user to a service admin and redirect 2 users to support
  • Debian packages site: deploy my ports/cruft changes
  • Debian wiki: poke at HP page history and advise a contributor, whitelist 13 email address, whitelist 1 domain, check out history of a banned IP, direct 1 hoster to DebConf17 sponsors team, direct 1 user to OpenStack packaging, direct 1 user to InstallingDebianOn and h-node.org, direct 2 users to different ways to help Debian and direct 1 emeritus DD on repository wiki page reorganisation
  • Debian QA: fix an issue with the PTS news, remove some debugging cruft I left behind, fix the usertags on a QA bug and deploy some code fixes
  • Debian mentors: security upgrades and service restarts
  • Openmoko: security upgrades and reboots

Communication

Sponsors The valgrind backport, samba and libthrift-perl bug reports were sponsored by my employer. All other work was done on a volunteer basis.

21 January 2011

Rapha&#235;l Hertzog: People behind Debian: Michael Vogt, synaptic and APT developer

Michael and his daughter Marie

Michael has been around for more than 10 years and has always contributed to the APT software family. He s the author of the first real graphical interface to APT synaptic. Since then he created software-center as part of his work for Ubuntu. Being the most experienced APT developer, he s naturally the coordinator of the APT team. Check out what he has to say about APT s possible evolutions. My questions are in bold, the rest is by Michael. Who are you? My name is Michael Vogt, I m married and have two little daughters. We live in Germany (near to Trier) and I work for Canonical as a software developer. I joined Debian as a developer in early 2000 and started to contribute to Ubuntu in 2004. What s your biggest achievement within Debian or Ubuntu? I can not decide on a single one so I will just be a bit verbose. From the very beginning I was interested in improving the package manager experience and the UI on top for our users. I m proud of the work I did with synaptic. It was one of the earliest UIs on top of apt. Because of my work on synaptic I got into apt development as well and fixed bugs there and added new features. I still do most of the uploads here, but nowadays David Kalnischkies is the most active developer. I also wrote a bunch of tools like gdebi, update-notifier, update-manager, unattended-upgrade and software-properties to make the update/install situation for the user easier to deal with. Most of the tools are written in python so I added a lot of improvements to python-apt along the way, including the initial high level apt interface and a bunch of missing low-level apt_pkg features. Julian Andres Klode made a big push in this area recently and thanks to his effort the bindings are fully complete now and have good documentation. My most recent project is software-center. Its aim is to provide a UI strongly targeted for end-users. The goal of this project is to make finding and installing software easy and beautiful. We have a fantastic collection of software to offer and software-center tries to present it well (including screenshots, instant search results and soon ratings&reviews). This builds on great foundations like aptdaemon by Sebastian Heinlein, screenshots.debian.net by Christoph Haas, ddtp.debian.org by Michael Bramer, apt-xapian-index by Enrico Zini and many others (this is what I love about free software, it usually adds , rarely takes away ). What are your plans for Debian Wheezy? For apt I would love to see a more plugable architecture for the acquire system. It would be nice to be able to make apt-get update (and the frontends that use this from libapt) be able to download additional data (like debtags or additional index file that contains more end-user targeted information). I also want to add some scripts so that apt (optionally) creates btrfs snapshots on upgrade and provide some easy way to rollback in case of problems. There is also some interesting work going on around making the apt problem resolver a more plugable part. This way we should be able to do much faster development. software-center will get ratings&reviews in the upstream branch, I really hope we can get that into Wheezy. If you could spend all your time on Debian, what would you work on? In that case I would start with a refactor of apt to make it more robust about ABI breaks. It would be possible to move much faster once this problem is solved (its not even hard, it just need to be done). Then I would add a more complete testsuite. Another important problem to tackle is to make maintainer scripts more declarative. I triaged a lot of upgrade bug reports (mostly in ubuntu though) and a lot of them are caused by maintainer script failures. Worse is that depending on the error its really hard for the user to solve the problem. There is also a lot of code duplication. Having a central place that contains well tested code to do these jobs would be more robust. Triggers help us a lot here already, but I think there is still more room for improvement. What s the biggest problem of Debian? That s a hard question :) I mostly like Debian the way it is. What frustrated me in the past were flamewars that could have been avoided. To me being respectful to each other is important, I don t like flames and insults because I like solving problems and fighting like this rarely helps that. The other attitude I don t like is to blame people and complain instead of trying to help and be positive (the difference between it sucks because it does not support $foo instead of it would be so helpful if we had $foo because it enables me to let me do $bar ). For a long time, I had the feeling you were mostly alone working on APT and were just ensuring that it keeps working. Did you also had this feeling and are things better nowadays ? I felt a bit alone sometimes :) That being said, there were great people like Eugene V. Lyubimkin and Otavio Salvador during my time who did do a lot of good work (especially at release crunch times) and helped me with the maintenance (but got interested in other area than apt later). And now we have the unstoppable David Kalnischkies and Julian Andres Klode. Apt is too big for a single person, so I m very happy that especially David is doing superb work on the day-to-day tasks and fixes (plus big project like multiarch and the important but not very thankful testsuite work). We talk about apt stuff almost daily, doing code reviews and discuss bugs. This makes the development process much more fun and healthy. Julian Andres Klode is doing interesting work around making the resolver more plugable and Christian Perrier is as tireless as always when it comes to the translations merging. I did a quick grep over the bzr log output (including all branch merges) and count around ~4300 total commits (including all revisions of branches merged). Of that there ~950 commits from me plus an additional ~500 merges. It was more than just ensuring that it keeps working but I can see where this feeling comes from as I was never very verbose. Apt also was never my only project, I am involved in other upstream work like synaptic or update-manager or python-apt etc). This naturally reduced the time available to hack on apt and spend time doing the important day-to-day bug triage, response to mailing list messages etc. One the python-apt side Julian Andres Klode did great work to improve the code and the documentation. It s a really nice interface and if you need to do anything related to packages and love python I encourage you to try it. Its as simple as: 
import apt
cache = apt.Cache()
cache["update-manager"].mark_install()
cache.commit()
Of course you can do much more with it (update-manager, software-center and lots of more tools use it). With pydoc apt you can get a good overview. The apt team always welcomes contributors. We have a mailing list and a irc channel and it s a great opportunity to solve real world problems. It does not matter if you want to help triage bugs or write documentation or write code, we welcome all contributors. You re also an Ubuntu developer employed by Canonical. Are you satisfied with the level of cooperation between both projects? What can we do to get Ubuntu to package new applications developed by Canonical directly in Debian? Again a tricky question :) When it comes to cooperation there is always room for improvement. I think (with my Canonical hat on) we do a lot better than we did in the past. And it s great to see the current DPL coming to Ubuntu events and talking about ways to improve the collaboration. One area that I feel that Debian would benefit is to be more positive about NMUs and shared source repositories (collab-maint and LowThresholdNmu are good steps here). The lower the cost is to push a patch/fix (e.g. via direct commit or upload) the more there will be. When it comes to getting packages into Debian I think the best solution is to have a person in Debian as a point of contact to help with that. Usually the amount of work is pretty small as the software will have a debian/* dir already with useful stuff in it. But it helps me a lot to have someone doing the Debian uploads, responding to the bugmail etc (even if the bugmail is just forwarded as upstream bugreports :) IMO it is a great opportunity especially for new packagers as they will not have to do a lot of packaging work to get those apps into Debian. This model works very well for me for e.g. gdebi (where Luca Falavigna is really helpful on the Debian side). Is there someone in Debian that you admire for his contributions? There are many people I admire. Probably too many to mention them all. I always find it hard to single out individual people because the project as a whole can be so proud of their achievements. The first name that comes to my mind is Jason Gunthorpe (the original apt author) who I ve never met. The next is Daniel Burrows who I met and was inspired by. David Kalnischkies is doing great work on apt. From contributing his first (small) patch to being able to virtually fix any problem and adding big features like multiarch support in about a year. Sebastian Heinlein for aptdaemon. Christian Perrier has always be one of my heroes because he cares so much about i18n. Christoph Haas for screenshots.debian.net, Michael Bramer for his work on debian translated package descriptions.
Thank you to Michael for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

4 comments Liked this article? Click here. My blog is Flattr-enabled.

31 December 2010

Russell Coker: Links December 2010

Aaron Huey gave a disturbing TED talk titled America s native prisoners of war [1]. He says the last chapter in any successful genocide is the one in which the oppressor can remove their hands and say my god, what are these people doing to themselves, they re killing each other, they re killing themselves while we watch them die .Peter Haas gave an interesting TED talk about how the poor engineering work in Haiti contributed to the significant death toll from the earthquake [2]. He advocates training for builders to prevent death and property damage from the next earthquake which is a lot cheaper than cleaning up the mess after buildings have fallen down.Wired has an interesting article on Phylo, a new crowd-sourced science game where you can sequence DNA [3]. The article also has links to other crowd-sourced science games.Brendan Scott gives a good summary of some of the most interesting news articles related to Wikileaks [4].Petter Reinholdtsen published a detailed and informative letter that Peruvian Congressman Edgar Villanueva wrote to Microsoft on the topic of a Peruvian bill to compell the government to use free software [5]. This has a lot of great ideas for anyone who wants to lobby their government for free software related legislation.Melissa McEwan wrote an informative blog post about why she doesn t trust men [6]. I can t do justice to this with a summary so just read it if you are male.Diana Laufenberg gave an interesting and inspiring TED talk about ways of teaching children [7]. Her main point was about embracing failure, having children learn from their mistakes. I think that perhaps embracing failure is only going to work with an exceptional teacher such as Diana, and that the majority of teachers would probably fail if they tried to implement it. She does have some really interesting examples of how she teaches so it s worth watching even if you don t agree with the central point.

3 December 2010

Pau Garcia i Quiles: KIPI plugin for screenshots.debian.net

The screenshots.debian.net site is a public repository of screenshots taken from applications contained in the Debian GNU/Linux distribution and its derivates like Ubuntu. It was created by Cristoph Haas to help people get an impression of what a certain software will look like on your desktop before you install it. Everybody can take screenshots and upload them through the upload form. So far there are about 3100 screenshots, which is actually a small number if you consider Debian Squeeze contains about 15000 source packages After I hacked a bit on KSnapshot, the KDE snapshot tool, Sune asked me to add an export to screnshots.debian.net command. I had not thought of it before but that looked like a great idea, it would make the process or taking-screenshot, visit-upload-form, upload-each-screen so much easier!. So I started to develop my first KIPI plugin. Here is the resulting Debian Screenshots plugin, as you would use it in KSnapshot:

Being a KIPI plugin, it is available not only from KSnapshot but also from Digikam, Gwenview and the other applications which support KIPI plugins:

Which makes me think: what about having a debshots (the software powering screenshots.debian.net) installation over at kde.org and have screenshots for all the KDE applications? What do you think? Promo team? Sysadmins? Setting it up shouldn t be difficult: I m no Python or Pylons expert (I m more of a Ruby and Wt C++ guy :-) and got it running in a VM in an afternoon, even solving some SQLAlchemy 0.6 issues.

9 November 2010

Julien Valroff: I am a Debian Developer!

A few months after starting the NM process, I have just been accepted as a Debian Developer. My account name is simply: julien I have been a Debian user for about 10 years now, and have begun contributing to Debian in 2005. I have then been accepted as a Debian Maintainer in 2007. This post is mainly to thank: Also thanks to all people who have already sent their congratulations, it makes me very proud!

20 August 2010

Gerfried Fuchs: Screenshots on packages.debian.org

People often want to know what an application looks like before they install it. This is one of the reason why Christoph Haas started with the screenshots.debian.net service. People can upload screenshots for their favorite applications for others to take a look at. Finally, as convenience feature, they are now added on packages.debian.org. When you click on the screenshot of a package you can see the list of all of the available ones. In the case when no screenshot is available yet the page will show a placeholder image which gives you a convenience link to the screenshots page of the package where you can submit one for the benefit of all. Please make sure that you are following the guidelines for submissions, otherwise your upload might get rejected. On a not totally unrelated topic, the packages.ubuntu.com site finally will also show you packages in maverick. It doesn't show screenshots yet if you think adding support for screenshots in there would be a good thing too, please let me know! Enjoy!

/debian permanent link Comments: 5 Flattr this

6 June 2010

Petter Reinholdtsen: A manual for standards wars...

Via the blog of Rob Weir I came across the very interesting essay named The Art of Standards Wars (PDF 25 pages). I recommend it for everyone following the standards wars of today.

17 November 2009

Christoph Haas: Curious mice

You better not leave confidential documents near your optical mouse. You never know where it s looking at.

26 September 2009

Christoph Haas: This should work fine nowadays (about vino, Compiz and wrong optimism)

I ve been back from KDE (4.2!) to Gnome and with the mighty powers of Compiz my desktop is the envy of my Windows coworkers and still geeky functional. Well, somehow functional. For a while I wondered why I can t properly use VNC to remote-control my desktop from home. The connection works and I see the desktop but nothing seems to happen when I click or type. Now I figured out the desktop in the VNC client window is just not refreshed while my mouse and keyboard actions really do what they were supposed to on the VNC server. This is my sad tale of how I learned that a regression may mean driving the motorcycle to work in the middle of the night after cursing VNC. :) After a little searching through the lazyweb I found out that it s a bug reported on Ubuntu already. Apparently Compiz doesn t use the XDAMAGE extension properly. Somehow when using the graphics adapter s magic 3D powers X11 loses control of what sections are changed I m not an X11 guru. At least the client doesn t know when to redraw areas of the screen that have changed. The suggested workarounds were pretty ridiculous. Start another x11vnc session without compiz and connect to that. Very funny. I want to control my running desktop not create a new one. So I switched off compiz at work and voila VNC works again. Some eye-candy and shortcuts are gone but remote controlling the desktop is far more important while im on call. But I quickly missed Compiz. The missing eye-candy wasn t that bad. But metacity s window placement algorithm sucks as badly as back in 2002. As if I can hear it whisper: I have no idea where to put that stupid window. You tell me. I researched a little further and somehow the bug is known to Gnome, too. And other users suffering from the problem are pointed to the VirtualGL page of Wikipedia. Apparently the problem was fixed for several people until a regression was reported where things failed again. Seems like the vino developers removed the piece of code that detected the use of 3D window managers as a 2008 christmas present. Their reason: This should work fine nowadays. Hell, no, it doesn t! Somehow I downgraded vino to version 2.24.1-3 and ran the vino server manually (/usr/lib/vino/vino-server sm-disable) to see if it prints any error message. Indeed it does: 
Composite detected, disabling XDamage extension.
This is a workaround while XDamage extension does not work correctly with 3d desktops.
Hopefully it will work at next vino release.
This time it worked well, too . I upgraded to the current version 2.28.0-1 (as in unstable by today) and suddenly my VNC connection had the same problems again. Aha! Let s try the Lenny version 2.22.2-1. Works, too. This has taken me hours to debug. Finally I know I can report a bug against the vino package. I wonder how long it would have taken a mortal to figure that out. :( [Sorry to my sponsorees who's package sponsoring requests were delayed again due to this mess.]

19 September 2009

Christoph Haas: Tired of Nagios and Cacti? Try Zabbix.

One of my professional duties in my past ten years was monitoring systems. Even my diploma thesis was dedicated to distributed monitoring (altough my professor sucked badly). Apart from a few custom-programmed scripts to analyze special situations (e.g. proxy clusters) I used tools that fellow administrators will find familiar: Nagios and Cacti. And another less famous text-configuration-based monitoring tool called Cricket. That worked well somehow but Cricket was hard to learn for my coworkers and Cacti seems unreliable and fundamentally broken in terms of SNMP checking. Besides why do I have to set up availability checking in Nagios and set up checking of the same parameters in another software to draw graphs? Just recently I came across an open-source software I hadn t heard of before: Zabbix. And although it has a few rough edges it seems way more professional than other common tools (the commercial tools I saw are even worse than the open-source variants). I tried it and after a lot of reading and trying it looks like it has a good potential to replace Nagios and Cacti. So I thought I d sum up my personal experiences with all of these tools. Nagios. Their makers claim that it s the industry-standard in IT infrastructure monitoring . Honestly it s a great tool but considering how many years it has been existing it barely evolved. During my diploma thesis I wrote an alternative software that I called MrNetwork that dealt with flaws that Nagios hasn t even fixed today. Still Nagios is a tool I have used for many years and it is very reliable.Nagios service detail screen Advantages: Annoyances: Cricket. As Nagios does not support plotting graphs of the monitored values I was in need of another piece of software. Basically Cricket is a frontend to RRD (which stores data in a rotating/round-robin file that keeps data of the last X minutes/hours/days). It has a textual configuration that takes a lot of getting used to. It s main principle is inheritance of settings they call it configuration tree . Which means you have a master DEFAULTS file that contains general settings like how to query SNMP. In a subdirectory you define a certain class of devices that you want to monitor e.g. routers (the DEFAULTS are inherited to this level). Within the routers directory you can just define a list of routers you want to monitor. All settings are inherited from above (parent directories). It s more a geek tool for shell lovers. Cricket screenshotAdvantages: Annoyances: Cacti. Another frontend to RRD and a pretty sophisticated one. Nearly everything is configured through its web interface. And the result is beautiful. It s not entirely reliable though and SNMP support (at least in version 0.8.7b) is a big fail. I like Cacti for its user interface is much better than Cactis but it s less reliable and flexible. Cacti screenshotAdvantages: Disadvantages: Zenoss. People pointed me to Zenoss which is supposed to offer the same functionality as other monitoring systems but is much more integrated. So this short list is more a quick one-day-experimental expression than a thorough analysis. But in the end much of the fuss is just marketing. Advantages: Annoyances: Zabbix. This is the tool I have been evaluating for the past two weeks and currently I m moving all checks away from Nagios and Cacti and put it into Zabbix. So the following list is more a first impression instead of a year-long experience. (I m using Zabbix 1.6.5 here. Debian Lenny s 1.4 version lacked some interesting features like proper SNMP handling so I backported the newer version to Lenny.) Zabbix seems very close to the perfect monitoring system I had always dreamt of. I would have designed it differently in some aspects though. Zabbix screenshotAdvantages: Annoyances: See also: Ben Rockwood s blog Further similar software I didn test thoroughly: Hyperic and Opsview (I ll surely update this blog post while I learn more about Zabbix.) All of the above tools are great. I m not meaning to say that Zenoss is total crap for example. The differences are subtle. And whether a piece of software suits your needs really depends on your expectations. I love that all this software is available as open-source. And a totally unscientific but fun analysis of the community is counting the number of active people in the respective channels on the Freenode IRC network: Either Nagios has the largest fanbase or perhaps that means that the majority of people needs help with it. :)
http://en.wikipedia.org/wiki/Network_monitoring

8 May 2009

Christoph Haas: Thanks for the great time, KDE

After years of being a dedicated KDE user it seems like it s finally time for me to go back to Gnome. I once tried KDE 4.x from the Debian experimental branch knowing that it s probably unstable and will require a lot of patience. But I was willing to report bugs and dive into new things. It didn t take long though to find out that graphics were extraordinarily slow with the new desktop effects . The worst was resizing a konsole (KDE terminal window). It took 25 seconds until the window was resized and redrawn! On the interweb I found KDE developers rant about NVidia who didn t support some special extension they needed. They suggested to get an ATI card instead. (No, thanks, I was struggling with the crappy radeon and fglrx drivers on my new work laptop and am sick of ATI for Xorg.) Or I could use a bare XTerm instead. Yeah, right. So I couldn t use the desktop effects. After all I used the non-free NVidia driver for years and it never let me down. I don t mind doing without eye candy. But the desktop effects were about the only serious way to switch between virtual desktops. Point the mouse into a corner or edge of the screen and get a zoom-out bird s view of the virtual desktops. Nice but quickly annoying. Besides you had to push the mouse pointer a few pixels into the edge (it s not sufficient to just hit the edge) which means that after 10 desktop switches I had to lift the mouse and set it back on the mousepad. Another annoying issue is that up to now I can t assign functionality to every key on my keyboard any more. I had a neat trick by having 3 3=9 virtual desktops and use my numpad keys 1 to 9 to page through them. 0 on the numpad opened a new shell window. + on the numpad maximized a window. But now I couldn t assign any of those numpad keys. I pressed the 1 but KDE didn t detect Numpad_1 but just 1 . Guess how useful it is to not be able to type the 1 any more. There is a low-priority bug report for KDE from users who want this fixed but it will likely not happen any time soon. So the only way to switch desktops was to poke on the tiny pager widget in my panel. Yuk. Not to mention that I couldn t put icons on the desktop any longer. There was just a sissy widget that I could use for that purpose but it didn t work like a conqueror or dolphin at all. It was pretty amusing when in KDE 4.2 there was suddenly a feature to re-enable the old behavior. Let me guess, KDE developers - I wasn t the only user you couldn t persuade of the new paradigm? Next evil thing was performance. KDE 4 takes ages to load. Simple window managers get me started within seconds. Even Gnome. But KDE takes 30 seconds until it s loaded plus some disk grinding in the background. And memory usage wasn t nice either. And what finally made me mad was stabilty - or rather lack thereof. Everytime the screen saver ran my kbluetooth module crashed. kmail crashed even more frequently than before. (I loved kmail for it s great customizability and feature set.) plasma (the actual window manager) died at least twice a day. Suddenly the panels were looking ugly or crashing. As a Debian developer I have to follow the unstable branch on my workstation. And there is no distinction between KDE 3 and KDE 4 any more so a dist-uprade inevitably gets all of my applications upgraded to KDE 4 (plus an hour of manual engineering in aptitude because half of my applications weren t upgraded automatically due to different application names and I had a mixed environment). So there I was with KDE 4.2. I really worked with it for a month. I tried hard to get acquainted to its bloatedness and instability. It s really a nice technological preview of what might be possible in the future. Everything is really automatic. Mounting external media. New ways to find programs (although I m not happy with the new K start menu). Funny effects and nice eye candy. But it s far from the rock solid desktop environment that I was used from KDE 3.5.8. So there I am on Gnome now. I was switching back and forth. One day working on KDE 4.2 and the other day using Gnome 2.26. And although it makes me feel sad I think it s finally time to say goodbye to KDE 4. I have invested a good amount of time and energy into it - even just from a user perspective. But it s not ready. And I finally need to get some work done. A number of projects suffered from my playfulness (my ispmail tutorial, screenshots.debian.net, debexpo, my kids) and it s time to get back to work. And Gnome was pretty surprising. With compiz I have the KDE desktop effects working (here they really do) and they are all customizable. I can assign keys properly again (even the numpad). Everything is pretty fast and stable. I have replaced kmail and its buggy shared calendar feature by Icedove (aka Thunderbird) with the Lightning plugin. Worked from the first minute and hasn t crashed in a week (just have to remember next time to install libstdc++5 to be able to use Lightning). With enigmail I finally have a proper simple working PGP key management that doesn t die and is buggy like hell like in kgpg. And there s the nice Nostalgy add-on that allows me to file emails into IMAP folder quickly again - a feature that just kmail seemed to support so far. Oh, and I see new mail instantly because Icedove supports the IDLE command while kmail has never learned that neat little trick. The Gnome file manager isn t really great for my taste but it s doing its job (dolphin is just much nicer). It s not as nice as KDE but I had to learn that working and stable weighs more than nice. The hard way. And many things I liked about KDE 4 (like starting a new program with Alt-F2 and having a smart application finder and simple calculator in one) are built into Gnome, too. So why bother. Thanks, KDE, for the years of fun and service. I ll surely miss you. Get back to me should you grow up one day.

16 February 2009

Christoph Haas: Debian 5.0 cake

Thanks to all involved people for the Debian 5.0 Lenny release! My wife made me a cake for my coworkers so we can celebrate a little. But even my boss believes that it must have been my birthday because no sane person would bring a cake to celebrate bits and bytes. Well, here it is (black forest cake style): My wife\'s contribution to the Debian 5.0 Lenny release (Sorry for the quality of the photo. My real camera died yesterday.) ((Now I have to verify the complaints of a coworker that LXDE still doesn t install.))

2 February 2009

Obey Arthur Liu: Debian Summer of Code 08 : Where are they now (part 3/3)

Welcome back for the last part of the reviews. You may want to look at the previous parts : part 1 and part 2. Jigdo-ivory, a JavaScript Jigdo client Presentation Debian CDs and DVDs take up a huge mount of space on download servers. Using jigdo to download those images can significantly reduce the amount of bandwidth and space needed on the central servers. Unfortunately, jigdo currently needs special client software to be downloaded/installed first. Adding support directly into a browser-based application could potentially make a very big difference for first-time users here. Jigdo was created in 2001. It allowed to create ISOs from .debs grabbed from regular mirrors. It eliminated the need to duplicate the entire contents of the package repository into ISO files for each release, or even more importantly, for weekly snapshots of testing/unstable/whatever. You may find the complete proposal from the student here. The original idea originated from the Debian-CD people, who wanted to explore ideas about creating a light web client. The project was mentored by Steve McIntyre, who developed a new version of the Jigdo tools, jigit, which is much more efficient. Student Dustin Rayner was a 5th year senior undergraduate student at the Oklahoma Christian University in Oklahoma City, Oklahoma. I studied Computer Engineering for 3 years as a Computer Engineering student before deciding to pursue a Mathematics and Computer Science degree. Result This project was unsuccessful due to numerous issues. First, because of an inadequate technical preparation of the original proposal. The Debian-CD people were too optimistic with the possibilities of Javascript. In the end, the copying and checksumming part of the Jigdo process were implemented but the checksumming (with a Javascript implementation of md5) was so slow that it was unusable (think 50kb/s on a regular laptop at full CPU charge). The student did the right thing to investigate Java and ActiveX but it was too late unfortunately and he ultimately lacked the experience and knowledge in the relevant technologies. If the proposal is tried again, the student would be requested to have much more experience with Java (and possibly ActiveX). Those would be much more efficient for the task, as they are the most used technologies among on-line anti-virus scanners, which have a workload somewhat similar to Jigdo. I could not find further public involvement of Dustin Rayner within Debian. Aptitude-gtk, usability and GTK+ GUI for the Aptitude package manager Presentation A GTK+ GUI for Aptitude that will work alongside improved current ncurses and command-line interfaces. This will offer an alternative to Synaptic with an interface design geared toward usability and advanced functionality. Debian currently supports multiple non-command-line package managers, the most used being Synaptic and Aptitude. Synaptic uses a GTK+ interface but offers no command-line mode. Aptitude offers a command-line mode but no X interface, although it offers a ncurses interface.
Comparing the interfaces of Synaptic and Aptitude reveal many design differences. Although Synaptic may be more accessible to beginners, Aptitude offers many interface behaviors and functions that are useful to the regular to advanced users : fully hyperlinked tabbed navigation between packages and versions of packages, mostly modeless interface, interactive dependency conflict resolver The proposal was introduced by the student in coordination with Daniel Burrows, the mentor and developer of Aptitude. Student Obey Arthur Liu was a 22 year old french student of Computer Science and Applied Mathematics at Grenoble Institute of Technology - ENSIMAG, in France. Did I mention that he s also yours truly ? If you want to know more, you might be interested in my previous post. Result This project was successful. The interface was mostly done and functional by the end of the summer. Daniel merged the code into the main post-lenny branch. Development is still ongoing and packages are released into Experimental. For further information, just read the rest of my blog. I could find some further public involvements of Obey Arthur Liu within Debian. Doh! Lintian for fuller automated setups Presentation lintian, the Debian package checker, at the moment presents possible problems in three categories: errors, warnings and informational messages. This leads to several problems, most importantly that the severity and certainty of a check can t be expressed separately. In the course of this project, the student should design and implement in lintian an improvement of the current situation, for example by using a two-letter code (one for certainty, one for severity). This project would make lintian errors much more fine-grained and help in maintaining pertinent quantitative analysis of package quality. The project was mentored by Marc Brockschmidt. The project proposal was commonly introduced by the Lintian team. Student UPDATED: Jord Polo Bard s has done a lot of work with translation in Catalan, his native tongue. He can usually be found on #debian-catalan. He also maintains a few packages as a DM. Result This project was successful. The classification was entirely done. Jord also helped with the new lintian.debian.org website. The Lintian team was very satisfied with the revamped errors list and new website. They have an immediate impact on packages quality reporting. Jord is still active within Debian, helping package a few games. Debexpo, a generic web-based package repository Presentation mentors.debian.net is currently a very specialized web-based repository that allows everybody to contribute software packages to Debian without the need to be a Debian Developer (or Debian Maintainer). It has successfully helped simplifying the sponsoring process in the last years. However it needs to be refactored and in the process should be turned into a generic piece of software that can be used for other Debian source/binary package repositories, too. Mentors is a very good initiative to recruit new packages maintainers (and needs your help!) and the software underlying it could be reused for many different purposes (think PPA). The project was mentored by Christoph Haas. The project proposal was commonly introduced by the mentors team. Student Jonny Lamb was a Computer Science student in the United Kingdom. He was already quite involved within Debian, maintaining a lot of significant packages. Result This project was successful. The whole proposal was perfectly executed. Jonny now continues to develop debexpo, with the mailing-lists and commit logs showing interesting activity. Of course, help for debexpo is appreciated to get it into full shape. Jonny has since become a Debian Developer (here is his AM report). Congratulations to him. It s nice to end on a nice note isn t it ? Now that we re done with the individual reports, I m going to write down my recommendations report. Hopefully it will help with next year s Summer of Code.

13 January 2009

Christoph Haas: NFS: sec=sys or ruin your day

And once again I was bitten by problems on my Debian laptop mounting directories from the file server via NFS. After a Debian dist-upgrade I couldn t login to KDE any more. Shell login worked somehow but I quickly found out I could neither read nor write any files there. Apparently an ls -al showed me the right permissions (not just unmappable numeric UIDs or something of that kind) and an id -a confirmed that my LDAP PAM configuration still worked. But reading or writing any file just lead to Permission denied . I m not sure if it was an update in the nfs-common package in Sid. But I had the same problem before and it took me hours until I finally figured out that Debian s NFS seems to use non-standard defaults. Namely the sec parameter when mounting the NFS share. According to the Solaris documentation the default is sec=sys which means that NFS uses the locally acquired UIDs and GIDs. Like /etc/passwd, NIS or LDAP/PAM. But on Debian it seems to default to sec=krb5 or something. As I have close to no idea how to set up Kerberos and don t want to (and talking to other people hardly anyone has used Kerberos either) I figured that it s not really a sane default. I didn t even ask for NFSv4 - just NFSv3. Perhaps I undeliberately set some /etc/default/nfs-common configuration setting wrong or whatever. It was just strange. So I set sec=sys in the options of the NFS share of my /etc/fstab and the problem was fixed. Actually I wonder what network file systems other people use in a Debian environment. NFS somehow feels antiquated to me anyway.

23 December 2008

Emilio Pozuelo Monfort: Collaborative maintenance

The Debian Python Modules Team is discussing which DVCS to switch to from SVN. Ondrej Certik asked how to generate a list of commiters to the team s repository, so I looked at it and got this:
emilio@saturno:~/deb/python-modules$ svn log egrep "^r[0-9]+ cut -f2 -d sed s/-guest// sort uniq -c sort -n -r
865 piotr
609 morph
598 kov
532 bzed
388 pox
302 arnau
253 certik
216 shlomme
212 malex
175 hertzog
140 nslater
130 kobold
123 nijel
121 kitterma
106 bernat
99 kibi
87 varun
83 stratus
81 nobse
81 netzwurm
78 azatoth
76 mca
73 dottedmag
70 jluebbe
68 zack
68 cgalisteo
61 speijnik
61 odd_bloke
60 rganesan
55 kumanna
52 werner
50 haas
48 mejo
45 ucko
43 pabs
42 stew
42 luciano
41 mithrandi
40 wardi
36 gudjon
35 jandd
34 smcv
34 brettp
32 jenner
31 davidvilla
31 aurel32
30 rousseau
30 mtaylor
28 thomasbl
26 lool
25 gaspa
25 ffm
24 adn
22 jmalonzo
21 santiago
21 appaji
18 goedson
17 toadstool
17 sto
17 awen
16 mlizaur
16 akumar
15 nacho
14 smr
14 hanska
13 tviehmann
13 norsetto
13 mbaldessari
12 stone
12 sharky
11 rainct
11 fabrizio
10 lash
9 rodrigogc
9 pcc
9 miriam
9 madduck
9 ftlerror
8 pere
8 crschmidt
7 ncommander
7 myon
7 abuss
6 jwilk
6 bdrung
6 atehwa
5 kcoyner
5 catlee
5 andyp
4 vt
4 ross
4 osrevolution
4 lamby
4 baby
3 sez
3 joss
3 geole
2 rustybear
2 edmonds
2 astraw
2 ana
1 twerner
1 tincho
1 pochu
1 danderson
 As it s likely that the Python Applications Packaging Team will switch too to the same DVCS at the same time, here are the numbers for its repo:

emilio@saturno:~/deb/python-apps$ svn log egrep "^r[0-9]+ cut -f2 -d sed s/-guest// sort uniq -c sort -n -r
401 nijel
288 piotr
235 gothicx
159 pochu
76 nslater
69 kumanna
68 rainct
66 gilir
63 certik
52 vdanjean
52 bzed
46 dottedmag
41 stani
39 varun
37 kitterma
36 morph
35 odd_bloke
29 pcc
29 gudjon
28 appaji
25 thomasbl
24 arnau
20 sc
20 andyp
18 jalet
15 gerardo
14 eike
14 ana
13 dfiloni
11 tklauser
10 ryanakca
10 nxvl
10 akumar
8 sez
8 baby
6 catlee
4 osrevolution
4 cody-somerville
2 mithrandi
2 cjsmo
1 nenolod
1 ffm
 Here I m the 4th most committer :D And while I was on it, I thought I could do the same for the GNOME and GStreamer teams:
emilio@saturno:~/deb/pkg-gnome$ svn log egrep "^r[0-9]+ cut -f2 -d sed s/-guest// sort uniq -c sort -n -r
5357 lool
2701 joss
1633 slomo
1164 kov
825 seb128
622 jordi
621 jdassen
574 manphiz
335 sjoerd
298 mlang
296 netsnipe
291 grm
255 ross
236 ari
203 pochu
198 ondrej
190 he
180 kilian
176 alanbach
170 ftlerror
148 nobse
112 marco
87 jak
84 samm
78 rfrancoise
75 oysteigi
73 jsogo
65 svena
65 otavio
55 duck
54 jcurbo
53 zorglub
53 rtp
49 wasabi
49 giskard
42 tagoh
42 kartikm
40 gpastore
34 brad
32 robtaylor
31 xaiki
30 stratus
30 daf
26 johannes
24 sander-m
21 kk
19 bubulle
16 arnau
15 dodji
12 mbanck
11 ruoso
11 fpeters
11 dedu
11 christine
10 cpm
7 ember
7 drew
7 debotux
6 tico
6 emil
6 bradsmith
5 robster
5 carlosliu
4 rotty
4 diegoe
3 biebl
2 thibaut
2 ejad
1 naoliv
1 huats
1 gilir
emilio@saturno:~/deb/pkg-gstreamer$ svn log egrep "^r[0-9]+ cut -f2 -d sed s/-guest// sort uniq -c sort -n -r
891 lool
840 slomo
99 pnormand
69 sjoerd
27 seb128
21 manphiz
8 he
7 aquette
4 elmarco
1 fabian
 Conclusions:
- Why do I have the full python-modules and pkg-gstreamer trees, if I have just one commit to DPMT, and don t even have commit access to the GStreamer team?
- If you don t want to seem like you have done less commits than you have actually done, don t change your alioth name when you become a DD ;) (hint: pox-guest and piotr in python-modules are the same person)
- If the switch to a new VCS was based on a vote where you have one vote per commit, the top 3 commiters in pkg-gnome could win the vote if they chosed the same! For python-apps it s the 4 top commiters, and the 7 ones for python-modules. pkg-gstreamer is a bit special :)

19 November 2008

Christoph Haas: screenshots.debian.net gets slashdotted

Bear with me that I keep you posted with boring news of screenshots.debian.net. But after further optimisation (caching the start page, serving screenshots directly from nginx instead of via Pylons) we decided that it’s time to get us on Slashdot. Thanks a lot to zepolen for teaching me about memcached and threading! The slashdot people have approved my announcement and we are already getting hammered by people following the link to screenshots.debian.net. Currently the server load is still near 100%, we have 10 Mbps network traffic and 60 request/second. Watching the web server logs is currently pretty cool. :) 21:08:30 up 455 days, 15:08, 2 users, load average: 1.27, 0.94, 0.69 Network traffic when slashdotting began Yes, I\'m on slashdot! :) Honestly I hardly ever read Slashdot. But apparently many others do. And if you are especially bored you can read the comments on the posting there. So much childishness. “You should have made it a distribution-independent site.” “How should I make a screenshot of libfoo-bar?” Why can’t people just be happy with what they get for free and make proper suggestions? Well, go create something yourself. Expecting something perfect from the start is pretty naive. But not unexpected. :)

Next.