Search Results: "ag"

1 December 2010

Pietro Abate: bypassing the apt-get solver

Here at mancoosi we have been working for quite a while to promote and advance solver technology for FOSS distributions. We are almost at the end of the project and it is important to make the mancoosi technology relevant for the community. On goal of the project is to provide a prototype that uses part of the results of mancoosi that can based to install/remove/upgrade packages on a user machine. We certainly don't want to create yet another meta installer. This would be very time consuming and certainly going beyond the scope of the project. The idea is to create a prototype, that can work as an apt-get drop in replacement that will allow everybody to play with different solvers and installation criteria. A very first integration step is a small shell script apt-mancoosi that tries to put together different tools that we have implemented during the project. Roberto wrote extensively about his experience with apt-mancoosi a while ago showing that somehow the mancoosi tools are already usable, as proof of concept, to experiment with all solvers participating to the Misc competition. On notable obstacle we encountered with apt-mancoosi is how to pipe the result of an external solver to apt-get to effectively install the packages proposed as solution. Apt-mancoosi fails to be a drop-in replacement for apt-get exactly for this reason. The "problem" is quite simple : The idea at the beginning was to pass to apt-get, on the command line, a request that effectively represents a complete solution. We expected that, since this was already a locked-down solution, apt-get would have just installed all packages without any further modification to the proposed installation set. Of course, since apt-get is designed to satisfy a user request, and not just to install packages, we quickly realized that our evil plan was doomed to failure. The only option left, was to use libapt directly, but the idea of programming in c++ quickly made me to desist. After a bit of research (not that much after all), I finally found a viable solution to our problems in ''python-apt'' that is a low level and wrapper around libapt. This definitely made my day. Now the juicy details. the problem was to convince apt to completely bypass the solver part and just call the installer. First a small intro. python-apt has an extensive documentation with a couple of tutorials. Using python-apt is actually pretty easy (some snippet from the python-apt doco) :
import apt

# First of all, open the cache
cache = apt.Cache()
# Now, lets update the package list
cache.update()
here we open the cache (apt.Cache is a wrapper around the low level binding in the apt_pkg module), then we update the package list. This is equivalent to apt-get update . Installing a package is equally easy :
import apt
cache = apt.Cache()
pkg = cache['python-apt']

# Mark python-apt for install
pkg.mark_install()

# Now, really install it
cache.commit()
Now, the method mark_install of the module package will effectively run the solver to resolve and mark all the dependencies of the package python-apt. This is the default behavior when apt-get is used on the command line. This method has however three optional arguments that are just what I was looking for, that is autoFix, autoInst and fromUser . The explanation from the python-apt doco is quite clear.
mark_install(*args, **kwds)
Mark a package for install.
If autoFix is True, the resolver will be run, trying to fix broken packages. This is the default.
If autoInst is True, the dependencies of the packages will be installed automatically. This is the default.
If fromUser is True, this package will not be marked as automatically installed. This is the default. Set it to False if you want to be able to automatically remove the package at a later stage when no other package depends on it.
What we want is to set autoFix and autoInst to false to completely bypass the solver. So imagine that an external solver can give use a string of the form : bash+ dash=1.4 baobab- that basically asks to install bash to the newest version, dash at version 1.4 and remove baobab. Suppose also that this is a complete solution, that is, all dependencies are satisfied and there are no conflicts. The work flow of mpm (mancoosi package manager) is as follows : We already have a first prototype on the mancoosi svn. It's not released yet as we are waiting to do more testing, add more options and make it stable enough for testing. Maybe one day, this will be uploaded to debian. This is the trace of a successful installation of a package in a lenny chroot. The solver used here is the p2 solver
dev:~/mpm# ./mpm.py -c apt.conf install baobab
Running p2cudf-paranoid-1.6 solver ...
Validate solution ...
loading CUDF ...
loading solution ...
Summary of proposed changes:
new: 30
removed: 0
replaced: 0
upgraded: 0
downgraded: 0
unsatisfied recommends:: 8
changed: 30
uptodate: 322
notuptodate: 116

New packages: baobab (2.30.0-2) dbus-x11 (1.2.24-3) gconf2 (2.28.1-5)
gconf2-common (2.28.1-5) gnome-utils-common (2.30.0-2)
libatk1.0-0 (1.30.0-1) libcairo2 (1.8.10-6) libdatrie1 (0.2.4-1)
libdbus-glib-1-2 (0.88-2) libgconf2-4 (2.28.1-5) libgtk2.0-0 (2.20.1-2)
libgtk2.0-common (2.20.1-2) libgtop2-7 (2.28.1-1) libgtop2-common (2.28.1-1)
libidl0 (0.8.14-0.1) libjasper1 (1.900.1-7+b1) liborbit2 (1:2.14.18-0.1)
libpango1.0-0 (1.28.3-1) libpango1.0-common (1.28.3-1)
libpixman-1-0 (0.16.4-1) libthai-data (0.1.14-2) libthai0 (0.1.14-2)
libtiff4 (3.9.4-5) libxcb-render-util0 (0.3.6-1) libxcb-render0 (1.6-1)
libxcomposite1 (1:0.4.2-1) libxcursor1 (1:1.1.10-2) libxrandr2 (2:1.3.0-3)
psmisc (22.11-1) shared-mime-info (0.71-3)
Removed packages:
Replaced packages:
Upgraded packages:

Selecting previously deselected package libatk1.0-0.
(Reading database ... 28065 files and directories currently installed.)
Unpacking libatk1.0-0 (from .../libatk1.0-0_1.30.0-1_i386.deb) ...
Selecting previously deselected package libpixman-1-0.
Unpacking libpixman-1-0 (from .../libpixman-1-0_0.16.4-1_i386.deb) ...
Selecting previously deselected package libxcb-render0.
Unpacking libxcb-render0 (from .../libxcb-render0_1.6-1_i386.deb) ...
Selecting previously deselected package libxcb-render-util0.
Unpacking libxcb-render-util0 (from .../libxcb-render-util0_0.3.6-1_i386.deb) ...
Selecting previously deselected package libcairo2.
Unpacking libcairo2 (from .../libcairo2_1.8.10-6_i386.deb) ...
Selecting previously deselected package libdbus-glib-1-2.
Unpacking libdbus-glib-1-2 (from .../libdbus-glib-1-2_0.88-2_i386.deb) ...
Selecting previously deselected package libidl0.
Unpacking libidl0 (from .../libidl0_0.8.14-0.1_i386.deb) ...
Selecting previously deselected package liborbit2.
Unpacking liborbit2 (from .../liborbit2_1%3a2.14.18-0.1_i386.deb) ...
Selecting previously deselected package gconf2-common.
Unpacking gconf2-common (from .../gconf2-common_2.28.1-5_all.deb) ...
Selecting previously deselected package libgconf2-4.
Unpacking libgconf2-4 (from .../libgconf2-4_2.28.1-5_i386.deb) ...
Selecting previously deselected package libgtk2.0-common.
Unpacking libgtk2.0-common (from .../libgtk2.0-common_2.20.1-2_all.deb) ...
Selecting previously deselected package libjasper1.
Unpacking libjasper1 (from .../libjasper1_1.900.1-7+b1_i386.deb) ...
Selecting previously deselected package libpango1.0-common.
Unpacking libpango1.0-common (from .../libpango1.0-common_1.28.3-1_all.deb) ...
Selecting previously deselected package libdatrie1.
Unpacking libdatrie1 (from .../libdatrie1_0.2.4-1_i386.deb) ...
Selecting previously deselected package libthai-data.
Unpacking libthai-data (from .../libthai-data_0.1.14-2_all.deb) ...
Selecting previously deselected package libthai0.
Unpacking libthai0 (from .../libthai0_0.1.14-2_i386.deb) ...
Selecting previously deselected package libpango1.0-0.
Unpacking libpango1.0-0 (from .../libpango1.0-0_1.28.3-1_i386.deb) ...
Selecting previously deselected package libtiff4.
Unpacking libtiff4 (from .../libtiff4_3.9.4-5_i386.deb) ...
Selecting previously deselected package libxcomposite1.
Unpacking libxcomposite1 (from .../libxcomposite1_1%3a0.4.2-1_i386.deb) ...
Selecting previously deselected package libxcursor1.
Selecting previously deselected package libxrandr2.
Unpacking libxrandr2 (from .../libxrandr2_2%3a1.3.0-3_i386.deb) ...
Selecting previously deselected package shared-mime-info.
Unpacking shared-mime-info (from .../shared-mime-info_0.71-3_i386.deb) ...
Selecting previously deselected package libgtk2.0-0.
Unpacking libgtk2.0-0 (from .../libgtk2.0-0_2.20.1-2_i386.deb) ...
Selecting previously deselected package libgtop2-common.
Unpacking libgtop2-common (from .../libgtop2-common_2.28.1-1_all.deb) ...
Selecting previously deselected package libgtop2-7.
Unpacking libgtop2-7 (from .../libgtop2-7_2.28.1-1_i386.deb) ...
Selecting previously deselected package psmisc.
Unpacking psmisc (from .../psmisc_22.11-1_i386.deb) ...
Selecting previously deselected package dbus-x11.
Unpacking dbus-x11 (from .../dbus-x11_1.2.24-3_i386.deb) ...
Selecting previously deselected package gconf2.
Unpacking gconf2 (from .../gconf2_2.28.1-5_i386.deb) ...
Selecting previously deselected package gnome-utils-common.
Unpacking gnome-utils-common (from .../gnome-utils-common_2.30.0-2_all.deb) ...
Selecting previously deselected package baobab.
Unpacking baobab (from .../baobab_2.30.0-2_i386.deb) ...
Processing triggers for man-db ...
Setting up libatk1.0-0 (1.30.0-1) ...
Setting up libpixman-1-0 (0.16.4-1) ...
Setting up libxcb-render0 (1.6-1) ...
Setting up libxcb-render-util0 (0.3.6-1) ...
Setting up libcairo2 (1.8.10-6) ...
Setting up libdbus-glib-1-2 (0.88-2) ...
Setting up libidl0 (0.8.14-0.1) ...
Setting up liborbit2 (1:2.14.18-0.1) ...
Setting up gconf2-common (2.28.1-5) ...

Creating config file /etc/gconf/2/path with new version
Setting up libgconf2-4 (2.28.1-5) ...
Setting up libgtk2.0-common (2.20.1-2) ...
Setting up libjasper1 (1.900.1-7+b1) ...
Setting up libpango1.0-common (1.28.3-1) ...
Cleaning up font configuration of pango...
Updating font configuration of pango...
Cleaning up category xfont..
Updating category xfont..
Setting up libdatrie1 (0.2.4-1) ...
Setting up libthai-data (0.1.14-2) ...
Setting up libthai0 (0.1.14-2) ...
Setting up libpango1.0-0 (1.28.3-1) ...
Setting up libtiff4 (3.9.4-5) ...
Setting up libxcomposite1 (1:0.4.2-1) ...
Setting up libxcursor1 (1:1.1.10-2) ...
Setting up libxrandr2 (2:1.3.0-3) ...
Setting up shared-mime-info (0.71-3) ...
Setting up libgtk2.0-0 (2.20.1-2) ...
Setting up libgtop2-common (2.28.1-1) ...
Setting up libgtop2-7 (2.28.1-1) ...
Setting up psmisc (22.11-1) ...
Setting up dbus-x11 (1.2.24-3) ...
Setting up gconf2 (2.28.1-5) ...
update-alternatives: using /usr/bin/gconftool-2 to provide /usr/bin/gconftool (gconftool) in auto mode.
Setting up gnome-utils-common (2.30.0-2) ...
Setting up baobab (2.30.0-2) ...
Broken: 0
InstCount: 30
DelCount: 0
dev:~/mpm#
I think we'll keep working on this python prototype for a while, but this is not certainly what we want to propose to the community. The mancoosi package manager is probably going to be written in Ocaml and integrated with dose3 and libcudf. This will allow us to gain speed and have a solid language to develop with (nothing against python, but we don't feel that a scripting language is suitable for an essential component as a package manager). Time will tell. For the moment this is just vapor-ware ...

Michal Čihař: Cleanup of personal Debian repository

When I accidentally looked at my personal Debian repository, I realized that it is time to remove obsolete things from it. There were packages for polld, for which which I don't see need anymore (it is now properly handled by kernel), hacked acpid packages, which were doing special things for Toshiba M30 notebook (which I don't use with Linux anymore and nobody else probably knew about these packages) or wmmp, a Window Maker applet for MPD, which I haven't used for quite a long time. All those are gone now and I don't think anybody will miss them (if you do, please tell me). What stayed there are daily snapshots of dictionaries for StarDict and gnaughty package, which was rejected as possibly causing troubles.

Filed under: Debian 0 comments Flattr this!

Raphaël Hertzog: Support 5 free software with Flattr

Flattr FOSS LogoA new month means new free software projects to support with Flattr FOSS. I m happy to see that it s gaining traction outside of the Debian world as well. I saw quite a few new entries for free software projects so that I don t have to fear running out of suggestions in the next few months. :-)
Let s go over the 5 projects that I recommend you for December:
  1. Getting Things GNOME (flattr link) is a GNOME task manager. Its name is a play on David Allen s Getting Things Done (GTD) but the software doesn t enforce the GTD methodology. You can implement GTD however since the software is very flexible and lets you organize the tasks with arbitrary tags. It s a promising software and it can be extended with many plugins.
  2. The W3C validator (flattr link) is used by thousands of web developers to verify that their HTML pages are well formed. But did you know that it was free software? Yes you can contribute code, or you can help them with a flattr.
  3. Bitlbee (flattr link) is a gateway between IRC and many other instant messaging protocols. Geeks are known to be IRC addicted (at least I am using it for Debian development) and with bitlbee it s one reason less to watch something else than the IRC client. :-)
  4. Arch Hurd (flattr link) is a port of the ArchLinux distribution to the GNU Hurd kernel.
  5. Paste.debian.net (flattr link) is a simple website where you can share some textual content for a limited amount of time (you usually paste the content from some other applications, hence its name). Very handy when you want to quickly show something to others on IRC. This service is run by Debian developer Alexander Wirt.
That s it for this month. A quick question to finish this issue: I count at least 15 Debian contributors using Flattr currently, would you be interested by a small directory listing them?

One comment Liked this article? Click here. My blog is Flattr-enabled.

Dirk Eddelbuettel: RQuantLib 0.3.5

The new RQuantLib release 0.3.5 is now on CRAN and in Debian. RQuantLib combines (some of) the quantitative analytics of QuantLib with the R statistical computing environment and language. Most of the changes were made two and four weeks ago: first in response to some warnings triggered by R 2.12.0 on the included manual pages which needed a brush-up, and then again is some consolidation of manual pages and some other minor tweaks. The release was then held back at CRAN as we noticed that manual pages, when collated to a single large document, triggered a segmentation fault in the latex compiler. Oddly enough only in Europe (if the a4paper option was used) and not here (where I use uspaper). Long story short, this turns out to be a bug in the latex toolchain (which we reported as Debian bug report 604754) which is apparently is known but has no known fix yet (a sample file was supplied with the bug report if you want to take a look). With that, special thanks go to Kurt Hornik and Brian Ripley on the R Core team who made a change to how R processes the manual which made it resilient to the latex bug so that normal release of the package could proceed (and the shiny manual is available too). Thanks to CRANberries, there is also a diff to the previous release 0.3.4. Full changelog details, examples and more details about this package are at my RQuantLib page.

Jaldhar Vyas: The Anaclepses of Darwin Fenderman

Nanowrimo winner 2010 I've been participating in National Novel Writing Month for several years now but this is the first time I've actually managed to complete writing 50,000 words in one month. My Science Fiction novel "The Anaclepses of Darwin Fenderman" is set in a future where the last individualist must fight for free will against, robots, telepathic dogs, and Phillip K. Dick. It came out fairly well I think considering the haste in which it was written. Of course here's a hell of a lot of clean up that needs to be done before I can dare to show anyone else but in the coming months I want to see if I can get it into a fit state for publishing. Meanwhile, here's a teaser:
"Where is it then?"
"Out there somewhere.", he said gesturing vaguely at the window.
"Beyond the Hudson? But there's nothing but wasteland out there."

Axel Beckert: Useful but Unknown Unix Tools: htop

You probably know about top , in Debian and Ubuntu part of the procps package. Ever wanted to see CPU and memory usage as bars and not numbers? Ever wanted to kill a process from inside top by just selecting its row instead of having to type its pid? Ever tried to press a cursor key inside top? It makes a noise and says Unknown command - try h for help . Short said: top is not that interactive. Ever wanted top to be more colorful? Well, there is a solution to all these issues. It s called htop (Debian package) and is some kind of colorful, ncurses based cross-over between top and a Midnight Commander for processes:

htop on single core machine htop on single core machine with MC theme htop on idling 16 core machine htop on 16 core machine under load
More screenshots of htop at screenshot.debian.net and at the project s site at SourceForge. It can do many things, top can t do:

One more cool thing about htop: It s also available on the Nokia N900 as app . The only thing I found so far that top can do, but htop can t, is top s batch mode (e.g. with top -b -n 1 ) where it runs non-interactively and its output can even be piped to other processes. The Xymon/Hobbit monitoring system uses that as input for some machine statistics.

30 November 2010

Sergio Talens-Oliag: freebsd-utils-8.1-2 to 2.4.patch

Sergio Talens-Oliag: The FreakyWall (Part 3: Packages)

In this post I'll describe the changes made to the kernel and some of the Squeeze packages for the Freaky Wall. The plan is to submit whishlist bugs to the BTS on the hope of having all what is needed for this project available on Debian after the Squeeze release, as my feeling is that a freeze is not the right time to push this changes... ;) I'm giving access here to all the changes made to the source packages, but if anyone wants the binary packages (amd64 only) send me an email and I'll give you the URL of an apt repository that contains all the modified packages (it's the one at work, that contains other modified packages) or, if there is interest, I can put them on people.debian.org. Kernel To be able to build the firewall we need a kFreeBSD kernel with some options not compiled on the version distributed with Debian. To compile the kernel I've followed the procedure described on the following debian-bsd mailing list post: http://lists.debian.org/debian-bsd/2010/09/msg00023.html Basically I've done the following:
    apt-get build-dep kfreebsd-8
    apt-get source kfreebsd-8
    cd kfreebsd-8-8.1
    cat >> debian/arch/amd64/amd64.config << EOF
    # Add pflog, pfsync, ALTQ and CARP support
    # ----------------------------------------
    # http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/firewalls-pf.html
    device      pf
    device      pflog
    device      pfsync
    options         ALTQ
    options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
    options         ALTQ_RED        # Random Early Detection (RED)
    options         ALTQ_RIO        # RED In/Out
    options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
    options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
    options         ALTQ_NOPCC      # Required for SMP build
    # http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/carp.html
    device      carp
    EOF
    vi debian/changelog 
    dpkg-buildpackage -B -uc
Once the package was built I installed the new kernel package and rebooted the machine. Utilities To be able to use some utilities related to pf I have built patched versions of three packages: On the next post I'll describe how I've configured the system, the network interfaces and the different utilities patched and compiled on this post.

Sergio Talens-Oliag: libpcap-1.1.1-2 to 2.1.patch

Sergio Talens-Oliag: tcpdump 4.1.1-1 to 1.1.patch

Jordi Mallach: The unforeseen consequences of our GR 11 Summer walk

I knew walking all over the Pyrenees during a whole month would come with some side effects. I could imagine having a few muscular issues in my legs and some back pain for a while after getting back; after all we did over 7 hours of exercise every single day during a month. What I got after our hike was totally unexpected... I should have run a half marathon in Gandia on the 21st, and instead I stayed at home doing some assignments. When we came back, I was in a really good condition, and wanted to keep the good shape we had built the month before. Given I haven't been able to swim for nearly a year now, due to the Misterious Shoulder Injury and I don't have enough time to go out cycling regularly, I centered my efforts in running, with the idea of starting to get prepared for cross/mountain races this season. Training had to wait a little because some days after getting back, I got a sudden pain on both knees, which even made climbing stairs difficult. It stuck for a while, and when it finally went away, more or less, I went ahead and tried going out for a run. I haven't been training for over two years, so I indended to increase the distances proggressively. The first day I went out, I ran really nicely for the first 25 minutes, and suddenly I started getting an intense pain in the outer side of my right knee, to the point I could barely walk back home. I've tried letting them rest for weeks, and every time I try them, the pain comes back. My physiotherapist says my body has developed quite a few asymmetries, probably caused by the many days I had to walk crippled by a big blister in my right foot. As time passes, I feel I'm losing all the physical improvements I had developed during the summer, and I can't do anything about it. The new approach to tackle this is yoga. I know my body isn't too elastic, and the lack of stretching during the walk made it even worse, so I'll try to forget about real traning during some time and focus on healing my muscles. Hopefully, this will help me resolve the nastiest physical problem I've ever come across.

Peter Makholm: Wrapping mod_perl in Plack

Plack is one of these wonderful adventures in the modern Perl world that makes it fun to write Web applications in Perl again. But I have a few applications written with Apache/mod_perl and they are not fun to work with. So what would you do? One option would be to take the long road and port these apps to use Plack instead of messing around with Apache2::RequestRec. For this to work you might need to review the full code base before seeing any signs of progress. Another option is to mock your Apache2::RequestRec object using Plack. This is the road explored by Plack::App::WrapApacheReq. With very little work this enables you to run your mod_perl application with any Plack handler you want. You can run your application as a stand alone server or serve it with nginx trough FastCGI. But the fun doesn t stop here. Debugging and profiling mod_perl have always been a PITA, but with Plack::App::WrapApacheReq it is easy. Just take the generic.psgi example and it enables you to run the Perl debugger easy or just to use NYTProf on your request handling code. The initial ideas for writing Plack::App::WrapApacheRec came from my mocking code to test a legacy mod_perl application. Even this gets more fun by using Plack. I havn t tested it yet, but with Plack::Test and Plack::Test::ExternalServer it should be trivial to run the same set of tests directly in a single process test suite or against your deployed server. Plack::App::WrapApacheRec is still in it s infancy. It only mocks as much of Apache2::RequestRec as I need to run a single legacy application and to run Plack::Handler::Apache2 (as an absurd example, but yes we are self hosting). But I think that with very little work we should be able to run most mod_perl applications. Take it out for a ride, if it complains about a missing method please report it with CPAN s RequestTracker. Patches would be appreciated (or pull requests on github), but just a list of unimplemented method you need would be excellent. Plack is fun, now working with mod_perl applications might become fun too.

Joey Hess: debugging musings

Debugging a computer program is such an interesting activity because it's not really a matter of fixing a program. It's a matter of fixing your own understanding to the point that the cause of the bug becomes obvious. So debugging means constantly challenging your assumptions, constantly looking for the overlooked insignificant thing that turns out to be crucial. I mostly debug by print statements. I don't like it much, but it works. I add some print statements around the problem area and see if what's going on matches what the model in my head thinks should be going on. If it doesn't, I try to fix the model, and if it does, I focus the print statements in tighter and tighter, deeper and deeper, until I finally reach the bug. It's nasty and ah-hoc, but it works really well, mostly. And I know I'm in good company doing it. But that method can fail; I myself can end up caught in a loop. Like today, tracking down an obscure bug in Branchable's auditing subsystem, when I found myself at what was apparantly the point of a bug: Function A was supposed to call function B, but B never seemed to be called. So surely A was buggy, and I instrumented it to bits with print statements and stuff, and started wondering if the libraries it relied on were somehow breaking it in non-obvious ways before it could call B, or maybe there was a bug in the runtime language that was breaking the call to B. Going deeper and deeper didn't help, because the thing I was overlooking was up at the top level -- one of the first print statements I inserted showed that all this code was running twice. Why? Didn't seem relevant to the bug so I ignored it. Of course what I was ignoring was a symptom of the real problem: The first run was by another subsystem, that happened to redefine function B temporarily.. Actually, one of the nice things about Haskell is that print statements are not very helpful in debugging pure code. Pure code tends to get loaded up in ghci and tested interactively, and the bug is then clear. Or a quickcheck test case gets written with an invariant that the pure code should satisfy. I spent a whole day a few weeks ago writing an inverse form of a buggy function in git-annex just so I could feed them both into quickcheck and automate debugging. (Most of that time was spent wrestling with unicode decomposition .. ugh.) Being used to getting down in the buggy code with print statements, that at the time felt rather like a waste of time unrelated to getting on with making my program work, but it was very much worth it since I got my program working 100% in even crazy edge cases, and got a test suite for free, too. Most of my debugging of Haskell code so far has not involved bugs in pure code. The most memorable bug was actually a bug in ghc's IO manager, which I'd have never tracked down without Josh's help. The problem with finding bugs in the runtime environment or compiler is that after you've found enough over the years, you're more likely to go down false paths distrusting runtime environments, like I did with the bug today. The beginner's assumption is that the runtime environment just works, but I by now may have over-challenged that assumption. :)

Debian News: DebConf 11 to take place July 24-30, 2011

DebConf 11 will take place from July 24 to 30, 2011, in the city of Banja Luka, Republika Srpska, Bosnia and Herzegovina.

The final calendar of the event is:

The call for papers and registration dates are still to be decided.

More information at the DebConf 11 website.

Debian News: Training Session on Python Packaging

As part of the Training Sessions initiative organized by the Debian Women project, this week a lesson about Python packaging will be held. The lesson, which is aimed at an intermediate audience, will focus specifically modules and applications packaging and will be held by Piotr O arowski who is member of Python Applications Packaging Team and Debian Python Modules Team.

The session will take place, as usual, on #debian-women channel on irc.debian.org on Thursday, 2nd December at 20:00 UTC.

Cause of the advanced topic is madatory to have some knowledge about Debian packaging (see, for example Lars Wirzenius tutorial) and about general Python development.

The lesson will consists of a first, theoretical, part in wich Piotr will explain us some basics; and of a Q&A part, entirely dedicated to more advanced problems.

See you on Thursday!

Francesca Ciceri

Michal &#268;iha&#345;: New phpMyAdmin features - AJAX and Javascript

phpMyAdmin 3.4 is just in the alpha state (second alpha release being done right now), so it's time to continue in presenting new features. As usual, just download 3.4 alpha or try them on our demo server. If you find any problems while playing with it, do not hesitate and ask or report bug. phpMyAdmin is now much faster in modern browsers, thanks to using AJAX in lot of places instead of loading whole new page. This is probably the most intrusive change made in 3.4 and at time of writing this post, there are still some rough edges, but the developers are polishing them right now. Also there is not only AJAX, but lot of user input can be validated on client side now allowing faster reactions on invalid input. This feature was implemented by Ninad Pundalik during Google Summer of Code 2010.

Filed under: Phpmyadmin 0 comments Flattr this!

Tollef Fog Heen: My Varnish is leaking memory

Every so often, we get bug reports about Varnish leaking memory. People have told Varnish to use 20 gigabytes for cache and they discover the process is eating 30 gigabytes of memory and they get confused about what's going on. So, let's take a look. First, a little bit of history. Varnish 2.0 had a fixed per-object workspace which was used for both header manipulations in vcl_fetch as well as for storing the headers of the object when vcl_fetch was done. The default size of this workspace was 8k. If we assume an average object size of 20k, that is almost 1/3 of the store being overhead. With 2.1, this changed. First, vcl_fetch doesn't have obj any longer, it only has beresp which is the backend response. At the end of vcl_fetch, the headers and other relevant bits of the backend response are copied into an object. This means we no longer have a fixed overhead, we use what we need. Of course, we're still subject to malloc's whims when it comes to page sizes and how it actually allocates memory. Less overhead means more objects in the store. More objects in the store, means, everything else being equal, more overhead outside the store (for the hash buckets or critbit tree and other structs). This is where lots of people get confused, since what they see is just Varnish consuming more memory. When moving from 2.0 to 2.1, people should lower their cache size. How much depends on the amount of objects they have, but if they have many and small objects, a significant reduction might be needed. For a machine dedicated to Varnish, we usually recommend making the cache size be 70-75% of the memory of the machine. A reasonable question to ask at this point is what all this overhead is being used for. Part of it is a per-thread overhead. Linux has a 10MB stack size by default, but luckily, most of it isn't allocated, so it only counts against virtual, not resident memory. In addition, we have a hash algorithm which has overhead and the headers from the objects are stored in the object itself and not in the stevedore (object store). Last, but by no means least, we usually see an overhead of around 1k per object, but I have seen up to somewhere above 2k. This doesn't sound like much, but when you're looking at servers with 10 million objects, 1k of overhead means 10 gigabytes of total overhead, leading to the confusion I talked about at the start.

Richard Darst: The first Debian-NYC Novice Night

Debian-NYC is having it's first Novice Night this Wednesday, December 1st, hosted at Eyebeam. It will run from about 18:30 until 22:00, but people can come and go.What are Novice Nights? They are a time for people to come and informally get help with Debian and derivatives. It's like an "installfest", but not just about installing. It's about getting help with what you already have installed. It's about helping to solve little problems. It's about meeting the community, and learning what resources are available, so that you can help yourself in the future.We'll see how the first one goes, and improve from there. I'm looking forward to it, and seeing how this series evolves.If you know anyone in the NYC area, point them to the Novice Night announcement and planning page and encourage them to come. Groups are welcome. There is also lots of other Debian-NYC events that happen, too.Tags: debian-nyc

Sam Hartman: Implementation Progress

At the end of September, things were quite exciting as we had our first project meeting. At that meeting those in the room saw a demonstration of the Moonshot GSS EAP mechanism and we discussed a number of open issues and began to plan for our test infrastructure. We ve made significant progress on the specification front and on explaining Moonshot to important communities since then. However there has been little public progress on the implementation front. Unfortunately, getting the necessary legal clearance and agreements to release code often takes longer than anyone would like; that is what is happening here. We re all eagerly awaiting final approval from the lawyers and JANET(UK) management. However, things have been moving behind the scenes. Throughout much of October, Luke Howard and Linus Nordberg were working on their respective parts of the code. I ve also been working on putting together the test and build infrastructure. As we discussed at the meeting, we re going to use Debian and Ubuntu as the basis for our testing. For example, we hope to release virtual machine images for these platforms for the major Moonshot components. Thus the primary build environment for our testing and virtualization will be for Debian. I ve been putting together that here. Right now, that branch will pull together packages of the SAML infrastructure that we need. I ve also been looking into virtualized test frameworks and believe I ve found one that meets our needs. I ve also put together some primitive build infrastructure that is independent of packaging available here. I ve set up a buildbot that builds both environments. So, as the code becomes available we ll be in a good position to start making it available.

Sam Hartman: Abfab at IETF 79

The ABFAB working group, which will be standardizing technologies that Moonshot depends on, had its first meeting at IETF 79 in Beijing, China. The meeting was quite productive. Because the meeting was the first of the working group, there were some introductory presentations. A group of authors are putting together a proposed architecture document; we presented the current state of our work. However things have evolved significantly since the working group meeting and I think it will make more sense to wait a couple of weeks to discuss the architecture document. Most of the time was spent on two presentations. The first was the status of the GSS mechanism. We discussed issues that were discovered while implementing the EAP GSS-API mechanism. Discussion in the room tended to support the proposals made in the slides. A few issues will need to come to the list. We had the most interesting discussion of SAML AAA integration. Minutes are available.

Next.

Previous.