Search Results: "gspr"

13 May 2022

Arturo Borrero Gonz lez: Toolforge GridEngine Debian 10 Buster migration

Toolforge logo, a circle with an anvil in the middle This post was originally published in the Wikimedia Tech blog, authored by Arturo Borrero Gonzalez. In accordance with our operating system upgrade policy, we should migrate our servers to Debian Buster. As discussed in the previous post, one of the most important and successful services provided by the Wikimedia Cloud Services team at the Wikimedia Foundation is Toolforge. Toolforge is a platform that allows users and developers to run and use a variety of applications with the ultimate goal of helping the Wikimedia mission from the technical side. As you may know already, all Wikimedia Foundation servers are powered by Debian, and this includes Toolforge and Cloud VPS. The Debian Project mostly follows a two year cadence for releases, and Toolforge has been using Debian Stretch for some years now, which nowadays is considered old-old-stable . In accordance with our operating system upgrade policy, we should migrate our servers to Debian Buster. Toolforge s two different backend engines, Kubernetes and Grid Engine, are impacted by this upgrade policy. Grid Engine is notably tied to the underlying Debian release, and the execution environment offered to tools running in the grid is limited to what the Debian archive contains for a given release. This is unlike in Kubernetes, where tool developers can leverage container images and decouple the runtime environment selection from the base operating system. Since the Toolforge grid original conception, we have been doing the same operation over and over again: We ve done this type of migration several times before. The last few ones were Ubuntu Precise to Ubuntu Trusty and Ubuntu Trusty to Debian Stretch. But this time around we had some special angles to consider.

So, you are upgrading the Debian release
  • You are migrating to Debian 11 Bullseye, no?
  • No, we re migrating to Debian 10 Buster
  • Wait, but Debian 11 Bullseye exists!
  • Yes, we know! Let me explain
We re migrating the grid from Debian 9 Stretch to Debian 10 Buster, but perhaps we should be migrating from Debian 9 Stretch to Debian 11 Bullseye directly. This is a legitimate concern, and we discussed it in September 2021. A timeline showing Debian versions since 2014 Back then, our reasoning was that skipping to Debian 11 Bullseye would be more difficult for our users, especially because greater jump in version numbers for the underlying runtimes. Additionally, all the migration work started before Debian 11 Bullseye was released. Our original intention was for the migration to be completed before the release. For a couple of reasons the project was delayed, and when it was time to restart the project we decided to continue with the original idea. We had some work done to get Debian 10 Buster working correctly with the grid, and supporting Debian 11 Bullseye would require an additional effort. We didn t even check if Grid Engine could be installed in the latest Debian release. For the grid, in general, the engineering effort to do a N+1 upgrade is lower than doing a N+2 upgrade. If we had tried a N+2 upgrade directly, things would have been much slower and difficult for us, and for our users. In that sense, our conclusion was to not skip Debian 10 Buster.

We no longer want to run Grid Engine In a previous blog post we shared information about our desired future for Grid Engine in Toolforge. Our intention is to discontinue our usage of this technology.

No grid? What about my tools? Toolforge logo, a circle with an anvil in the middle Traditionally there have been two main workflows or use cases that were supported in the grid, but not in our Kubernetes backend:
  • Running jobs, long-running bots and other scheduled tasks.
  • Mixing runtime environments (for example, a nodejs app that runs some python code).
The good news is that work to handle the continuity of such use cases has already started. This takes the form of two main efforts:
  • The Toolforge buildpacks project to support arbitrary runtime environments.
  • The Toolforge Jobs Framework to support jobs, scheduled tasks, etc.
In particular, the Toolforge Jobs Framework has been available for a while in an open beta phase. We did some initial design and implementation, then deployed it in Toolforge for some users to try it and report bugs, report missing features, etc. These are complex, and feature-rich projects, and they deserve a dedicated blog post. More information on each will be shared in the future. For now, it is worth noting that both initiatives have some degree of development already. The conclusion Knowing all the moving parts, we were faced with a few hard questions when deciding how to approach the Debian 9 Stretch deprecation:
  • Should we not upgrade the grid, and focus on Kubernetes instead? Let Debian 9 Stretch be the last supported version on the grid?
  • What is the impact of these decisions on the technical community? What is best for our users?
The choices we made are already known in the community. A couple of weeks ago we announced the Debian 9 Stretch Grid Engine deprecation. In parallel to this migration, we decided to promote the new Toolforge Jobs Framework, even if it s still in beta phase. This new option should help users to future-proof their tool, and reduce maintenance effort. An early migration to Kubernetes now will avoid any more future grid problems. We truly hope that Debian 10 Buster is the last version we have for the grid, but as they say, hope is not a good strategy when it comes to engineering. What we will do is to work really hard in bringing Toolforge to the service level we want, and that means to keep developing and enabling more Kubernetes-based functionalities. Stay tuned for more upcoming blog posts with additional information about Toolforge. This post was originally published in the Wikimedia Tech blog, authored by Arturo Borrero Gonzalez.

23 March 2020

Bits from Debian: New Debian Developers and Maintainers (January and February 2020)

The following contributors got their Debian Developer accounts in the last two months: The following contributors were added as Debian Maintainers in the last two months: Congratulations!

28 April 2011

Jan Hauke Rahm: Save the environment and come to LinuxTag 2011 in Berlin

It s the time of the year again. LinuxTag is right up and, of course, Debian is part of it. But, it wouldn t be us if we didn t have any problems at all. Well, the situation isn t real bad but could deserve some improvement. Improvement about what, you ask? Isn t that obvious? It s man power that we need (women shall feel included :)). Now, LinuxTag 2011 is going to happen, with or without you, on May 11th to 14th. Needless to say that you ll be missing quite a lot if you don t attend. I mean, even Zack will be there. And so will I (most probably at least). And even more Who s willing to miss that, really? So, pack your stuff, get a train ticket or your car or your friend s car or the car of a friend of your friend who knows a friend who doesn t know you but still wants to give you his car which you don t even need since you already have the car of that friend of your friend and now you have two cars and can even bring another bunch of fellas who probably own cars themselves and now you re thinking about the environment because of all the cars that will come to Berlin and because of that you suddenly buy a train ticket and leave all three cars at home and in the end only one thing counts: you ll be there. Oh, and before you drive off to your local train station, please put your name into our neat little list so we can count on you as booth staff that s because we need man power in case you missed that section or somehow forgot it while thinking about that guy who is the friend of your friend that owns a car but rather comes by train as it saves the environment.

21 April 2011

Joachim Breitner: A talk on Church s result about the Entscheidungsproblem

A newly founded Logic Group in Mumbai, which seems to be a private thing but some professors from the IIT Bombay are taking part, has approached me and asked if I can give a talk about Church s lambda calculus and how he used it to show that Hilbert s Entscheidungsproblem is not solvable. They are starting a series of talk on logic on the occasion of Turing s 100th birth year and because I am finally leaving the IIT campus in two days, my talk turned out to be the opening talk for the series. Roughly guessed about fifty people attended, some professors but most of them students from fields other than mathematics and computer science I hope that they were not too confused by distinguishing truth and provability, expressability and computability, of which I did not give a proper explanation.
I started with a historical exposition of the state of logic in the beginning 20th century, then gave an introduction to lambda calculus, encoding of natural numbers in then, G del encoding. Finally I sketched the proof of the unsolvability of the question whether a lambda term has a normal form and then concluded by showing how this implies that the Entscheidungsproblem is not solvable.
I have before hand written out the talk in full, and again I ended up saying completely different things or at least said things completely different. Nevertheless, I am sharing the (planned) text of my talk, including the timeline that I draw on the whiteboard.

23 January 2011

Joachim Breitner: A Mistake in Church s Paper

For a course at the IIT Bombay on functional programing, I was preparing a presentation on Alonzo Church s theorem, based on his original 1936 paper An Unsolvable Problem of Elementary Number Theory where he first showed that the Entscheidungsproblem is not solvable. In that paper, he states a theorem (Theorem II) If a formula has a normal form [ ] any sequence of reductions of the formula must (if continued) terminate in the normal form . This seems to be wrong. Consider the lambda expression KI , where K=( xy.x), I=( x.x) and =( x.xx)( x.xx). This has a normal form I. But because reduces to , there is a non-terminating sequence KI KI , in contradiction to Church s claim. The same example contradicts his Theorem III: If a formula has a normal form, every well-formed part of it has a normal form. , which is used in the very proof of his main result, Theorem XVIII. There is no proof for Theorem II in this paper. He cites it from the then forthcoming paper Some properties of Conversion by him and J. B. Rosser. There, proofs are given (see Theorem 2 and its Corollary) but they are quite impenetrable to me. So I was searching for some correction paper, or any other discussion of this issue, but could not find any. Does any reader of this blog know more? What happened in that times when a paper had an error in some proof? What would happen now? Or is this not an error after all, but just a subtle difference between the definitions of -calculus as we know and as they introduced it? Unrelated to this question: The professor asked me to write down a summary of (my interpretation of) the importance and impact of Church s paper with regard to G del and Hilbert s program, and how that relates to B hm s theorem. In the spirit of sharing, I have uploaded my thoughts on Church and B hm, comments are welcome. Update: Christian von Essen solved the riddle in a comment to this post: Church only considers lambda abstractions as well-formed when the bound variable actually occurs (freely) in the abstracted term. This does not allow for the K combinator and thus there is no problem.

19 February 2009

Yves-Alexis Perez: Cookies!

Hey hey, look what I had in my mailbox this morning: Cookies! Cookies! Just in time for Lenny release, here are my precious cookies for BugSprint. Thanks Franklin!

3 December 2008

Stefano Zacchiroli: thanks for the cookies Joss

You remember the bug sprint, don't you? I've already argued how nice the idea was, being a stereotypical example of how in Debian efficiency and fun can (and often must) go together. Well, Joss' cookies have been delivered to my office last Wednesday, unfortunately I wasn't there!!!, due to my participation first in the Debian QA Meeting in Extremadura, and then another academic workshop in Amsterdam. Sadly, I had to "delegate" the task of cookie tasting to other fellow Debian-ers which happen to work with me. Today I've finally got back to my office, and to much of my surprise, 5 cookies were still there waiting for me. In spite of the bad previsions about their health status after a week, they were delicious: simple biscuits, not too sweet, with walnut in pieces, and dark chocolate. So long Joss, and thanks for all the fish cookies.

3 November 2008

Enrico Zini: rc-buggy packages sorted by popularity

rc-buggy packages sorted by popularity After doing a round of tag reviews so that we release lenny with the latest tag contributions, I still had a bit of spare time. Since people are urging to fix the last bugs I thought I could contribute a list of RC-buggy packages sorted by popularity:
#!/usr/bin/python
import urllib2
import sys, re #, gzip
PKGRE = re.compile(r'<strong>Package: </strong><a href="[^"]+" name="([^"]+)">')
POPCONRE = re.compile(r'^(\d+)\s+(\S+)')
PKGMAPRE = re.compile(r'^([^(]+)\(([^)]+)')
COMMASPLIT = re.compile(r'\s*,\s*')
# Allow to download other package views
if len(sys.argv) > 1:
    url = sys.argv[1]
else:
    url = "http://bts.turmzimmer.net/details.php"
def read_packages(url):
    "Download the list of rc-buggy packages"
    fd = urllib2.urlopen(url)
    for line in fd:
        mo = PKGRE.search(line)
        if mo:
            yield mo.group(1)
# FIXME: this is a deprecated data source, but at the moment it is just what
#        is needed
def read_bintosrc(url = "http://qa.debian.org/data/ddpo/results/ddpo_packages"):
    "Download the binary to source package map"
    fd = urllib2.urlopen(url)
    for line in fd:
        mo = PKGMAPRE.search(line)
        if mo:
            src = mo.group(1)
            bin = COMMASPLIT.split(mo.group(2))
            for p in bin:   
                yield p, src
def read_popcon(url = "http://popcon.debian.org/by_inst"):
    "Download popcon information"
    fd = urllib2.urlopen(url)
    # It doesn't work on urllib2 objects because it wants tell()
    #fd = gzip.GzipFile(fileobj=fd, filename=url, mode="rb")
    for line in fd:
        mo = POPCONRE.match(line)
        if mo:
            yield mo.group(2), int(mo.group(1))
print "Reading mapping of binary packages to source packages..."
bin2src = dict(read_bintosrc())
print "Reading popcon scores..."
# We actually want the score of source packages, and for it we use the score of
# its binary package with the highest score
scores = dict()
for pkg, score in read_popcon():
    pkg = bin2src.get(pkg, pkg)
    if pkg not in scores:
        scores[pkg] = score
#scores = dict(read_popcon("http://popcon.debian.org/by_inst.gz"))
print "Reading list of rc-buggy packages..."
# The bug page has a mix of source and binary packages: convert all to source
# packages
packages = sorted(read_packages(url), key=lambda x:scores.get(bin2src.get(x,x), 0))
for p in packages:
    print scores.get(bin2src.get(p,p), 0), p
This can be run on people.debian.org as ~enrico/popzimmer (0 is the rank of packages that for some reason cannot be mapped to any rank):
$ ~enrico/popzimmer 
Reading mapping of binary packages to source packages...
Reading popcon scores...
Reading list of rc-buggy packages...
0 roxen-fonts-iso8859-2
2 dpkg
11 pam
61 glibc
61 libc6
123 initramfs-tools
149 bind9
150 grub
180 portmap
213 nfs-common
279 libgl1-mesa-dri
298 libsnmp-base
298 snmpd
309 avahi
397 mysql-dfsg-5.0
408 xserver-xorg-input-wacom
409 xserver-xorg-video-vesa
428 docbook-xml
499 openoffice.org-core
567 xine-lib
600 evolution-data-server
669 eog
671 xserver-xorg-video-intel
691 epiphany-browser
691 epiphany-gecko
695 kdelibs4c2a
698 uswsusp
754 guile-1.6
959 gpm
964 cups
990 kdebase
1002 ghostscript
1069 giflib
1076 xulrunner-1.9
1177 libapache2-mod-perl2
1217 vlc-nox
1229 kernel-package
1311 kdeutils-dev
1520 libparted1.8-10
1767 wireshark
1785 selinux-policy-default
1908 dia
1943 htop
1983 boost
2140 cantlr
2145 transmission
2159 emacs22
2312 swfdec-mozilla
2449 eclipse
2449 libswt3.2-gtk-java
2728 lilo
2898 libnss-ldap
3013 java-access-bridge
3051 blender
3099 libtemplate-perl
3499 fglrx-atieventsd
3602 virtualbox-ose
3619 jfsutils
3806 anjuta
3962 kbuild
4076 libgnustep-gui0.14
4222 libwebkit-1.0-1
4332 rosegarden
4412 matplotlib
4412 python-matplotlib
4950 mail-notification-evolution
5195 nut-cgi
5273 sbcl
5538 ruby1.9
5679 kmymoney2
6076 websvn
6147 miro
6264 gnat-4.1
6431 gallery2
6478 gnome-chess
6512 joystick
6862 istanbul
6937 wordpress
7476 luatex
7635 csound
7635 python-csoundac
8257 libpam-mount
8471 wmcpuload
8631 nbsmtp
8642 oxine
8774 axiom
8949 libengine-pkcs11-openssl
9052 clamav-getfiles
9221 open-iscsi
9262 ptex-bin
9701 egroupware-core
10012 debian-cd
10127 libjogl-java
10453 libjson-xs-perl
10698 gcjwebplugin
10736 request-tracker3.6
11294 python-extclass
11518 rkward
11600 alevt
12929 otrs2
13205 bindgraph
13956 icecc-monitor
14223 netmaze
14505 cdebconf
14526 mumble
14687 fnonlinear
14724 recite
14789 jbidwatcher
15766 libnss-ldapd
15812 glpi
15905 lustre-source
15982 win32-loader
16232 ampache
17168 libghc6-missingh-dev
17311 acl2
17830 apertium
18462 emacspeak
18469 glassfish
19257 isight-firmware-tools
19719 libnagios-object-perl
20144 libwx-perl
20465 dns2tcp
21614 python-kinterbasdb-dbg
21690 hf
21902 libcobra-java
22365 tomcat6
22573 libphp-snoopy
23501 coco-java
23926 boxbackup-server
24009 libghc6-wash-dev
26323 libdesktop-notify-perl
26439 opendb
28426 adolc
28795 xorp
29986 libgdata-java
31262 libjboss-serialization-java
31472 pixelpost
31658 cournol
32939 mediamate
34365 gforge-plugin-scmcvs
34659 libgearman-client-async-perl
42233 libhamcrest-java
48076 mahara
55303 mxallowd
85881 roxen-fonts-iso8859-1

So, if you don't know from what package to start fixing bugs, "Begin at the beginning,", the King said, very gravely, "and go on till you come to the end: then stop."

29 October 2008

Joost Yervante Damad: RC-Bugs / Debian Bug Sprint

The Debian Lenny release has still quite a big list of Release Critical bugs, and as Debian developer I feel that I should do my share, and at least look at the list of bugs and see if there's any I could do something for.

Almost always though, I find that I won't touch the remaining bugs in the list.because of one or more of these reasons:

  1. a package I really don't care about
  2. hugely complex package or might also use something obscure like cddb
  3. it seems like people are already looking into it
  4. it requires a political solution, not a technical one

As part of the effort to get Lenny released, Joss Mouette started the Debian Bug Sprint.
This is really a cool concept, and it is a shame not more people participated.

The reason for me it is cool is that it forced me to break the rules I mentioned above, because I got a bug which fitted 1 and 4 of my list above!

Turned out this bug is really a border case in interpretation of Debian policy.

The package is perfectly usable without any extern dependencies, hence it is currently in the main section of Debian. However it doesn't end there. The package also has a download script that can fetch firmware images for certain printers. It appears for people with these printers the package is NOT usable without external files.

Aparantly though, the current, as one person involved in the bug calls it, "spirit" of Debian is to tolerate this, as it is good enough that it is usable without external dependencies for SOME persons.

However the bug submitter doesn't agree, and thinks this is a case that needs addressing withing the Debian project.

I suggested splitting up the package and moving the download script to contrib, but this was mostly dismissed as idea.

The end result is that the maintainer decided to escalate the problem to the Debian CTTE.
This means it probable won't be solved by today, soo I will have to bake cookies for someone :)

28 October 2008

Stefano Zacchiroli: releasing yummy cookies

Cookies, bugs, and releases Christian knows how to be very inspiring and provocative with his blog posts, the APT case is a very good example of how useful that can be. I believe today's post is yet another example of that, many thanks Bubulle! Still, I stopped believing in the incompatibilities among having discussions (possibly heavy, possibly flamish) and releasing. Fact is: The BugSprint has been anticipated well before the inflamatory membership reform proposal, and repeatedly advertised by Joss in various ways (kudos). Still the participation has been way too low. That has nothing to do with other "distractions".

25 October 2008

Josselin Mouette: 5 days to win cookies

The Debian bug sprint has started! 27 players registered in the hope to win cookies. The list of players and their assignments can be viewed at the usual place. Good luck to everyone. NB: for those who wonder, the assigned bugs were chosen as the 27 oldest RC bugs without a patch and randomized with shuf.

24 October 2008

Josselin Mouette: Bug Sprint

Only one day left for registration ! Register now and win cookies !

21 October 2008

Julien Danjou: Debian bug sprint

I want cookies.