Arturo Borrero Gonz lez: Toolforge GridEngine Debian 10 Buster migration
This post was originally published in the Wikimedia Tech blog, authored by Arturo Borrero Gonzalez.
In accordance with our operating system upgrade policy, we should migrate our servers to Debian Buster.
As discussed in the previous post, one of the most important and successful services provided by the Wikimedia Cloud Services team at
the Wikimedia Foundation is Toolforge. Toolforge is a platform that allows users and developers to run and use a variety of applications
with the ultimate goal of helping the Wikimedia mission from the technical side.
As you may know already, all Wikimedia Foundation servers are powered by Debian, and this includes Toolforge and Cloud VPS. The Debian Project mostly
follows a two year cadence for releases, and Toolforge has been using Debian Stretch for some years now, which nowadays is considered old-old-stable .
In accordance with our operating system upgrade policy, we should migrate our servers to Debian Buster.
Toolforge s two different backend engines, Kubernetes and Grid Engine, are impacted by this upgrade policy. Grid Engine is notably tied to the underlying
Debian release, and the execution environment offered to tools running in the grid is limited to what the Debian archive contains for a given release.
This is unlike in Kubernetes, where tool developers can leverage container images and decouple the runtime environment selection from the base operating
system.
Since the Toolforge grid original conception, we have been doing the same operation over and over again:
So, you are upgrading the Debian release
- Prepare a parallel grid deployment with the new operating system.
- Ask our users (tool developers) to evaluate a newer version of their runtime and programming languages.
- Introduce a migration window and coordinate a quick migration.
- Finally, drop the old operating system from grid servers.