Search Results: "Thomas Lange"

6 May 2024

Thomas Lange: Removing tens of thousands of web pages

In January I've removed tens of thousands of web pages on www.debian.org. Have you noticed it? In the past From 1997 onwards, we had web pages for security announcements. We had to manually prepare a .data and a .wml file which then generated a web page for each security announcement (DSA or DLA). We have listed the 6 most recent messages in a short list that was created from these files. Most of the work that went into the Debian web pages was creating these files. Our search engine often listed the pages with security announcements instead of a more relevant web page for a particular topic. Preparation At DebConf Kosovo (2022) I started with a proof of concept and wrote a script, that generates this list without using the .data/.wml files in the Git repository, but instead reading the primary sources of security information[1]. This new list now includes links to the security tracker and the email of the announcement. Following web pages and scripts were also using these .data and .wml files: Before I could remove all the security web pages, I had to adjust the scripts, that create the above information. When I looked at the OVAL files and the apache logs of our web server, I saw that more than 99% of the web traffic was generated by these XML files (134TB of 135TB total in two weeks). They were not compressed and were around 50MB in size. With the help of Carsten Sch nert we managed to modify the python scripts that generate this OVAL file without using the .data/.wml files and now we only provide bzip2 compressed XML files[2]. The RSS feeds are created by the new Perl script which reads the DSA/DLA list the security tracker and determines the URL of the email of all entries. This script also generates the list of the most recent DSA/DLA entries. Currently we show the last 350 entries which covers more than the last year and includes links to the announcement email and the security tracker. The huge list of crossreferences is not needed any more, since the mapping of CVE to DSA is already included in the DSA list[3] of the security tracker. The amount of translations of the DSA/DLA was very different. French translations were almost all done, but all other languages did translations for a couple of months or years only. E.g. in 2022, Italian had 2 translations, Russian 15, Danish 212, French and English each 279. But from 2023 on only French translations were made. By generating the list of DSA/DLA we lost the ability to translate these web pages, but since these announcements are made of simple, identical sentences it is easy to use an automatic translation service if needed. Now the translation statistics of all web pages are more accurate. Instead of 12200 pages that need to be translated (including all these old DSA/DLA) there are now only 2500 pages to translate[4]. Languages that had a lot of old translations of DSA/DLA lost some percentage but languages that are doing translations of newer web pages won in the statistics of how many pages are translated. Examples: Before
German (de)   3501  28.5%
Italian (it)  1005   8.2%
Danish (da)   6336  51.7%
After
German (de)   1486  59.0%
Italian (it)   909  36.1%
Danish (da)    982  39.0%
Cleanup of all the security web pages Finally in January, I could remove all web pages of the security announcements in one git commit[5]. Using several git rm -rf commands this commit removed 54335 files, including around 9650 DSA/DLA data files, 44189 wml files, nearly 500 Makefiles. Outcome No more manual work is needed for the security team and we now have direct links from a DSA-NNN/DLA-NNN to the email in our mailing list archive. This was not possible before. The search results became more accurate. But we still host a lot of other old content on the Debian web pages which may be removed in the future. [1] https://www.debian.org/security/#infos [2] https://www.debian.org/security/oval/ [3] https://salsa.debian.org/security-tracker-team/security-tracker/-/raw/master/data/DSA/list [4] https://www.debian.org/devel/website/stats [5] https://salsa.debian.org/webmaster-team/webwml/-/commit/2aa73ff15bfc4eb2afd85c

24 January 2024

Thomas Lange: FAI 6.2 released

After more than one a year, a new minor FAI version is available, but it includes some interesting new features. Here a the items from the NEWS file: fai (6.2) unstable; urgency=low In the past the command fai-cd was only used for creating installation ISOs, that could be used from CD or USB stick. Now it possible to create a live ISO. Therefore you create your live chroot environment using 'fai dirinstall' and then convert it to a bootable live ISO using fai-cd. See man fai-cd(8) for an example. Years ago I had the idea to use the remaining disk space on an USB stick after copying an ISO onto it. I've blogged about this recently: https://blog.fai-project.org/posts/extending-iso-images/ The new FAI version includes the tool mk-data-partition for adding a data partition to the ISO itself or to an USB stick. FAI detects this data partition, mounts it to /media/data and can then use various configurations from it. You may want to copy your own set of .deb packages or your whole FAI config space to this partition. FAI now automatically searches this partition for usable FAI configuration data and packages. FAI will install all packages from pkgs/<CLASSNAME> if the equivalent class is defined. Setting FAI_CONFIG_SRC=detect:// now looks into the data partition for the subdirectory 'config' and uses this as the config space. So it's now possible to modify an existing ISO (that is read-only) and make changes to the config space. If there's no config directory in the data partition FAI uses the default location on the ISO. The tool fai-kvm, which starts virtual machines can now boot an ISO not only as CD but also as USB stick. Sometimes users want to adjust the list of disks before the partitioning is startet. Therefore FAI provides several new functions including You can select individual disks by their model name or even the serial number. Two new FAI flags were added (tmux and screen) that make it easy to run FAI inside a tmux or screen session. And finally FAI uses systemd. Yeah! This technical change was waiting since 2015 in a merge request from Moritz 'Morty' Str be, that would enable using systemd during the installation. Before FAI still was using old-style SYSV init scripts and did not started systemd. I didn't tried to apply the patch, because I was afraid that it would need much time to make it work. But then in may 2023 Juri Grabowski just gave it a try at MiniDebConf Hamburg, and voil it just works! Many, many thanks to Moritz and Juri for their bravery. The whole changelog can be found at https://tracker.debian.org/media/packages/f/fai/changelog-6.2 New ISOs for FAI are also available including an example of a Xfce desktop live ISO: https://fai-project.org/fai-cd/ The FAIme service for creating customized installation ISOs will get its update later. The new packages are available for bookworm by adding this line to your sources.list: deb https://fai-project.org/download bookworm koeln

16 December 2023

Thomas Lange: Adding a writeable data partition to an ISO image

Some years ago a customer needed a live ISO containing a customized FAI environment (not for installing but for extended hardware stress tests), but on an USB stick with the possibility to store the logs of the tests on the USB stick. But an ISO file system (iso9660) remains read-only, even when put onto an USB stick. I had the idea to add another partition onto the USB stick after the ISO was written to it (using cp or dd). You can use fdisk with an ISO file, add a new partition, loop mount the ISO and format this partition. That's all. This worked perfect for my customer. I forgot this idea for a while but a few weeks ago I remembered it. What could be possible when my FAI (Fully Automatic Installation) image would also provide such a partition? Which things could be provided on this partition? Could I provide a FAI ISO and my users would be able to easily put their own .deb package onto it without remastering the ISO or building an ISO on their own? Now here's the shell script, that extends an ISO or an USB stick with an ext4 or exFAT partition and set the file system label to MY-DATA. https://github.com/faiproject/fai/blob/master/bin/mk-data-partition Examples how to use mk-data-partition
Add a data partition of size 1G to the Debian installer ISO using an ext4 partition
# mk-data-partition -s 1G debian-12.2.0-amd64-netinst.iso
Create the data partition using an exFAT file system on USB named /dev/sdb.
First copy (or dd) the ISO onto the USB stick. Then add the data partition
to the USB stick.
# cp faicd64-large_6.0.3.iso /dev/sdb
# mk-data-partition -F /dev/sdb
Create the data partition and copy directories A and B to it
# mk-data-partition -c debian-12.2.0-amd64-netinst.iso A B
The next FAI version will use this in different parts of an installation. A blog post about this will follow. A new idea for our Debian installer ISO Here are my ideas how the Debian installer could use such a partition if it automatically detects and mounts it (by it's file system label): The advantage of this approach is that there's no need for the user to remaster the official Debian installer ISO, which is not easy for end users. We only have to extend the installer to use files from this data partition in some portions of the installation. Additional udebs, packages or firmware could automatically be used by the installer. Companies could easily create an OEM installer of Debian. What do you think about this idea? Please send feedback to lange@debian.org

5 December 2023

Thomas Lange: FAI.me service now supports installing recommended packages

The FAI.me service for creating customized installation and cloud images has a new feature by a user requested it. You can now enable installing recommended packages for your custom package list. By default FAIme does only install the dependencies needed, but not the recommended packages. This was a very easy enhancement, only a few lines in the web interface and nearly no changes in the backend were needed. The web interface of the FAI.me service is available at https://fai-project.org/FAIme

8 September 2023

Thomas Lange: FAI.me service now support backports for Debian 12 (bookworm)

The FAI.me service for creating customized installation and cloud images now supports the backports kernel for the stable release Debian 12 (aka bookworm). If you enable the backports option in the web interface, you currently get kernel 6.4. This will help you if you have newer hardware that is not support by the default kernel 6.1. The backports option is also still available for the older distributions. The web interface of the FAI.me service is available at https://fai-project.org/FAIme

15 June 2023

Thomas Lange: 20.000 customized images created by the FAI.me build service

The counter of the FAI.me build service has reached 20.000. This counter was added shortly after the service was started in November 2017. Since then, this service has built more than 21.000 installation images and more than 1300 cloud disk images. In the last few month we had averaged 100 requests per week. Some statistics which settings are popular: I still have some more ideas for the future: Build your own custom Live ISO Thanks for all your feedback I got to improve this service. The build service is available on the FAI project website at https://fai-project.org/FAIme

11 June 2023

Thomas Lange: New FAI ISO images for bookworm available and FAI Live ISO

After Debian 12 aka bookworm was released yesterday, I've also created new FAI ISO images using Debian 12. The defaut ISO (large) uses FAI 6.0.3, kernel 6.1 and can install the XFCE and GNOME desktop without internet connection, since all needed packages are included into the ISO. Additional you can install Ubuntu 22.04 or Rocky Linux 9 with this FAI ISO. During these installations, the packages will be downloade via network. There's also the variant FAI ISO UBUNTU, which includes all Ubuntu packages needed for a Ubuntu server or Ubuntu desktop installation. If you need a small image, you can take the FAI ISO small, which only includes the packages for a XFCE desktop without LibreOffice. This ISO is only 880MB in size. Currently I'm working on a new feature, so FAI can create Live images, that are bootable. It's like the tool live-build which Debian uses for their official Debian Live images. A first verison of the ISO using the XFCE desktop can be downloaded from https://fai-project.org/fai-cd There you also find all other FAI ISOs.

27 April 2023

Thomas Lange: New feature for FAI.me build service

After the initial installation of a new machine, you often want to login as root via ssh. Therefore it's convenient to provide a ssh public key for a passwordless login. This can now be done by just adding your user name from salsa.debian.org, gitlab.com or github.com. You can also give a customized URL from where to download the keys. Before it was only possible to use a github account name. The FAI.me build service then creates a customized installation ISO for you, which will automatically install the ssh public key into the root account. Also the ready-to-boot cloud images support this feature. The build service is available on the FAI project website at https://fai-project.org/FAIme

12 April 2023

Thomas Lange: FAI creates your own Ubuntu installation ISO

A new service is available on the FAI project website https://fai-project.org/FAIme Build your own customized installation ISO for Ubuntu! You can select if you want to install an Ubuntu LTS 22.04 server or desktop and enable Ubuntu LTS packages which are also called Hardware Enablement (HWE). Different disk partition schemes including LVM are available and you can select a language and keyboard layout and add your own list of packages which are then automatically installed from the customized installation media. It's possible to upload your public ssh key or specify your github account name for easy access to the root account. Advanced users may upload a postinst script, which can be executed during the first boot of the computer. It's also possible to add options for grub.

8 March 2023

Thomas Lange: Launch of new FAI project website

After more than 13 years, I've launched a new design for the FAI project web site https://fai-project.org It now uses Materialize CSS and will work much better on mobile devices. Thanks to Thorsten B lo who did the first part of converting the web pages to the new design. I hope you all enjoy the new layout. FAI

17 January 2023

Thomas Lange: FAI 6.0 released and new ISO images using Debian 12 bookworm/testing

After more than a year, a new major FAI release is ready to download. Following new features are included: We have included a lot of bug fixes for free of course. Even if FAI 6.0 will only be included into Debian bookworm, you can install it on a bullseye FAI server and create a nfsroot using bookworm without any problems. The combination of a bullseye FAI server with FAI 6.0 and a bullseye nfsroot should also work. New ISO images are available at https://fai-project.org/fai-cd/ The FAI.me build service is not yet using FAI 6.0, but support will be added in the future. FAI

5 January 2022

Thomas Lange: FAI.me service now support backports for Debian 11 (bullseye)

The FAI.me service for creating customized installation and cloud images now supports a backports kernel for the stable release Debian 11 (aka bullseye). If you enable the backports option, you will currently get kernel 5.14. This will help you if you have newer hardware that is not support by the default kernel 5.10. The backports option is also still available for the images when using the old Debian 10 (buster) release. The URL of the FAI.me service is https://fai-project.org/FAIme/ FAI.me

21 December 2021

Thomas Lange: New ISO images using Debian 11

I've created new ISO images for FAI. Now, they install Debian 11 using kernel 5.10. The ISOs are available from https://fai-project.org/fai-cd There's also a Ubuntu version of the FAI CD which installs Ubuntu LTS 20.04 in two variants. A desktop and a server installation is available. If you want to install the Debian oldstable version (Debian 10), please use the FAI.me service. FAI

25 May 2021

Thomas Lange: Adding a custom postinst script to the FAI image

The FAIme service now supports uploading a custom shell script. This script is added to the installation ISO and gets executed during the first boot on request. You can upload a plain shell script or a compressed version using gzip. Using the postinst script you can adjust the new installed system to your local needs after FAI has done the initial installation. The FAIme service is a web service, that creates customized installation images by just a few clicks. It's also possible to create custom cloud images via the web interface.

23 March 2021

Thomas Lange: More than 10.000 customized ISO image created by FAIme

The FAIme service was started in November 2017. After 3,5 years it created more than 10.000 customized installation and cloud images. And we still have enough CPU power and disk space for more users. FAIme

26 January 2021

Thomas Lange: Making Debian available

This is the subject of an interesting thread on the debian-devel mailing list. It started with ".. The current policy of hiding other versions of Debian is limiting the adoption of your OS by people like me.." It seems that this user managed to contact us developers and give us some important information how we can improve the user experience. The following discussion shows that all our users need non-free firmware to get their wireless network cards run. Do we provide such installation images for our users? Sure. We build them regularly, host them on our servers, we also sign the hash sum with our official signing key. But we hide them very well and still call them unofficial. Why? I would like to have a more positive name for those images. Ubuntu has the HWE (Hardware Enablement) kernel. Maybe Debian firmware enablement images? We should better promote the images that fits best for our users. BTW, the URL for all these useful images is https://cdimage.debian.org/cdimage/unofficial/non-free/images-including-firmware/ Since I'm not using the Debian installer or live image often, I thought my own installation tool would already do better. In FAI , I install the package firmware-linux-nonfree if I need some nonfree firmware. But it appears that this package does not depend on any WiFi firmware package. Oops. So, I've filed a bug report #980758 and propose to add another meta package that depends on a list of firmware packages for WiFi cards. I've now added a workaround to the FAIme service. You can now generate fully automated customized installation images including nonfree firmware for the stable and testing release. The stable release images can also use a newer kernel and firmware from backports. All other package are still from stable. Another useful image variant in my opinion. Debian FAIme

1 August 2020

Holger Levsen: 20200801-debconf3

DebConf3 This tshirt is 17 years old and from DebConf3. I should probably wash it at 60 celcius for once... DebConf3 was my first DebConf and took place in Oslo, Norway, in 2003. I was very happy to be invited, like any Debian contributor at that time, and that Debian would provide food and accomodation for everyone. Accomodation was sleeping on the floor in some classrooms of an empty school and I remember having tasted grasshoppers provided by a friendly Gunnar Wolf there, standing in line on the first day with the SSH maintainer (OMG!1 (update: I originally wrote here that it wasn't Colin back then, but Colin mailed me to say that he was indeed maintaining SSH even back then, so I've met a previous maintainer there)) and meeting the one Debian person I had actually worked with before: Thomas Lange or MrFAI (update: Thomas also mailed me and said this was at DebConf5). In Oslo I also was exposed to Skolelinux / Debian Edu for the first time, saw a certain presentation from the FTP masters and also noticed some people recording the talks, though as I learned later these videos were never released to the public. And there was this fiveteen year old called Toresbe, who powered on the PDP's which were double his age. And then actually made use of them. And and and. I'm very happy I went to this DebConf. Without going my Debian journey would have been very different today. Thanks to everyone who made this such a welcoming event. Thanks to anyone who makes any event welcoming! :)

28 April 2020

Thomas Lange: FAI 5.9.4, Ubuntu 20.04 support, FAIme service

The new FAI version 5.9.4 adds support for the new Ubuntu LTS version 20.04 (focal). New FAI installation images (including a Ubuntu only ISO) are available from https://fai-project.org/fai-cd I've added support for the Debian testing release (currently bullseye) to the FAIme service for building customized installation and cloud images. This service still supports the oldstable and stable release including variants with a backports kernel. The FAI.me build service is also using the newest FAI version and the customized ISO images can be booted in an UEFI or legacy BIOS environment. https://fai-project.org/FAIme

8 October 2017

Thomas Lange: FAI 5.4 enters the embedded world

Since DebConf 17 I was working on cross-architecture support for FAI. The new FAI release supports creating cross-architecture disk images, for e.g. you can build an image for Arm64 (aarch64) on a host running 64-bit x86 Linux (amd64) in about 6 minutes. The release announcement has more details, and I also created a video showing the build process for an Arm64 disk image and booting this image using Qemu. I'm happy to join the Debian cloud sprint in a week, where more FAI related work is waiting. FAI embedded ARM

7 September 2017

Thomas Lange: My recent FAI activities

During DebConf 17 in Montr al I had a FAI demo session (video), where I showed how to create a customized installation CD and how to create a diskimage using the same configuration. This diskimage is ready for use with a VM software or can be booted inside a cloud environment. During the last weeks I was working on FAI 5.4 which will be released in a few weeks. I you want to test it use
deb https://fai-project.org/download beta-testing koeln
in your sources.list file. The most important new feature will be the cross architecture support. I managed to create an ARM64 diskimage on a x86 host and boot this inside Qemu. Currently I learn how to flash images onto my new Hikey960 board for booting my own Debian images on real hardware. The embedded world is still new for me and very different in respect to the boot process. At DebConf, I also worked on debootstrap. I produced a set of patches which can speedup debootstrap by a factor of 2. See #871835 for details. FAI debootstrap ARM

Next.