Search Results: "will"

12 April 2021

Russell Coker: Storage Trends 2021

The Viability of Small Disks Less than a year ago I wrote a blog post about storage trends [1]. My main point in that post was that disks smaller than 2TB weren t viable then and 2TB disks wouldn t be economically viable in the near future. Now MSY has 2TB disks for $72 and 2TB SSD for $245, saving $173 if you get a hard drive (compared to saving $240 10 months ago). Given the difference in performance and noise 2TB hard drives won t be worth using for most applications nowadays. NVMe vs SSD Last year NVMe prices were very comparable for SSD prices, I was hoping that trend would continue and SSDs would go away. Now for sizes 1TB and smaller NVMe and SSD prices are very similar, but for 2TB the NVMe prices are twice that of SSD presumably partly due to poor demand for 2TB NVMe. There are also no NVMe devices larger than 2TB on sale at MSY (a store which caters to home stuff not special server equipment) but SSDs go up to 8TB. It seems that NVMe is only really suitable for workstation storage and for cache etc on a server. So SATA SSDs will be around for a while. Small Servers There are a range of low end servers which support a limited number of disks. Dell has 2 disk servers and 4 disk servers. If one of those had 8TB SSDs you could have 8TB of RAID-1 or 24TB of RAID-Z storage in a low end server. That covers the vast majority of servers (small business or workgroup servers tend to have less than 8TB of storage). Larger Servers Anandtech has an article on Seagates roadmap to 120TB disks [2]. They currently sell 20TB disks using HAMR technology Currently the biggest disks that MSY sells are 10TB for $395, which was also the biggest disk they were selling last year. Last year MSY only sold SSDs up to 2TB in size (larger ones were available from other companies at much higher prices), now they sell 8TB SSDs for $949 (4* capacity increase in less than a year). Seagate is planning 30TB disks for 2023, if SSDs continue to increase in capacity by 4* per year we could have 128TB SSDs in 2023. If you needed a server with 100TB of storage then having 2 or 3 SSDs in a RAID array would be much easier to manage and faster than 4*30TB disks in an array. When you have a server with many disks you can expect to have more disk failures due to vibration. One time I built a server with 18 disks and took disks from 2 smaller servers that had 4 and 5 disks. The 9 disks which had been working reliably for years started having problems within weeks of running in the bigger server. This is one of the many reasons for paying extra for SSD storage. Seagate is apparently planning 50TB disks for 2026 and 100TB disks for 2030. If that s the best they can do then SSD vendors should be able to sell larger products sooner at prices that are competitive. Matching hard drive prices is not required, getting to less than 4* the price should be enough for most customers. The Anandtech article is worth reading, it mentions some interesting features that Seagate are developing such as having 2 actuators (which they call Mach.2) so the drive can access 2 different tracks at the same time. That can double the performance of a disk, but that doesn t change things much when SSDs are more than 100* faster. Presumably the Mach.2 disks will be SAS and incredibly expensive while providing significantly less performance than affordable SATA SSDs. Computer Cases In my last post I speculated on the appearance of smaller cases designed to not have DVD drives or 3.5 hard drives. Such cases still haven t appeared apart from special purpose machines like the NUC that were available last year. It would be nice if we could get a new industry standard for smaller power supplies. Currently power supplies are expected to be almost 5 inches wide (due to the expectation of a 5.25 DVD drive mounted horizontally). We need some industry standards for smaller PCs that aren t like the NUC, the NUC is very nice, but most people who build their own PC need more space than that. I still think that planning on USB DVD drives is the right way to go. I ve got 4PCs in my home that are regularly used and CDs and DVDs are used so rarely that sharing a single DVD drive among all 4 wouldn t be a problem. Conclusion I m tempted to get a couple of 4TB SSDs for my home server which cost $487 each, it currently has 2*500G SSDs and 3*4TB disks. I would have to remove some unused files but that s probably not too hard to do as I have lots of old backups etc on there. Another possibility is to use 2*4TB SSDs for most stuff and 2*4TB disks for backups. I m recommending that all my clients only use SSDs for their storage. I only have one client with enough storage that disks are the only option (100TB of storage) but they moved all the functions of that server to AWS and use S3 for the storage. Now I don t have any clients doing anything with storage that can t be done in a better way on SSD for a price difference that s easy for them to afford. Affordable SSD also makes RAID-1 in workstations more viable. 2 disks in a PC is noisy if you have an office full of them and produces enough waste heat to be a reliability issue (most people don t cool their offices adequately on weekends). 2 SSDs in a PC is no problem at all. As 500G SSDs are available for $73 it s not a significant cost to install 2 of them in every PC in the office (more cost for my time than hardware). I generally won t recommend that hard drives be replaced with SSDs in systems that are working well. But if a machine runs out of space then replacing it with SSDs in a RAID-1 is a good choice. Moore s law might cover SSDs, but it definitely doesn t cover hard drives. Hard drives have fallen way behind developments of most other parts of computers over the last 30 years, hopefully they will go away soon.
Related posts:
  1. Storage Trends In considering storage trends for the consumer side I m looking...
  2. New Storage Developments Eweek has an article on a new 1TB Seagate drive....
  3. Finding Storage Performance Problems Here are some basic things to do when debugging storage...

11 April 2021

Jelmer Vernooij: The upstream ontologist

The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid packages maintained in Git. This post is part of a series about the progress of the Janitor. The upstream ontologist is a project that extracts metadata about upstream projects in a consistent format. It does this with a combination of heuristics and reading ecosystem-specific metadata files, such as Python s setup.py, rust s Cargo.toml as well as e.g. scanning README files.

Supported Data Sources It will extract information from a wide variety of sources, including:
Supported Fields Fields that it currently provides include:
  • Homepage: homepage URL
  • Name: name of the upstream project
  • Contact: contact address of some sort of the upstream (e-mail, mailing list URL)
  • Repository: VCS URL
  • Repository-Browse: Web URL for viewing the VCS
  • Bug-Database: Bug database URL (for web viewing, generally)
  • Bug-Submit: URL to use to submit new bugs (either on the web or an e-mail address)
  • Screenshots: List of URLs with screenshots
  • Archive: Archive used - e.g. SourceForge
  • Security-Contact: e-mail or URL with instructions for reporting security issues
  • Documentation: Link to documentation on the web:
  • Wiki: Wiki URL
  • Summary: one-line description of the project
  • Description: longer description of the project
  • License: Single line license description (e.g. GPL 2.0 ) as declared in the metadata[1]
  • Copyright: List of copyright holders
  • Version: Current upstream version
  • Security-MD: URL to markdown file with security policy
All data fields have a certainty associated with them ( certain , confident , likely or possible ), which gets set depending on how the data was derived or where it was found. If multiple possible values were found for a specific field, then the value with the highest certainty is taken.
Interface The ontologist provides a high-level Python API as well as two command-line tools that can write output in two different formats: For example, running guess-upstream-metadata on dulwich: 
 % guess-upstream-metadata
 <string>:2: (INFO/1) Duplicate implicit target name: "contributing".
 Name: dulwich
 Repository: https://www.dulwich.io/code/
 X-Security-MD: https://github.com/dulwich/dulwich/tree/HEAD/SECURITY.md
 X-Version: 0.20.21
 Bug-Database: https://github.com/dulwich/dulwich/issues
 X-Summary: Python Git Library
 X-Description:  
   This is the Dulwich project.
   It aims to provide an interface to git repos (both local and remote) that
   doesn't call out to git directly but instead uses pure Python.
 X-License: Apache License, version 2 or GNU General Public License, version 2 or later.
 Bug-Submit: https://github.com/dulwich/dulwich/issues/new
Lintian-Brush lintian-brush can update DEP-12-style debian/upstream/metadata files that hold information about the upstream project that is packaged as well as the Homepage in the debian/control file based on information provided by the upstream ontologist. By default, it only imports data with the highest certainty - you can override this by specifying the uncertain command-line flag.
[1]Obviously this won t be able to describe the full licensing situation for many projects. Projects like scancode-toolkit are more appropriate for that.

Vishal Gupta: Sikkim 101 for Backpackers

Host to Kanchenjunga, the world s third-highest mountain peak and the endangered Red Panda, Sikkim is a state in northeastern India. Nestled between Nepal, Tibet (China), Bhutan and West Bengal (India), the state offers a smorgasbord of cultures and cuisines. That said, it s hardly surprising that the old spice route meanders through western Sikkim, connecting Lhasa with the ports of Bengal. Although the latter could also be attributed to cardamom (kali elaichi), a perennial herb native to Sikkim, which the state is the second-largest producer of, globally. Lastly, having been to and lived in India, all my life, I can confidently say Sikkim is one of the cleanest & safest regions in India, making it ideal for first-time backpackers.

Brief History
  • 17th century: The Kingdom of Sikkim is founded by the Namgyal dynasty and ruled by Buddhist priest-kings known as the Chogyal.
  • 1890: Sikkim becomes a princely state of British India.
  • 1947: Sikkim continues its protectorate status with the Union of India, post-Indian-independence.
  • 1973: Anti-royalist riots take place in front of the Chogyal's palace, by Nepalis seeking greater representation.
  • 1975: Referendum leads to the deposition of the monarchy and Sikkim joins India as its 22nd state.
Languages
  • Official: English, Nepali, Sikkimese/Bhotia and Lepcha
  • Though Hindi and Nepali share the same script (Devanagari), they are not mutually intelligible. Yet, most people in Sikkim can understand and speak Hindi.
Ethnicity
  • Nepalis: Migrated in large numbers (from Nepal) and soon became the dominant community
  • Bhutias: People of Tibetan origin. Major inhabitants in Northern Sikkim.
  • Lepchas: Original inhabitants of Sikkim

Food
  • Tibetan/Nepali dishes (mostly consumed during winter)
    • Thukpa: Noodle soup, rich in spices and vegetables. Usually contains some form of meat. Common variations: Thenthuk and Gyathuk
    • Momos: Steamed or fried dumplings, usually with a meat filling.
    • Saadheko: Spicy marinated chicken salad.
    • Gundruk Soup: A soup made from Gundruk, a fermented leafy green vegetable.
    • Sinki : A fermented radish tap-root product, traditionally consumed as a base for soup and as a pickle. Eerily similar to Kimchi.
  • While pork and beef are pretty common, finding vegetarian dishes is equally easy.
  • Staple: Dal-Bhat with Subzi. Rice is a lot more common than wheat (rice) possibly due to greater carb content and proximity to West Bengal, India s largest producer of Rice.
  • Good places to eat in Gangtok
    • Hamro Bhansa Ghar, Nimtho (Nepali)
    • Taste of Tibet
    • Dragon Wok (Chinese & Japanese)

Buddhism in Sikkim
  • Bayul Demojong (Sikkim), is the most sacred Land in the Himalayas as per the belief of the Northern Buddhists and various religious texts.
  • Sikkim was blessed by Guru Padmasambhava, the great Buddhist saint who visited Sikkim in the 8th century and consecrated the land.
  • However, Buddhism is said to have reached Sikkim only in the 17th century with the arrival of three Tibetan monks viz. Rigdzin Goedki Demthruchen, Mon Kathok Sonam Gyaltshen & Rigdzin Legden Je at Yuksom. Together, they established a Buddhist monastery.
  • In 1642 they crowned Phuntsog Namgyal as the first monarch of Sikkim and gave him the title of Chogyal, or Dharma Raja.
  • The faith became popular through its royal patronage and soon many villages had their own monastery.
  • Today Sikkim has over 200 monasteries.

Major monasteries
  • Rumtek Monastery, 20Km from Gangtok
  • Lingdum/Ranka Monastery, 17Km from Gangtok
  • Phodong Monastery, 28Km from Gangtok
  • Ralang Monastery, 10Km from Ravangla
  • Tsuklakhang Monastery, Royal Palace, Gangtok
  • Enchey Monastery, Gangtok
  • Tashiding Monastery, 35Km from Ravangla

Reaching Sikkim
  • Gangtok, being the capital, is easiest to reach amongst other regions, by public transport and shared cabs.
  • By Air:
    • Pakyong (PYG) :
      • Nearest airport from Gangtok (about 1 hour away)
      • Tabletop airport
      • Reserved cabs cost around INR 1200.
      • As of Apr 2021, the only flights to PYG are from IGI (Delhi) and CCU (Kolkata).
    • Bagdogra (IXB) :
      • About 20 minutes from Siliguri and 4 hours from Gangtok.
      • Larger airport with flights to most major Indian cities.
      • Reserved cabs cost about INR 3000. Shared cabs cost about INR 350.
  • By Train:
    • New Jalpaiguri (NJP) :
      • About 20 minutes from Siliguri and 4 hours from Gangtok.
      • Reserved cabs cost about INR 3000. Shared cabs from INR 350.
  • By Road:
    • NH10 connects Siliguri to Gangtok
    • If you can t find buses plying to Gangtok directly, reach Siliguri and then take a cab to Gangtok.
  • Sikkim Nationalised Transport Div. also runs hourly buses between Siliguri and Gangtok and daily buses on other common routes. They re cheaper than shared cabs.
  • Wizzride also operates shared cabs between Siliguri/Bagdogra/NJP, Gangtok and Darjeeling. They cost about the same as shared cabs but pack in half as many people in luxury cars (Innova, Xylo, etc.) and are hence more comfortable.

Gangtok
  • Time needed: 1D/1N
  • Places to visit:
    • Hanuman Tok
    • Ganesh Tok
    • Tashi View Point [6,800ft]
    • MG Marg
    • Sikkim Zoo
    • Gangtok Ropeway
    • Enchey Monastery
    • Tsuklakhang Palace & Monastery
  • Hostels: Tagalong Backpackers (would strongly recommend), Zostel Gangtok
  • Places to chill: Travel Cafe, Caf Live & Loud and Gangtok Groove
  • Places to shop: Lal Market and MG Marg

Getting Around
  • Taxis operate on a reserved or shared basis. In case of the latter, you can pool with other commuters your taxis will pick up and drop en-route.
  • Naturally shared taxis only operate on popular routes. The easiest way to get around Gangtok is to catch a shared cab from MG Marg.
  • Reserved taxis for Gangtok sightseeing cost around INR 1000-1500, depending upon the spots you d like to see
  • Key taxi/bus stands :
    • Deorali stand: For Darjeeling, Siliguri, Kalimpong
    • Vajra stand: For North & East Sikkim (Tsomgo Lake & Nathula)
    • Rumtek taxi: For Ravangla, Pelling, Namchi, Geyzing, Jorethang and Singtam.
Exploring Gangtok on an MTB

North Sikkim
  • The easiest & most economical way to explore North Sikkim is the 3D/2N package offered by shared-cab drivers.
  • This includes food, permits, cab rides and accommodation (1N in Lachen and 1N in Lachung)
  • The accommodation on both nights are at homestays with bare necessities, so keep your hopes low.
  • In the spirit of sustainable tourism, you ll be asked to discard single-use plastic bottles, so please carry a bottle that you can refill along the way.
  • Zero Point and Gurdongmer Lake are snow-capped throughout the year
3D/2N Shared-cab Package Itinerary
  • Day 1
    • Gangtok (10am) - Chungthang - Lachung (stay)
  • Day 2
    • Pre-lunch : Lachung (6am) - Yumthang Valley [12,139ft] - Zero Point - Lachung [15,300ft]
    • Post-lunch : Lachung - Chungthang - Lachen (stay)
  • Day 3
    • Pre-lunch : Lachen (5am) - Kala Patthar - Gurdongmer Lake [16,910ft] - Lachen
    • Post-lunch : Lachen - Chungthang - Gangtok (7pm)
  • This itinerary is idealistic and depends on the level of snowfall.
  • Some drivers might switch up Day 2 and 3 itineraries by visiting Lachen and then Lachung, depending upon the weather.
  • Areas beyond Lachen & Lachung are heavily militarized since the Indo-China border is only a few miles away.

East Sikkim

Zuluk and Silk Route
  • Time needed: 2D/1N
  • Zuluk [9,400ft] is a small hamlet with an excellent view of the eastern Himalayan range including the Kanchenjunga.
  • Was once a transit point to the historic Silk Route from Tibet (Lhasa) to India (West Bengal).
  • The drive from Gangtok to Zuluk takes at least four hours. Hence, it makes sense to spend the night at a homestay and space out your trip to Zuluk

Tsomgo Lake and Nathula
  • Time Needed : 1D
  • A Protected Area Permit is required to visit these places, due to their proximity to the Chinese border
  • Tsomgo/Chhangu Lake [12,313ft]
    • Glacial lake, 40 km from Gangtok.
    • Remains frozen during the winter season.
    • You can also ride on the back of a Yak for INR 300
  • Baba Mandir
    • An old temple dedicated to Baba Harbhajan Singh, a Sepoy in the 23rd Regiment, who died in 1962 near the Nathu La during Indo China war.
  • Nathula Pass [14,450ft]
    • Located on the Indo-Tibetan border crossing of the Old Silk Route, it is one of the three open trading posts between India and China.
    • Plays a key role in the Sino-Indian Trade and also serves as an official Border Personnel Meeting(BPM) Point.
    • May get cordoned off by the Indian Army in event of heavy snowfall or for other security reasons.

West Sikkim
  • Time needed: 3N/1N
  • Hostels at Pelling : Mochilerro Ostillo

Itinerary

Day 1: Gangtok - Ravangla - Pelling
  • Leave Gangtok early, for Ravangla through the Temi Tea Estate route.
  • Spend some time at the tea garden and then visit Buddha Park at Ravangla
  • Head to Pelling from Ravangla

Day 2: Pelling sightseeing
  • Hire a cab and visit Skywalk, Pemayangtse Monastery, Rabdentse Ruins, Kecheopalri Lake, Kanchenjunga Falls.

Day 3: Pelling - Gangtok/Siliguri
  • Wake up early to catch a glimpse of Kanchenjunga at the Pelling Helipad around sunrise
  • Head back to Gangtok on a shared-cab
  • You could take a bus/taxi back to Siliguri if Pelling is your last stop.

Darjeeling
  • In my opinion, Darjeeling is lovely for a two-day detour on your way back to Bagdogra/Siliguri and not any longer (unless you re a Bengali couple on a honeymoon)
  • Once a part of Sikkim, Darjeeling was ceded to the East India Company after a series of wars, with Sikkim briefly receiving a grant from EIC for gifting Darjeeling to the latter
  • Post-independence, Darjeeling was merged with the state of West Bengal.

Itinerary

Day 1 :
  • Take a cab from Gangtok to Darjeeling (shared-cabs cost INR 300 per seat)
  • Reach Darjeeling by noon and check in to your Hostel. I stayed at Hideout.
  • Spend the evening visiting either a monastery (or the Batasia Loop), Nehru Road and Mall Road.
  • Grab dinner at Glenary whilst listening to live music.

Day 2:
  • Wake up early to catch the sunrise and a glimpse of Kanchenjunga at Tiger Hill. Since Tiger Hill is 10km from Darjeeling and requires a permit, book your taxi in advance.
  • Alternatively, if you don t want to get up at 4am or shell out INR1500 on the cab to Tiger Hill, walk to the Kanchenjunga View Point down Mall Road
  • Next, queue up outside Keventers for breakfast with a view in a century-old cafe
  • Get a cab at Gandhi Road and visit a tea garden (Happy Valley is the closest) and the Ropeway. I was lucky to meet 6 other backpackers at my hostel and we ended up pooling the cab at INR 200 per person, with INR 1400 being on the expensive side, but you could bargain.
  • Get lunch, buy some tea at Golden Tips, pack your bags and hop on a shared-cab back to Siliguri. It took us about 4hrs to reach Siliguri, with an hour to spare before my train.
  • If you ve still got time on your hands, then check out the Peace Pagoda and the Darjeeling Himalayan Railway (Toy Train). At INR 1500, I found the latter to be too expensive and skipped it.

Tips and hacks
  • Download offline maps, especially when you re exploring Northern Sikkim.
  • Food and booze are the cheapest in Gangtok. Stash up before heading to other regions.
  • Keep your Aadhar/Passport handy since you need permits to travel to North & East Sikkim.
  • In rural areas and some cafes, you may get to try Rhododendron Wine, made from Rhododendron arboreum a.k.a Gurans. Its production is a little hush-hush since the flower is considered holy and is also the National Flower of Nepal.
  • If you don t want to invest in a new jacket, boots or a pair of gloves, you can always rent them at nominal rates from your hotel or little stores around tourist sites.
  • Check the weather of a region before heading there. Low visibility and precipitation can quite literally dampen your experience.
  • Keep your itinerary flexible to accommodate for rest and impromptu plans.
  • Shops and restaurants close by 8pm in Sikkim and Darjeeling. Plan for the same.

Carry
  • a couple of extra pairs of socks (woollen, if possible)
  • a pair of slippers to wear indoors
  • a reusable water bottle
  • an umbrella
  • a power bank
  • a couple of tablets of Diamox. Helps deal with altitude sickness
  • extra clothes and wet bags since you may not get a chance to wash/dry your clothes
  • a few passport size photographs

Shared-cab hacks
  • Intercity rides can be exhausting. If you can afford it, pay for an additional seat.
  • Call shotgun on the drives beyond Lachen and Lachung. The views are breathtaking.
  • Return cabs tend to be cheaper (WB cabs travelling from SK and vice-versa)

Cost
  • My median daily expenditure (back when I went to Sikkim in early March 2021) was INR 1350.
  • This includes stay (bunk bed), food, wine and transit (shared cabs)
  • In my defence, I splurged on food, wine and extra seats in shared cabs, but if you re on a budget, you could easily get by on INR 1 - 1.2k per day.
  • For a 9-day trip, I ended up shelling out nearly INR 15k, including 2AC trains to & from Kolkata
  • Note : Summer (March to May) and Autumn (October to December) are peak seasons, and thereby more expensive to travel around.

Souvenirs and things you should buy

Buddhist souvenirs :
  • Colourful Prayer Flags (great for tying on bikes or behind car windshields)
  • Miniature Prayer/Mani Wheels
  • Lucky Charms, Pendants and Key Chains
  • Cham Dance masks and robes
  • Singing Bowls
  • Common symbols: Om mani padme hum, Ashtamangala, Zodiac signs

Handicrafts & Handlooms
  • Tibetan Yak Wool shawls, scarfs and carpets
  • Sikkimese Ceramic cups
  • Thangka Paintings

Edibles
  • Darjeeling Tea (usually brewed and not boiled)
  • Wine (Arucha Peach & Rhododendron)
  • Dalle Khursani (Chilli) Paste and Pickle

Header Icon made by Freepik from www.flaticon.com is licensed by CC 3.0 BY

Jonathan Dowland: 2020 in short fiction

Cover for *Episodes*
Following on from 2020 in Fiction: In 2020 I read a couple of collections of short fiction from some of my favourite authors. I started the year with Christopher Priest's Episodes. The stories within are collected from throughout his long career, and vary in style and tone. Priest wrote new little prologues and epilogues for each of the stories, explaining the context in which they were written. I really enjoyed this additional view into their construction.
Cover for *Adam Robots*
By contrast, Adam Robert's Adam Robots presents the stories on their own terms. Each of the stories is written in a different mode: one as golden-age SF, another as a kind of Cyberpunk, for example, although they all blend or confound sub-genres to some degree. I'm not clever enough to have decoded all their secrets on a first read, and I would have appreciated some "Cliff's Notes on any deeper meaning or intent.
Cover for *Exhalation*
Ted Chiang's Exhalation was up to the fantastic standard of his earlier collection and had some extremely thoughtful explorations of philosophical ideas. All the stories are strong but one stuck in my mind the longest: Omphalos) With my daughter I finished three of Terry Pratchett's short story collections aimed at children: Dragon at Crumbling Castle; The Witch's Vacuum Cleaner and The Time-Travelling Caveman. If you are a Pratchett fan and you've overlooked these because they're aimed at children, take another look. The quality varies, but there are some true gems in these. Several stories take place in common settings, either the town of Blackbury, in Gritshire (and the adjacent Even Moor), or the Welsh border-town of Llandanffwnfafegettupagogo. The sad thing was knowing that once I'd finished them (and the fourth, Father Christmas's Fake Beard) that was it: there will be no more.
Cover for Interzone, issue 277
8/31 of the "books" I read in 2020 were issues of Interzone. Counting them as "books" for my annual reading goal has encouraged me to read full issues, whereas before I would likely have only read a couple of stories from each issue. Reading full issues has rekindled the enjoyment I got out of it when I first discovered the magazine at the turn of the Century. I am starting to recognise stories by authors that have written stories in other issues, as well as common themes from the current era weaving their way into the work (Trump, Brexit, etc.) No doubt the Pandemic will leave its mark on 2021's stories.

Junichi Uekawa: Wrote a timezone checker page.

Wrote a timezone checker page. timezone. Shows the current time in blue line. I haven't made anything configurable but will think about it later.

10 April 2021

Charles Plessy: Debian Bullseye: more open

Debian Bullseye will provide the command /usr/bin/open for your greatest comfort at the command line. On a system with a graphical desktop environment, the command should have a similar result as when opening a document from a mouse-and-click file browser. Technically, /usr/bin/open is a symbolic link managed by update-alternatives to point towards xdg-open if available and otherwise run-mailcap.

Kentaro Hayashi: Grow your ideas for Debian Project

There may be some "If it could be ..." ideas for Debian Project. If idea is concreate and worth to make things forward, it should make a proposal for Project Funding. salsa.debian.org But it is a just an idea, or no afford to act as an executor role, that idea will not be achieved. I thought that It needs an incubator - complemental project. salsa.debian.org I've salvaged an idea from closed MR Add proposal about "Formalize reimbursement process" (!5) Merge Requests Freexian SARL / Project Funding GitLab I'm not confident whether mechanism works, but Debian needs change.

8 April 2021

Sean Whitton: consfigurator-live-build

One of my goals for Consfigurator is to make it capable of installing Debian to my laptop, so that I can stop booting to GRML and manually partitioning and debootstrapping a basic system, only to then turn to configuration management to set everything else up. My configuration management should be able to handle the partitioning and debootstrapping, too. The first stage was to make Consfigurator capable of debootstrapping a basic system, chrooting into it, and applying other arbitrary configuration, such as installing packages. That s been in place for some weeks now. It s sophisticated enough to avoid starting up newly installed services, but I still need to add some bind mounting. Another significant piece is teaching Consfigurator how to partition block devices. That s quite tricky to do in a sufficiently general way I want to cleanly support various combinations of LUKS, LVM and regular partitions, including populating /etc/crypttab and /etc/fstab. I have some ideas about how to do it, but it ll probably take a few tries to get the abstractions right. Let s imagine that code is all in place, such that Consfigurator can be pointed at a block device and it will install a bootable Debian system to it. Then to install Debian to my laptop I d just need to take my laptop s disk drive out and plug it into another system, and run Consfigurator on that system, as root, pointed at the block device representing my laptop s disk drive. For virtual machines, it would be easy to write code which loop-mounts an empty disk image, and then Consfigurator could be pointed at the loop-mounted block device, thereby making the disk image file bootable. This is adequate for virtual machines, or small single-board computers with tiny storage devices (not that I actually use any of those, but I want Consfigurator to be able to make disk images for them!). But it s not much good for my laptop. I casually referred to taking out my laptop s disk drive and connecting it to another computer, but this would void my laptop s warranty. And Consfigurator would not be able to update my laptop s NVRAM, as is needed on UEFI systems. What s wanted here is a live system which can run Consfigurator directly on the laptop, pointed at the block device representing its physical disk drive. Ideally this live system comes with a chroot with the root filesystem for the new Debian install already built, so that network access is not required, and all Consfigurator has to do is partition the drive and copy in the contents of the chroot. The live system could be set up to automatically start doing that upon boot, but another option is to just make Consfigurator itself available to be used interactively. The user boots the live system, starts up Emacs, starts up Lisp, and executes a Consfigurator deployment, supplying the block device representing the laptop s disk drive as an argument to the deployment. Consfigurator goes off and partitions that drive, copies in the contents of the chroot, and executes grub-install to make the laptop bootable. This is also much easier to debug than a live system which tries to start partitioning upon boot. It would look something like this: 
    ;; melete.silentflame.com is a Consfigurator host object representing the
    ;; laptop, including information about the partitions it should have
    (deploy-these :local ...
      (chroot:partitioned-and-installed
        melete.silentflame.com "/srv/chroot/melete" "/dev/nvme0n1"))
Now, building live systems is a fair bit more involved than installing Debian to a disk drive and making it bootable, it turns out. While I want Consfigurator to be able to completely replace the Debian Installer, I decided that it is not worth trying to reimplement the relevant parts of the Debian Live tool suite, because I do not need to make arbitrary customisations to any live systems. I just need to have some packages installed and some files in place. Nevertheless, it is worth teaching Consfigurator how to invoke Debian Live, so that the customisation of the chroot which isn t just a matter of passing options to lb_config(1) can be done with Consfigurator. This is what I ve ended up with in Consfigurator s source code: 
(defpropspec image-built :lisp (config dir properties)
  "Build an image under DIR using live-build(7), where the resulting live
system has PROPERTIES, which should contain, at a minimum, a property from
CONSFIGURATOR.PROPERTY.OS setting the Debian suite and architecture.  CONFIG
is a list of arguments to pass to lb_config(1), not including the '-a' and
'-d' options, which Consfigurator will supply based on PROPERTIES.
This property runs the lb_config(1), lb_bootstrap(1), lb_chroot(1) and
lb_binary(1) commands to build or rebuild the image.  Rebuilding occurs only
when changes to CONFIG or PROPERTIES mean that the image is potentially
out-of-date; e.g. if you just add some new items to PROPERTIES then in most
cases only lb_chroot(1) and lb_binary(1) will be re-run.
Note that lb_chroot(1) and lb_binary(1) both run after applying PROPERTIES,
and might undo some of their effects.  For example, to configure
/etc/apt/sources.list, you will need to use CONFIG not PROPERTIES."
  (:desc (declare (ignore config properties))
         #?"Debian Live image built in $ dir ")
  (let* (...)
    ;; ...
     (eseqprops
      ;; ...
      (on-change
          (eseqprops
           (on-change
               (file:has-content ,auto/config ,(auto/config config) :mode #o755)
             (file:does-not-exist ,@clean)
             (%lbconfig ,dir)
             (%lbbootstrap t ,dir))
           (%lbbootstrap nil ,dir)
           (deploys ((:chroot :into ,chroot)) ,host))
        (%lbchroot ,dir)
        (%lbbinary ,dir)))))
Here, %lbconfig is a property running lb_config(1), %lbbootstrap one which runs lb_bootstrap(1), etc. Those properties all just change directory to the right place and run the command, essentially, with a little extra code to handle failed debootstraps and the like. The ON-CHANGE and ESEQPROPS combinators work together to sequence the interaction of the Debian Live suite and Consfigurator. This way, we only rebuild the chroot if the configuration changed, and we only rebuild the image if the chroot changed. Now over in my personal consfig: 
(try-register-data-source
 :git-snapshot :name "consfig" :repo #P"src/cl/consfig/" ...)
(defproplist hybrid-live-iso-built :lisp ()
  "Build a Debian Live system in /srv/live/spw.
Typically this property is not applied in a DEFHOST form, but rather run as
needed at the REPL.  The reason for this is that otherwise the whole image will
get rebuilt each time a commit is made to my dotfiles repo or to my consfig."
  (:desc "Sean's Debian Live system image built")
  (live-build:image-built.
      '("--archive-areas" "main contrib non-free" ...)
      "/srv/live/spw"
    (os:debian-stable "buster" :amd64)
    (basic-props)
    (apt:installed "whatever" "you" "want")
    (git:snapshot-extracted "/etc/skel/src" "dotfiles")
    (file:is-copy-of "/etc/skel/.bashrc" "/etc/skel/src/dotfiles/.bashrc")
    (git:snapshot-extracted "/root/src/cl" "consfig")))
The first argument to LIVE-BUILD:IMAGE-BUILT. is additional arguments to lb_config(1). The third argument onwards are the properties for the live system. The cool thing is GIT:SNAPSHOT-EXTRACTED the calls to this ensure that a copy of my Emacs configuration and my consfig end up in the live image, ready to be used interactively to install Debian, as described above. I ll need to add something like (chroot:host-chroot-bootstrapped melete.silentflame.com "/srv/chroot/melete") too. As with everything Consfigurator-related, Joey Hess s Propellor is the giant upon whose shoulders I m standing.

7 April 2021

Emmanuel Kasper: Manually install a single node Kubernetes cluster on Debian

Debian has work-in-progress packages for Kubernetes, which work well enough enough for a testing and learning environement. Bootstraping a cluster with the kubeadm deployer with these packages is not that hard, and is similar to the upstream kubeadm documentation

Install necessary packages in a VMInstall a throwaway VM with Vagrant.
apt install vagrant vagrant-libvirt
vagrant init debian/testing64
Bump the RAM and CPU of the VM, Kubernetes needs at least 2 gigs and 2 cores.
awk  -i inplace '1;/^Vagrant.configure\("2"\) do \ config/  print "  config.vm.provider :libvirt do  vm   vm.memory=2048 end" ' Vagrantfile
awk  -i inplace '1;/^Vagrant.configure\("2"\) do \ config/  print "  config.vm.provider :libvirt do  vm   vm.cpus=2 end" ' Vagrantfile
Start the VM, login, update the package index.
vagrant up
vagrant ssh
sudo apt update
Install a container engine, here we use docker.io, we could also use containerd (both are packaged in Debian) or cri-o.
sudo apt install --yes --no-install-recommends docker.io curl
Install kubernetes binaries. This will install kubelet, the system service which will manage the containers, and kubectl the user/admin tool to manage the cluster.
sudo apt install --yes kubernetes- node,client  containernetworking-plugins
Although it is not technically mandatory, we will use kubeadm, the most popular installer to create a Kubernetes cluster. Kubeadm is not packaged in Debian, we have to download an upstream binary.
wget https://dl.k8s.io/v1.20.5/kubernetes-server-linux-amd64.tar.gz

sha512sum kubernetes-server-linux-amd64.tar.gz
28529733bf34f5d5b72eabe30a81df98cc7f8e529590f807745cd67986a2c5c3eb86cebc7ecbcfc3df3c50416306e5d150948f2483933ea46c2aebaeb871ea8f  kubernetes-server-linux-arm64.tar.gz

sudo tar --directory=/usr/local/sbin --strip-components 3 -xaf kubernetes-server-linux-amd64.tar.gz kubernetes/server/bin/kubeadm
sudo chmod +x /usr/local/sbin/kubeadm 
sudo kubeadm version
kubeadm version: &version.Info Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:08:27Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"
Add a kubelet systemd unit:
RELEASE_VERSION="v0.4.0"
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/$ RELEASE_VERSION /cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service"   sudo tee /etc/systemd/system/kubelet.service
sudo systemctl enable kubelet
and a default config file for kubeadm
RELEASE_VERSION="v0.4.0"
sudo mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/$ RELEASE_VERSION /cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf"   sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
finally we need to help kubelet find the components needed for container networking
echo 'KUBELET_EXTRA_ARGS="--cni-bin-dir=/usr/lib/cni"'   sudo tee /etc/default/kubelet

Create a clusterInitialize a cluster with kubeadm: this will download container images for the Kubernetes control plane (= the brain of the cluster), and start the containers via the kubelet service. Yes a good part of Kubernetes itself run in containers.
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
...
...
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Follow the instructions from the kubeadm output, and verify you have a single node cluster, with the status NotReady.
kubectl get nodes 
NAME      STATUS     ROLES                  AGE    VERSION
testing   NotReady   control-plane,master   9m9s   v1.20.5
At that point you should also have a bunch of containers running on the node:
sudo docker ps --format ' .Names '
k8s_kube-apiserver_kube-apiserver-testing_kube-system_2711c230d39ccda1e74d1d6386a05cee_0
k8s_POD_kube-apiserver-testing_kube-system_2711c230d39ccda1e74d1d6386a05cee_0
k8s_etcd_etcd-testing_kube-system_4749b1bca3b1a73fd09c8e299d7030fe_0
k8s_POD_etcd-testing_kube-system_4749b1bca3b1a73fd09c8e299d7030fe_0
...
The kubelet service also needs an external network plugin to get the cluster in Ready state.
sudo systemctl status kubelet
...
Mar 28 09:28:43 testing kubelet[9405]: E0328 09:28:43.958059    9405 kubelet.go:2188] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Let s add that network plugin. Download the flannel network plugin definition, and schedule flannel to run on all nodes of your cluster:
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply --filename=kube-flannel.yml
After a dozen of seconds your node should be in ready status.
kubectl get nodes 
NAME      STATUS   ROLES                  AGE   VERSION
testing   Ready    control-plane,master   16m   v1.20.5

Deploy a test applicationOur node is now in Ready status, but we cannot run application on it, since we only have a master node, an administrative node which by default cannot run user applications.
kubectl describe node testing   grep ^Taints
Taints:             node-role.kubernetes.io/master:NoSchedule
Let s allow node testing to run user applications:
kubectl taint node testing node-role.kubernetes.io/master-
Deploy a nginx container:
kubectl run my-nginx-pod --image=docker.io/library/nginx --port=80 --labels="app=http-content"
Create a Kubernetes service to access this pod externally:
cat service.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-k8s-service
spec:
  type: NodePort
  ports:
    - port: 80
      nodePort: 30000
  selector:
    app: http-content 

kubectl create --filename service.yaml
Access the service via IP adress:
curl 192.168.121.63:30000
...
Thank you for using nginx.

NotesI will try to get this blog post in a Debian Wiki article, or maybe in the kubernetes-node documentation. Blog posts deprecate and disappear, wiki and project docs live longer.

Norbert Preining: Debian KDE/Plasma and Digikam Status 2021-04-07

Two months have passed since the last status update, but not much has changed since Debian is more or less frozen for the release of Bullseye, and only critical bugfixes are allowed. As reported before Debian/bullseye will have Plasma 5.20.5, Frameworks 5.78, Apps 20.12. Debian/experimental already carries Plasma 5.21.4 and Frameworks 5.80, and that is also the level at the OSC builds. Debian Bullseye We are in hard freeze now, and only targeted fixes are allowed, but Bullseye is carrying a good mixture consisting of the KDE Frameworks 5.78, including several backports of fixes from 5.79 to get smooth operation. Plasma 5.20.5, again with several cherry picks for bugs will be in Bullseye, too. The KDE/Apps are mostly at 20.12 level, and the KDE PIM group packages (akonadi, kmail, etc) are at 20.08. Debian experimental Frameworks 5.80 (and soon 5.81) and Plasma 5.21.4 are in Debian/experimental. OBS packages (short reminder: you need to import my OBS gpg key to make these repos work!) The OBS packages as usual follow the latest release, and currently ship KDE Frameworks 5.80, KDE Apps 20.12.3, and Plasma 5.21.4. The package sources are as usual (note the different path for the Plasma packages and the App packages, containing the release version!), for Debian/unstable: 
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/frameworks/Debian_Unstable/ ./
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/plasma521/Debian_Unstable/ ./
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/apps2012/Debian_Unstable/ ./
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/other/Debian_Unstable/ ./
and the same with Testing instead of Unstable for Debian/testing. Digikam Digikam has seen a new release 7.2.0, and packages are available in my OBS archives: 
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/other/Debian_Unstable/ ./
and again, same with Testing instead of Unstable for Debian/testing.

5 April 2021

Kees Cook: security things in Linux v5.9

Previously: v5.8 Linux v5.9 was released in October, 2020. Here s my summary of various security things that I found interesting: seccomp user_notif file descriptor injection
Sargun Dhillon added the ability for SECCOMP_RET_USER_NOTIF filters to inject file descriptors into the target process using SECCOMP_IOCTL_NOTIF_ADDFD. This lets container managers fully emulate syscalls like open() and connect(), where an actual file descriptor is expected to be available after a successful syscall. In the process I fixed a couple bugs and refactored the file descriptor receiving code. zero-initialize stack variables with Clang
When Alexander Potapenko landed support for Clang s automatic variable initialization, it did so with a byte pattern designed to really stand out in kernel crashes. Now he s added support for doing zero initialization via CONFIG_INIT_STACK_ALL_ZERO, which besides actually being faster, has a few behavior benefits as well. Unlike pattern initialization, which has a higher chance of triggering existing bugs, zero initialization provides safe defaults for strings, pointers, indexes, and sizes. Like the pattern initialization, this feature stops entire classes of uninitialized stack variable flaws. common syscall entry/exit routines
Thomas Gleixner created architecture-independent code to do syscall entry/exit, since much of the kernel s work during a syscall entry and exit is the same. There was no need to repeat this in each architecture, and having it implemented separately meant bugs (or features) might only get fixed (or implemented) in a handful of architectures. It means that features like seccomp become much easier to build since it wouldn t need per-architecture implementations any more. Presently only x86 has switched over to the common routines. SLAB kfree() hardening
To reach CONFIG_SLAB_FREELIST_HARDENED feature-parity with the SLUB heap allocator, I added naive double-free detection and the ability to detect cross-cache freeing in the SLAB allocator. This should keep a class of type-confusion bugs from biting kernels using SLAB. (Most distro kernels use SLUB, but some smaller devices prefer the slightly more compact SLAB, so this hardening is mostly aimed at those systems.) new CAP_CHECKPOINT_RESTORE capability
Adrian Reber added the new CAP_CHECKPOINT_RESTORE capability, splitting this functionality off of CAP_SYS_ADMIN. The needs for the kernel to correctly checkpoint and restore a process (e.g. used to move processes between containers) continues to grow, and it became clear that the security implications were lower than those of CAP_SYS_ADMIN yet distinct from other capabilities. Using this capability is now the preferred method for doing things like changing /proc/self/exe. debugfs boot-time visibility restriction
Peter Enderborg added the debugfs boot parameter to control the visibility of the kernel s debug filesystem. The contents of debugfs continue to be a common area of sensitive information being exposed to attackers. While this was effectively possible by unsetting CONFIG_DEBUG_FS, that wasn t a great approach for system builders needing a single set of kernel configs (e.g. a distro kernel), so now it can be disabled at boot time. more seccomp architecture support
Michael Karcher implemented the SuperH seccomp hooks, Guo Ren implemented the C-SKY seccomp hooks, and Max Filippov implemented the xtensa seccomp hooks. Each of these included the ever-important updates to the seccomp regression testing suite in the kernel selftests. stack protector support for RISC-V
Guo Ren implemented -fstack-protector (and -fstack-protector-strong) support for RISC-V. This is the initial global-canary support while the patches to GCC to support per-task canaries is getting finished (similar to the per-task canaries done for arm64). This will mean nearly all stack frame write overflows are no longer useful to attackers on this architecture. It s nice to see this finally land for RISC-V, which is quickly approaching architecture feature parity with the other major architectures in the kernel. new tasklet API
Romain Perier and Allen Pais introduced a new tasklet API to make their use safer. Much like the timer_list refactoring work done earlier, the tasklet API is also a potential source of simple function-pointer-and-first-argument controlled exploits via linear heap overwrites. It s a smaller attack surface since it s used much less in the kernel, but it is the same weak design, making it a sensible thing to replace. While the use of the tasklet API is considered deprecated (replaced by threaded IRQs), it s not always a simple mechanical refactoring, so the old API still needs refactoring (since that CAN be done mechanically is most cases). x86 FSGSBASE implementation
Sasha Levin, Andy Lutomirski, Chang S. Bae, Andi Kleen, Tony Luck, Thomas Gleixner, and others landed the long-awaited FSGSBASE series. This provides task switching performance improvements while keeping the kernel safe from modules accidentally (or maliciously) trying to use the features directly (which exposed an unprivileged direct kernel access hole). filter x86 MSR writes
While it s been long understood that writing to CPU Model-Specific Registers (MSRs) from userspace was a bad idea, it has been left enabled for things like MSR_IA32_ENERGY_PERF_BIAS. Boris Petkov has decided enough is enough and has now enabled logging and kernel tainting (TAINT_CPU_OUT_OF_SPEC) by default and a way to disable MSR writes at runtime. (However, since this is controlled by a normal module parameter and the root user can just turn writes back on, I continue to recommend that people build with CONFIG_X86_MSR=n.) The expectation is that userspace MSR writes will be entirely removed in future kernels. uninitialized_var() macro removed
I made treewide changes to remove the uninitialized_var() macro, which had been used to silence compiler warnings. The rationale for this macro was weak to begin with ( the compiler is reporting an uninitialized variable that is clearly initialized ) since it was mainly papering over compiler bugs. However, it creates a much more fragile situation in the kernel since now such uses can actually disable automatic stack variable initialization, as well as mask legitimate unused variable warnings. The proper solution is to just initialize variables the compiler warns about. function pointer cast removals
Oscar Carter has started removing function pointer casts from the kernel, in an effort to allow the kernel to build with -Wcast-function-type. The future use of Control Flow Integrity checking (which does validation of function prototypes matching between the caller and the target) tends not to work well with function casts, so it d be nice to get rid of these before CFI lands. flexible array conversions
As part of Gustavo A. R. Silva s on-going work to replace zero-length and one-element arrays with flexible arrays, he has documented the details of the flexible array conversions, and the various helpers to be used in kernel code. Every commit gets the kernel closer to building with -Warray-bounds, which catches a lot of potential buffer overflows at compile time. That s it for now! Please let me know if you think anything else needs some attention. Next up is Linux v5.10.

2021, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

Jelmer Vernooij: The Buildlog Consultant

Reading build logs Build logs for Debian packages can be quite long and difficult for a human to read. Anybody who has looked at these logs trying to figure out why a build failed will have spent time scrolling through them and skimming for certain phrases (lines starting with error: for example). In many cases, you can spot the problem in the last 10 or 20 lines of output but it s also quite common that the error is somewhere at the beginning of many pages of error output.
The buildlog consultant The buildlog consultant project attempts to aid in this process by parsing sbuild and non-Debian (e.g. the output of make ) build logs and trying to identify the key line that explains why a build failed. It can then either display this specific line, or a fragment of the log around surrounding the key line.
Classification In addition to finding the key line explaining the failure, it can also classify and parse the error in many cases and return a result code and some metadata. For example, in a failed build of gnss-sdr that has produced 2119 lines of output, the reason for the failure is that log4cpp is missing which is on line 641: 
634
635
636
637
638
639
640
641
642
643
644
645
646
647
 -- Required GNU Radio Component: ANALOG missing!
 -- Could NOT find GNURADIO (missing: GNURADIO_RUNTIME_FOUND)
 -- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
 -- Could NOT find LOG4CPP (missing: LOG4CPP_INCLUDE_DIRS
 LOG4CPP_LIBRARIES)
 CMake Error at CMakeLists.txt:593 (message):
   *** Log4cpp is required to build gnss-sdr
 -- Configuring incomplete, errors occurred!
 See also "/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/CMakeFiles/
 CMakeOutput.log".
 See also "/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/CMakeFiles/
 CMakeError.log".
In this case, the buildlog consultant can both figure out line was problematic and what the problem was: 
 % analyse-sbuild-log build.log
 Failed stage: build
 Section: build
 Failed line: 641:
   *** Log4cpp is required to build gnss-sdr
 Error: Missing dependency: Log4cpp
Or, if you'd like to do something else with the output, use JSON output: 
 % analyse-sbuild-log --json build.log
  "stage": "build", "section": "Build", "lineno": 641, "kind": "missing-dependency", "details":  "name": "Log4cpp""
How it works The consultant does some structured parsing (most notably it can parse the sections from a sbuild log), but otherwise is a large set of carefully crafted regular expressions and heuristics. It doesn t always find the problem, but has proven to be fairly accurate. It is constantly improved as part of the Debian Janitor project, and that exposes it to a wide variety of different errors. You can see the classification and error detection in action on the result codes page of the Janitor.
Using the buildlog consultant You can get the buildlog consultant from either pip or Debian unstable (package: python3-buildlog-consultant ). The buildlog consultant comes with two scripts analyse-build-log and analyse-sbuild-log, for analysing build logs and sbuild logs respectively.

Charles Plessy: Debian Analytica

A couple of days ago I wrote on debian-vote@ that a junior analyst could study the tally sheets of our general resolutions and find the cracks in our community. In the end, with a quite na ve approach and a time budget of a few hours, I did not manage anything of interest. The figure below shows one circle per voter and my position as a red dot. The circles are spaces according to the similarity of the vote profiles after I concatenated the results of all GRs until 2010. So if there is something to extract from these data, it will need a more expert analyst This said, I think that our future votes should all be anonymous, and that we should stop distributing that kind of data.

Russ Allbery: Book haul

Haven't done one of these posts in a while. We're well into award season now, plus the early pre-orders for 2021 have come in. A few in here I've already read and reviewed. C.L. Clark The Unbroken (sff)
Louis Hyman Temp (non-fiction)
T. Kingfisher Paladin's Strength (sff)
Mary Robinette Kowal The Relentless Moon (sff)
Arkady Martine A Desolation Called Peace (sff)
Cal Newport A World Without Email (non-fiction)
Cal Newport How to Become a Straight-A Student (non-fiction)
Karen Osborne Architects of Memory (sff)
David R. Palmer Tracking (sff)
Chandra Prescod-Weinstein The Disordered Cosmos (non-fiction)
C.L. Polk The Midnight Bargain (sff)
C.L. Polk Witchmark (sff)
Rebecca Roanhorse Black Sun (sff)
Elizabeth Sandifer Neoreaction a Basilisk (non-fiction)
Tasha Suri Empire of Sand (sff)
John Kennedy Toole A Confederacy of Dunces (mainstream)
Tor.com (ed.) Some of the Best from Tor.com: 2016 (sff anthology)
Tor.com (ed.) Some of the Best from Tor.com: 2020 (sff anthology)
Nghi Vo The Empress of Salt and Fortune (sff) March was not as good of a month for reading as January and February were, but there are so many good things awaiting my attention that hopefully April will provide more time and attention.

2 April 2021

Rapha&#235;l Hertzog: Challenging times for Freexian (4/4)

Note: This is the continuation of part 1, part 2 and part 3. You can get the full document as a single PDF. Feel free to share this document to anyone that might be interested to work towards the goals outlined. Conclusion I m very excited by the perspective that I outlined in this document. It really resonates with my own mission statement as a Debian developer (written a long time ago):
My main role in Debian is to help Debian to evolve so that it s always able to face the new challenges that are showing up.My approach is both corrective and proactive, I work to solve current problems and prepare for tomorrow s. This requires to remain sufficiently involved to identify new trends, see the deficiencies and be a force of proposition.Most of the changes require to interact with many people, and problems are often more relational than technical. I will ensure to follow the habits of interdependence (Think Win-Win, Seek First to Understand, Then to be Understood, Synergize) to find a solution acceptable to all and to inspire others to do the same.The easiest changes to implement are technical (such as improvements to distro-tracker) and require little interaction. This work is used to recharge me by offering me an immediate reward for my efforts.Finally, and this is a substantive effort, I want to create in the project working conditions that allow all contributors to give their best. It starts with developing a common vision
But I can t achieve this alone, I need help from passionate individuals sharing this vision. Let me know if you want to be one of those persons.

1 April 2021

Rapha&#235;l Hertzog: Challenging times for Freexian (3/4)

Note: This is the continuation of part 1 and part 2. Going forward: growing Freexian Part 2: Extending the team By all accounts, Freexian is still a small company which relies largely on me in many aspects. The growth of its business is however providing enough financial margin to allow looking into ways to recruit external help, be it through direct hiring (for French residents) or via long term contracting (for people based in other countries). If you believe you could be the right person for one of the roles listed below, or if you know someone that we should contact, please reach out to raphael@freexian.com. Project manager I m looking for someone that cares about Debian and that has the following skills:
  • knows how to manage developers and software projects
    • bonus points for any experience in environments mixing volunteers and paid contributors
  • is fluent and experienced enough in Python to be able to do software design and code reviews
    • bonus points for experience with: Django, Test Driven Development
That person would handle (some of) the following tasks:
  • lead the Debian project funding initiative to a success
    • find useful projects to fund, for example by
      • discussing with various Debian teams / contributors (including the DPL)
      • running a survey among Debian developers
      • doing your own analysis
    • help with drafting and specifying the various projects
    • help to find someone to implement and review the projects
    • coordinate with those persons during execution
  • manage other free software projects that Freexian would like to pursue
    • debusine: a software factory tailored for Debian packages
      • participate in design discussions, set milestones and goals
        • start with the short term needs of Freexian
        • but take into account the needs of Debian so that it can replace some aging infrastructure within Debian
      • coordinate with contractors, possibly implement some parts
    • infrastructure to run the various Freexian services and automate most of the administrative work (see Part2: From Debian LTS to Debian for the Enterprise )
  • maybe coordinate the team of paid LTS/ELTS contributors
Debian/Python Developer While the current priority is on the above role, there could also be room for a developer role with the following tasks:
  • Creation and maintenance of Debian packages
  • Technical support
  • Software development in Python (debusine, internal infrastructure)
  • Security support (contributor to Debian LTS)
Sales manager / sales representative Up until now, the growth of Freexian has mostly been organic, through word of mouth and increased awareness of Debian LTS within the Debian community. We never spent a single euro on advertising, except for one promotional video and for Debconf sponsorship (with a flyer and stickers). But if we can manage to make a positive impact on Debian through the funding that Freexian brings, then I m interested to grow the company so that we can pay more people to work on Debian. That growth likely would have to go through some more active sales work. At the same time, it is an opportunity for me to delegate (some of) the administrative work that lies solely on my shoulders (invoicing, day to day customer relationship, etc.). I assume it will be hard to find a member of the Debian community that has an interest in those areas, but who knows This article is to be continued in an upcoming post. Stay tuned!

31 March 2021

Gunnar Wolf: And what does the FSF have, anyway?

Following up with my previous post, it seems the FSF s board is taking good care of undermining the FSF itself. Over few days, it has:
  • Lost support from tens of organizations and companies, several of them important funders for the FSF s activities
  • Alienated long-standing staff within it, leading to many important resignations
  • Divided the overall free software community, gathering several thousand signatures both repudiating its actions and backing it
  • In the Debian project, we have started a General Resolution process, with opinions all over the spectrum, to gauge whether the project will back either of the positions or none.
Now Many people have pointed to the fact that the FSF has been a sort of a moral leader pushing free software awareness But if they lose their moral statre, what s in there? What power do they bear? Why do we care? And the answer, at least one of them, is simple and strong. The General Public License (GPL), both in its v2 and v3 revisions, read: 
Each version is given a distinguishing version number.  If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation.  If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
Years ago there was a huge argument on why Linux was licensed as GPLv2 only, without the option to relicense under GPLv3. Of course, by then, Linux had had thousands of authors, and they would all have to agree to change license so it would have been impossible even if it were wanted. But yes, some people decried several terms of GPLv3 not being aligned with their views of freedom. Well, so if the FSF board manages to have it their way, and get everybody mark them as irrelevant, they will still be the stewards of the GPL. Thousands of projects are licensed under the GPL v2 or v3 or later . Will we continue to trust the FSF s stewardship, if it just becomes a board of big egos, with no respect of what happens in the free software ecosystem? My suggestion is, for all project copyright holders that are in a position to do so, to drop the or-later terms and stick to a single, known GPL version.

Chris Lamb: Free software activities in March 2021

Here is my monthly update covering what I have been doing in the free software world during March 2021 (previous month):
  • Reviewed and merged a number of contributions from Dan Palmer for my django-autologin library aimed at applications that use the Django web-development framework that wish to include automatic "login" links in emails (etc.). Changes made include allowing callers to override max-age [...], ensuring we always retrieve the same user account [...] and to improve the usability of the public API [...].
  • Opened a pull request to make the build process for Scrapy (a framework for extracting data from websites) reproducible. [...]
Reproducible Builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. The project is proud to be a member project of the Software Freedom Conservancy. Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter. This month, I:
  • Categorised a huge number of packages and issues in the Reproducible Builds "notes" repository.
  • In Debian:
    • Kept isdebianreproducibleyet.com up to date. [...]
    • I submitted 4 patches to fix specific reproducibility issues in cdebootstrap, jalview, php8.0 & python-scrapy.
    • Whilst looking into reproducible builds issues, I noticed that the heudiconv package could not be built reproducibly. This was because the call to help2man fails, so the manual page includes a Python traceback instead of the actual manpage. I filed a bug with a patch as Debian bug #984778.
    • I uploaded flask-peewee (0.6.7-3) to Debian to make the build reproducible (#885326) and did the same for pyvows (3.0.0-3) (#977487) too, refreshing the packaging at the same time.

I also made the following changes to diffoscope, including uploading versions 169, 170 and 171 to Debian:
  • New features:
    • If zipinfo(1) shows a difference but we cannot uncover a difference within the underlying .zip or .apk file, add a comment to the output and actually show the binary comparison. (#246)
    • Ensure all our temporary directories have useful names. [...]
    • Ignore --debug and similar arguments when creating a (hopefully-useful) temporary directory suffix. [...]
  • Optimisations:
    • Avoid frequent long lines in RPM header outputs that cause extremely slow HTML output generation. (#245)
    • Use larger read buffer block sizes when extracting files from archives. [...]
    • Use a much-shorter HTML class name instead of diffponct to optimise HTML output. [...]
  • Output improvements:
    • Make error extracting X, falling back to binary comparison 'Y' error message in diffoscope's output nicer. [...]
    • Don't emit "Unable to stat file" debug messages at all. We have entirely-artificial directory "entries" such as ELF sections which, of course, will never exist as files. [...]
  • Logging improvements:
    • Add the target directory when logging which directory we are extracting containers to. [...]
    • Format report size messages when generating HTML reports. [...]
    • Don't emit a Returning a FooContainer logging message too, as we already emit Instantiating a FooContainer log message. [...]
    • Reduce "Unable to stat file" warnings to debug messages as these are sometimes by design. [...]
  • Misc improvements:
    • Clarify a comment regarding not extracting excluded files. [...]
    • Remove trailing newline from updated test file (re: #243). [...]
    • Fix test_libmix_differences failure on openSUSE Tumbleweed. (#244)
    • Move test_rpm to use the assert_diff utility helper.
Debian Uploads
  • redis:
    • 6.0.12-1 New upstream release.
    • 6.2.1-1 New upstream release.
  • python-django (3.2~rc1-1) New upstream beta release.
  • flask-peewee (0.6.7-3) (via the Debian Python packaging team) Upload to refresh packaging and to make the build reproducible. (#885326)
  • bfs (2.2-1) New upstream release.
  • pyvows (3.0.0-3) (via the Debian Python packaging team) Refresh packaging and make the build reproducible. (#977487)

Debian LTS This month I worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.
  • Frontdesk duties, responding to user/developer questions, attending monthly meeting, reviewing others' packages, participating in internal mailing list discussions, etc.
  • Investigated and triaged botan1.10 (CVE-2021-24115), busybox (CVE-2021-28831), courier-authlib (CVE-2021-28374), edk2 (CVE-2021-28210 & CVE-2021-28211), netty (CVE-2021-21295), open-build-service (CVE-2020-8031), openjpeg2 (CVE-2020-27844), python2.7 (CVE-2021-23336), rpm (CVE-2021-20248, [CVE-2021-20249](https://security-tracker.debian.org/tracker/CVE-2021-20249), CVE-2021-20266 & CVE-2021-20271), ruby-activerecord-session-store (CVE-2019-25025), ruby-carrierwave (CVE-2021-21288), salt (CVE-2020-28243, etc.), slic3r (CVE-2020-28591), squid3 (CVE-2020-25097 & CVE-2021-28116), velicity-tools (CVE-2020-13959), velocity (CVE-2020-13936) & yara (CVE-2021-3402).
  • Proposed an stable update for python-django in buster. (#983526)
  • Prepared and uploaded a stable update for redis in buster. (#983527)
  • Issued DLA 2595-1 and ELA 380-1 for velocity, a Java-based template engine for writing web applications. Velocity could be exploited to run arbitrary code by applications that allowed untrusted users to upload/modify templates.
  • Issued DLA 2597-1 and ELA 381-1 to address a cross-site scripting (XSS) vulnerability in velocity-tools, a collection of useful tools for the Velocity template engine. The default error page could be exploited to steal session cookies, perform requests in the name of the victim, used for phishing attacks and many other similar attacks.
  • Issued DLA 2600-1 and ELA 384-1 for pygments as it was discovered that there was a series of denial of service vulnerabilities in this syntax highlighting library for Python. A number of regular expressions had cubic (or even exponential) worst-case complexity which could cause a remote denial of service (DoS) when provided with malicious input.
  • Issued DLA 2603-1 to address a number of vulnerabilities in libmediainfo, a library reading metadata such as track names, lengths, etc. from media files.
You can find out more about the Debian LTS via the following video:

Rapha&#235;l Hertzog: Challenging times for Freexian (2/4)

Note: This is the continuation of part 1 where I presented Freexian and its purpose. Going forward: growing Freexian Part 1: From Debian LTS to Debian for the Enterprise Freexian s Debian LTS service has so far been entirely successful, with a steady growth over the years. Thanks to this, and even if there are always new challenges, it is fair to say that the Debian LTS team has met its goal in the last few years. While this started from the desire to make LTS a reality, many sponsors are only looking for a way to give back to Debian through their company, and to make sure that Debian fits their needs. But if you look at the bigger picture outside of this small LTS area, you will easily find many issues that need to be addressed if we want Debian to meet the needs of corporate users. Those issues can have widely different types and complexity. They can be as simple as missing the latest upstream version for an important package because the maintainer disappeared and nobody noticed before it was too late (i.e. the release was frozen); or a somewhat basic piece of software not yet packaged at all; or a release critical bug that was left unattended. On the other end of the spectrum, some corporate requirements will prove tougher to solve, for instance for large software suites that are complex to package, or could potentially have an impact elsewhere in Debian. Bringing those facts together, we would like to have Freexian s Debian LTS/ELTS offering evolve into a more general Debian Software Assurance offering, where you commit to a yearly budget for Debian sponsorship in the larger sense. That budget would fund different projects and the allocation between those projects would vary over time depending on the desires and needs of the sponsors/customers:
  • Technical support: the budget would always ensure that you have a few spare hours of technical support available in case you need them
  • Debian LTS: we want this to continue!
  • Debian ELTS: when the customer has not managed to migrate their Debian servers in time, they should be able to reallocate their budget towards ELTS and ensure their servers are secure until the migration has taken place.
  • Debian for the enterprise
    • Make sure that the packages used by sponsors are in good shape in Debian Testing/Unstable so that they are in the best shape for the next stable release.
    • Package new software that are relevant for corporate users. Offer to pool the maintenance work.
    • Fix bugs that customers are hitting.
    • Etc.
  • Debian project funding: that s the variable part of the budget (and would have a minimum of 10% like we do for Debian LTS right now). When the other projects do not consume the whole budget, we invest the remaining money into generic Debian improvements.
This major shift in our offering would also be an ideal opportunity to build a professional, free-software based infrastructure aimed at sustaining this business, making it easier to administer the various aspects of this work, and easily allowing many more sponsors to join (individuals included!). On a more pragmatic/operational note, this shift will bring a lot of challenges to the table, and those can hardly be handled with the current resources of Freexian: if we hope to properly implement this new strategy, we ll need some additional help. This article is to be continued in an upcoming post. Stay tuned!

Ben Hutchings: Debian LTS work, March 2021

In March I was assigned 16 hours of work by Freexian's Debian LTS initiative and carried over 12.25 hours from earlier months. I worked 25.75 hours and will carry over the remainder. I eventually settled on an apparently working patch series to fix the futex security issue in Linux 4.9. This went through upstream stable review and was included in 4.9.260. I applied the same fixes to the Debian package, along with some other security and regression fixes. I uploaded it and issued DLA-2586-1. Unfortunately the futex changes for Linux 4.9 still caused a regression (kernel WARNING in some circumstances). I worked to backport and test a further set of fixes that had already been applied to later kernel branches. These were included in upstream stable release 4.9.264 and should go into an updated Debian package soon. Following the Debian 10.9 point release, I also backported the updated Linux 4.19 package. I uploaded it and issued DLA-2610-1.

Next.

Previous.