Here s my (ninth) monthly update about the activities I ve done in the F/L/OSS world.
Debian
This was my 16th month of contributing to Debian.
I became a DM in late March last year and a DD last Christmas! \o/
This month was a little intense. I did a lot of different kinds of things in Debian this month. Whilst most of my time went on doing security stuff, I also sponsored a bunch of packages.
Here are the following things I did this month:
Uploads and bug fixes:
rails (2:5.2.4.3+dfsg-1) - fix a bunch of CVEs in Sid and Bullseye.
Sponsored ruby-ast for Abraham, libexif for Hugh, djangorestframework-gis and karlseguin-ccache for Nilesh, and twig-extensions, twig-i18n-extension, and mariadb-mysql-kbs for William.
GSoC Phase 1, Part 2!
Last month, I got selected as a Google Summer of Code student for Debian again! \o/
I am working on the Upstream-Downstream Cooperation in Ruby project.
The first half of the first month is blogged here, titled, GSoC Phase 1.
Also, I log daily updates at gsocwithutkarsh2102.tk.
Whilst the daily updates are available at the above site^, I ll breakdown the important parts of the later half of the first month here:
Spread the word/usage about this tool/library via adding them in the official RuboCop docs.
We had our third weekly meeting where we discussed the next steps and the things that are supposed to be done for the next set of cops.
Wrote more tests so as to cover different aspects of the GemspecGit cop.
Opened PR #4 for the next Cop, RequireRelativeToLib.
Introduced rubocop-packaging to the outer world and requested other upstream projects to use it! It is being used by 6 other projects already
Had our fourth weekly meeting where we pair-programmed (and I sucked :P) and figured out a way to make the second cop work.
Found a bug, reported at issue #5 and raised PR #6 to fix it.
And finally, people loved the library/tool (and it s outcome):
(for those who don t know, @bbatsov is the author of RuboCop, @lienvdsteen is an amazing fullstack engineer at GitLab, and @pboling is the author of some awesome Ruby tools and libraries!)
Continuation of GSoC for other Ruby related stuff!
Whilst I have already mentioned it multiple times but it s still not enough to stress how amazing Antonio Terceiro and David Rodr guez are!
They re more than just mentors to me!
Well, only they know how much I trouble them with different things, which are not only related to my GSoC project but also extends to the projects they maintain! :P
David maintains rubygems and bundler and Antonio maintains debci.
So on days when I decide to hack on rubygems or debci, only I know how kind and nice David and Anotonio are to me!
They very patiently walk me through with whatever I am stuck on, no matter what and no matter when.
Thus, with them around, I contributed to these two projects and more, with regards to working on rubocop-packaging.
Following are a few things that I raised:
Debian LTS
Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases
to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group
of volunteers and companies interested in making it a success.
This was my ninth month as a Debian LTS paid contributor. I was assigned 30.00 hours and worked on
the following things:
Uploaded a fix for CVE-2020-11082, for ruby-kaminari.
This upload was for Sid and Bullseye and this CVE was fixed in version 1.0.1-6.
Uploaded a fix for CVE-2020-10663, for ruby-json, ruby2.1, and ruby2.5.
These uploads were for Stretch and Buster and were fixed in the version 2.3.3-1+deb9u8, 2.1.0+dfsg-2+deb10u1, 2.3.3-1+deb9u8, and 2.5.5-3+deb10u2.
Other(s)
Sometimes it gets hard to categorize work/things into a particular category.
That s why I am writing all of those things inside this category.
This includes two sub-categories and they are as follows.
Personal:
This month I did the following things:
Wrote and published v0.1.0 of rubocop-packaging on RubyGems!
It s open-sourced and the repository is here.
Bug reports and pull requests are welcomed!
Integrated a tiny (yet a powerful) hack to align images in markdown for my blog.
Commit here.
Here is my monthly update covering what I have been doing in the free software world during June 2020 (previous month):
Opened two pull requests against the Ghostwriter distraction-free Markdown editor to:
Persist whether "focus mode" is enabled across between sessions. (#522)
Correct the ordering of the MarkdownAST::toString() debugging output. (#520)
Will McGugan's "Rich" is a Python library to output formatted text, tables, syntax etc. to the terminal. I filed a pull request in order to allow for easy enabling and disabling of displaying the file path in Rich's logging handler. (#115)
As part of my duties of being on the board of directors of the Open Source Initiative and Software in the Public Interest I attended their respective monthly meetings and participated in various licensing and other discussions occurring on the internet, as well as the usual internal discussions regarding logistics and policy etc.
Filed a pull request against the PyQtGraph Scientific Graphics and graphical user interface library to make the documentation build reproducibly. (#1265)
Reviewed and merged a large number of changes by Pavel Dolecek to my Strava Enhancement Suite, a Chrome extension to improve the user experience on the Strava athletic tracker.
For Lintian, the static analysis tool for Debian packages:
Don't emit breakout-link for architecture-independent .jar files under /usr/lib. (#963939)
Correct a reference to override_dh_ in the long description of the excessive-debhelper-overrides tag. [...]
Update data/fields/perl-provides for Perl 5.030003. [...]
Check for execute_after and execute_before spelling mistakes just like override_*. [...]
Reproducible Builds
One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes.
The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.
The project is proud to be a member project of the Software Freedom Conservancy. Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter.
This month, I:
Filed a pull request against the PyQtGraph to make the documentation build reproducibly. (#1265)
Addressed a regression in the octave-queueing package in the handling of absolute paths in Texinfo files. (#962187)
Submitted the following patches to fix reproducibility-related toolchain issues within Debian:
critcl: Please make the teapot.txt files reproducible. (#963600)
sasmodels: Regression in debian/patches/reproducible-c-models.patch. (#962050)
Contributed to a discussion regarding improving the reproducibility of packages that use the CMake build system. [...]
Added to a conversation regarding the nondeterministic execution of order of Debian maintainer scripts that results in the arbitrary allocation of UNIX group IDs referencing the Tails operating system. [...]
Replace obscure references to "WF" with "Wagner-Fischer" for clarity. [...]
Use a semantic AbstractMissingType type instead of remembering to check for both types of "missing" files. [...]
Add a comment regarding potential security issue in the .changes, .dsc and .buildinfo comparators. [...]
Drop a large number of unused imports. [...][...][...][...][...]
Make many code sections more Pythonic. [...][...][...][...]
Prevent some variable aliasing issues. [...][...][...]
Use some tactical f-strings to tidy up code [...][...] and remove explicit u"unicode" strings [...].
Refactor a large number of routines for clarity. [...][...][...][...]
trydiffoscope is the web-based version of diffoscope. This month, I specified a location for the celerybeat scheduler to ensure that the clean/tidy tasks are actually called which had caused an accidental resource exhaustion. (#12)
Frontdesk duties, including responding to user/developer questions, reviewing others' packages, participating in mailing list discussions as well as attending our contributor meeting.
Issued DLA 2233-1 to fix two issues in the Django web development framework in order to fix a potential data leakage via malformed memcached keys (CVE-2020-13254) and to prevent a cross-site scripting attack in the Django administration system (CVE-2020-13596). This was followed by DLA 2233-2 to address a regression as well as uploads to Debian stretch (1.10.7-2+deb9u9) and buster (1.11.29-1~deb10u1). (More info)
Issued DLA 2239-1 for a security module for using the TACACS+ authentication service to prevent an issue where shared secrets such as private server keys were being added in plaintext to various logs.
Issued DLA 2244-1 to address an escaping issue in PHPMailer, an email generation utility class for the PHP programming language.
Issued DLA 2252-1 for the ngircd IRC server as it was discovered that there was an out-of-bounds access vulnerability in the server-to-server protocol.
Issued DLA 2253-1 to resolve a vulnerability in the Lynis a security auditing tool because a shared secret could be obtained by simple observation of the process list when a data upload is being performed.
You can find out more about the project via the following video:
Uploads
The Fifth Risk starts with the presidential transition. Max Stier,
the first person profiled by Lewis in this book, is the founder of the
Partnership for Public Service. That foundation helped push through laws
to provide more resources and structure for the transition of the United
States executive branch from one president to the next. The goal was to
fight wasted effort, unnecessary churn, and pointless disruption in the
face of each administration's skepticism about everyone who worked for the
previous administration.
"It's Groundhog Day," said Max. "The new people come in and
think that the previous administration and the civil service are lazy
or stupid. Then they actually get to know the place they are
managing. And when they leave, they say, 'This was a really hard job,
and those are the best people I've ever worked with.' This happens
over and over and over."
By 2016, Stier saw vast improvements, despite his frustration with other
actions of the Obama administration. He believed their transition
briefings were one of the best courses ever produced on how the federal
government works. Then that transition process ran into Donald Trump.
Or, to be more accurate, that transition did not run into Donald Trump,
because neither he nor anyone who worked for him were there. We'll never
know how good the transition information was because no one ever listened
to or read it. Meetings were never scheduled. No one showed up.
This book is not truly about the presidential transition, though, despite
its presence as a continuing theme. The Fifth Risk is, at its
heart, an examination of government work, the people who do it, why it
matters, and why you should care about it. It's a study of the surprising
and misunderstood responsibilities of the departments of the United States
federal government. And it's a series of profiles of the people who
choose this work as a career, not in the upper offices of political
appointees, but deep in the civil service, attempting to keep that system
running.
I will warn now that I am far too happy that this book exists to be
entirely objective about it. The United States desperately needs basic
education about the government at all levels, but particularly the federal
civil service. The public impression of government employees is skewed
heavily towards the small number of public-facing positions and towards
paperwork frustrations, over which the agency usually has no control
because they have been sabotaged by Congress (mostly by Republicans,
although the Democrats get involved occasionally). Mental images of who
works for the government are weirdly selective. The Coast Guard could say
"I'm from the government and I'm here to help" every day, to the immense
gratitude of the people they rescue, but Reagan was still able to use that
as a cheap applause line in his attack on government programs.
Other countries have more functional and realistic social attitudes
towards their government workers. The United States is trapped in a
politically-fueled cycle of contempt and ignorance. It has to stop. And
one way to help stop it is someone with Michael Lewis's story-telling
skills writing a different narrative.
The Fifth Risk is divided into a prologue about presidential
transitions, three main parts, and an afterword (added in current
editions) about a remarkable government worker whom you likely otherwise
would never hear about. Each of the main parts talks about a different
federal department: the Department of Energy, the Department of
Agriculture, and the Department of Commerce. In keeping with the theme of
the book, the people Lewis profiles do not do what you might expect from
the names of those departments.
Lewis's title comes from his discussion with John MacWilliams, a former
Goldman Sachs banker who quit the industry in search of more personally
meaningful work and became the chief risk officer for the Department of
Energy. Lewis asks him for the top five risks he sees, and if you know
that the DOE is responsible for safeguarding nuclear weapons, you will be
able to guess several of them: nuclear weapons accidents, North Korea, and
Iran. If you work in computer security, you may share his worry about the
safety of the electrical grid. But his fifth risk was project management.
Can the government follow through on long-term hazardous waste safety and
cleanup projects, despite constant political turnover? Can it attract new
scientists to the work of nuclear non-proliferation before everyone with
the needed skills retires? Can it continue to lay the groundwork with
basic science for innovation that we'll need in twenty or fifty years?
This is what the Department of Energy is trying to do.
Lewis's profiles of other departments are similarly illuminating. The
Department of Agriculture is responsible for food stamps, the most
effective anti-poverty program in the United States with the possible
exception of Social Security. The section on the Department of Commerce
is about weather forecasting, specifically about NOAA (the National
Oceanic and Atmospheric Administration). If you didn't know that all of
the raw data and many of the forecasts you get from weather apps and web
sites are the work of government employees, and that AccuWeather has
lobbied Congress persistently for years to prohibit the NOAA from making
their weather forecasts public so that AccuWeather can charge you more for
data your taxes already paid for, you should read this book. The story of
American contempt for government work is partly about ignorance, but it's
also partly about corporations who claim all of the credit while selling
taxpayer-funded resources back to you at absurd markups.
The afterword I'll leave for you to read for yourself, but it's the story
of Art Allen, a government employee you likely have never heard of but
whose work for the Coast Guard has saved more lives than we are able to
measure. I found it deeply moving.
If you, like I, are a regular reader of long-form journalism and watch for
new Michael Lewis essays in particular, you've probably already read long
sections of this book. By the time I sat down with it, I think I'd read
about a third in other forms on-line. But the profiles that I had already
read were so good that I was happy to read them again, and the additional
stories and elaboration around previously published material was more than
worth the cost and time investment in the full book.
It was never obvious to me that anyone would want to read what had
interested me about the United States government. Doug Stumpf, my
magazine editor for the past decade, persuaded me that, at this
strange moment in American history, others might share my enthusiasm.
I'll join Michael Lewis in thanking Doug Stumpf.
The Fifth Risk is not a proposal for how to fix government, or
politics, or polarization. It's not even truly a book about the Trump
presidency or about the transition. Lewis's goal is more basic: The
United States government is full of hard-working people who are doing good
and important work. They have effectively no public relations department.
Achievements that would result in internal and external press releases in
corporations, not to mention bonuses and promotions, go unnoticed and
uncelebrated. If you are a United States citizen, this is your government
and it does important work that you should care about. It deserves the
respect of understanding and thoughtful engagement, both from the
citizenry and from the politicians we elect.
Rating: 10 out of 10
The twelveth release of littler as a CRAN package is now available, following in the fourteen-ish year history as a package started by Jeff in 2006, and joined by me a few weeks later.
littler is the first command-line interface for R as it predates Rscript. It allows for piping as well for shebang scripting via #!, uses command-line arguments more consistently and still starts faster. It also always loaded the methods package which Rscript only started to do in recent years.
littler lives on Linux and Unix, has its difficulties on macOS due to yet-another-braindeadedness there (who ever thought case-insensitive filesystems as a default where a good idea?) and simply does not exist on Windows (yet the build system could be extended see RInside for an existence proof, and volunteers are welcome!). See the FAQ vignette on how to add it to your PATH.
A few examples are highlighted at the Github repo, as well as in the examples vignette.
This release mostly responds to the recent docopt release 0.7.0 which brought a breaking change for quoted arguments. In short, it is for the better because an option --as-cran is now available parsed as opt$as_cran which is easier than the earlier form where we needed to back-tick protect as-cran containing an underscore. We also added a new portmanteau-ish option to roxy.r.
The NEWS file entry is below.
Changes in littler version 0.3.11 (2020-06-26)
Changes in examples
Scripts check.r and rcc.r updated to reflect updated docopt 0.7.0 behaviour of quoted arguments
The roxy.r script has a new ease-of-use option -f --full regrouping two other options.
The diffoscope maintainers are pleased to announce the release of diffoscope
version 149. This version includes the following changes:
[ Chris Lamb ]
* Update tests for file 5.39. (Closes: reproducible-builds/diffoscope#179)
* Downgrade the tlsh warning message to an "info" level warning.
(Closes: #888237, reproducible-builds/diffoscope#29)
* Use the CSS "word-break" property over manually adding U+200B zero-width
spaces that make copy-pasting cumbersome.
(Closes: reproducible-builds/diffoscope!53)
* Codebase improvements:
- Drop some unused imports from the previous commit.
- Prevent an unnecessary .format() when rendering difference comments.
- Use a semantic "AbstractMissingType" type instead of remembering to check
for both "missing" files and missing containers.
[ Jean-Romain Garnier ]
* Allow user to mask/filter reader output via --diff-mask=REGEX.
(MR: reproducible-builds/diffoscope!51)
* Make --html-dir child pages open in new window to accommodate new web
browser content security policies.
* Fix the --new-file option when comparing directories by merging
DirectoryContainer.compare and Container.compare.
(Closes: reproducible-builds/diffoscope#180)
* Fix zsh completion for --max-page-diff-block-lines.
[ Mattia Rizzolo ]
* Do not warn about missing tlsh during tests.
A very exciting RcppSimdJson release with the updated upstream simdjson release 0.4.0 as well as a first set of new JSON parsing functions just hit CRAN. RcppSimdJson wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire and collaborators. Via very clever algorithmic engineering to obtain largely branch-free code, coupled with modern C++ and newer compiler instructions, it results in parsing gigabytes of JSON parsed per second which is quite mindboggling. The best-case performance is faster than CPU speed as use of parallel SIMD instructions and careful branch avoidance can lead to less than one cpu cycle use per byte parsed; see the video of the recent talk by Daniel Lemire at QCon (which was also voted best talk). The very recent 0.4.0 release further improves the already impressive speed.
And this release brings a first set of actually user-facing functions thanks to Brendan which put in a series of PRs! The full NEWS entry follows.
Changes in version 0.0.6 (2020-06-25)
Created C++ integer-handling utilities for safe downcasting and integer return (Brendan in #16 closing #13).
New JSON functions .deserialize_json and .load_json (Brendan in #16, #17, #20, #21).
Upgrade Travis CI to 'bionic', extract package and version from DESCRIPTION (Dirk in #23).
Upgraded to simdjson 0.4.0 (Dirk in #25 closing #24).
The Bulwark has an interesting article on why they can t Reopen America [1]. I wonder how many changes will be long term. According to the Wikipedia List of Epidemics [2] Covid-19 so far hasn t had a high death toll when compared to other pandemics of the last 100 years. People s reactions to this vary from doing nothing to significant isolation, the question is what changes in attitudes will be significant enough to change society.
Transport
One thing that has been happening recently is a transition in transport. It s obvious that we need to reduce CO2 and while electric cars will address the transport part of the problem in the long term changing to electric public transport is the cheaper and faster way to do it in the short term. Before Covid-19 the peak hour public transport in my city was ridiculously overcrowded, having people unable to board trams due to overcrowding was really common. If the economy returns to it s previous state then I predict less people on public transport, more traffic jams, and many more cars idling and polluting the atmosphere.
Can we have mass public transport that doesn t give a significant disease risk? Maybe if we had significantly more trains and trams and better ventilation with more airflow designed to suck contaminated air out. But that would require significant engineering work to design new trams, trains, and buses as well as expense in refitting or replacing old ones.
Uber and similar companies have been taking over from taxi companies, one major feature of those companies is that the vehicles are not dedicated as taxis. Dedicated taxis could easily be designed to reduce the spread of disease, the famed Black Cab AKA Hackney Carriage [3] design in the UK has a separate compartment for passengers with little air flow to/from the driver compartment. It would be easy to design such taxis to have entirely separate airflow and if setup to only take EFTPOS and credit card payment could avoid all contact between the driver and passengers. I would prefer to have a Hackney Carriage design of vehicle instead of a regular taxi or Uber.
Autonomous cars have been shown to basically work. There are some concerns about safety issues as there are currently corner cases that car computers don t handle as well as people, but of course there are also things computers do better than people. Having an autonomous taxi would be a benefit for anyone who wants to avoid other people. Maybe approval could be rushed through for autonomous cars that are limited to 40Km/h (the maximum collision speed at which a pedestrian is unlikely to die), in central city areas and inner suburbs you aren t likely to drive much faster than that anyway.
Car share services have been becoming popular, for many people they are significantly cheaper than owning a car due to the costs of regular maintenance, insurance, and depreciation. As the full costs of car ownership aren t obvious people may focus on the disease risk and keep buying cars.
Passenger jets are ridiculously cheap. But this relies on the airline companies being able to consistently fill the planes. If they were to add measures to reduce cross contamination between passengers which slightly reduces the capacity of planes then they need to increase ticket prices accordingly which then reduces demand. If passengers are just scared of flying in close proximity and they can t fill planes then they will have to increase prices which again reduces demand and could lead to a death spiral. If in the long term there aren t enough passengers to sustain the current number of planes in service then airline companies will have significant financial problems, planes are expensive assets that are expected to last for a long time, if they can t use them all and can t sell them then airline companies will go bankrupt.
It s not reasonable to expect that the same number of people will be travelling internationally for years (if ever). Due to relying on economies of scale to provide low prices I don t think it s possible to keep prices the same no matter what they do. A new economic balance of flights costing 2-3 times more than we are used to while having significantly less passengers seems likely. Governments need to spend significant amounts of money to improve trains to take over from flights that are cancelled or too expensive.
Entertainment
The article on The Bulwark mentions Las Vegas as a city that will be hurt a lot by reductions in travel and crowds, the same thing will happen to tourist regions all around the world. Australia has a significant tourist industry that will be hurt a lot. But the mention of Las Vegas makes me wonder what will happen to the gambling in general. Will people avoid casinos and play poker with friends and relatives at home? It seems that small stakes poker games among friends will be much less socially damaging than casinos, will this be good for society?
The article also mentions cinemas which have been on the way out since the video rental stores all closed down. There s lots of prime real estate used for cinemas and little potential for them to make enough money to cover the rent. Should we just assume that most uses of cinemas will be replaced by Netflix and other streaming services? What about teenage dates, will kissing in the back rows of cinemas be replaced by Netflix and chill ? What will happen to all the prime real estate used by cinemas?
Professional sporting matches have been played for a TV-only audience during the pandemic. There s no reason that they couldn t make a return to live stadium audiences when there is a vaccine for the disease or the disease has been extinguished by social distancing. But I wonder if some fans will start to appreciate the merits of small groups watching large TVs and not want to go back to stadiums, can this change the typical behaviour of groups?
Restaurants and cafes are going to do really badly. I previously wrote about my experience running an Internet Cafe and why reopening businesses soon is a bad idea [4]. The question is how long this will go for and whether social norms about personal space will change things. If in the long term people expect 25% more space in a cafe or restaurant that s enough to make a significant impact on profitability for many small businesses.
When I was young the standard thing was for people to have dinner at friends homes. Meeting friends for dinner at a restaurant was uncommon. Recently it seemed to be the most common practice for people to meet friends at a restaurant. There are real benefits to meeting at a restaurant in terms of effort and location. Maybe meeting friends at their home for a delivered dinner will become a common compromise, avoiding the effort of cooking while avoiding the extra expense and disease risk of eating out. Food delivery services will do well in the long term, it s one of the few industry segments which might do better after the pandemic than before.
Work
Many companies are discovering the benefits of teleworking, getting it going effectively has required investing in faster Internet connections and hardware for employees. When we have a vaccine the equipment needed for teleworking will still be there and we will have a discussion about whether it should be used on a more routine basis. When employees spend more than 2 hours per day travelling to and from work (which is very common for people who work in major cities) that will obviously limit the amount of time per day that they can spend working. For the more enthusiastic permanent employees there seems to be a benefit to the employer to allow working from home. It s obvious that some portion of the companies that were forced to try teleworking will find it effective enough to continue in some degree.
One company that I work for has quit their coworking space in part because they were concerned that the coworking company might go bankrupt due to the pandemic. They seem to have become a 100% work from home company for the office part of the work (only on site installation and stock management is done at corporate locations). Companies running coworking spaces and other shared offices will suffer first as their clients have short term leases. But all companies renting out office space in major cities will suffer due to teleworking. I wonder how this will affect the companies providing services to the office workers, the cafes and restaurants etc. Will there end up being so much unused space in central city areas that it s not worth converting the city cinemas into useful space?
There s been a lot of news about Zoom and similar technologies. Lots of other companies are trying to get into that business. One thing that isn t getting much notice is remote access technologies for desktop support. If the IT people can t visit your desk because you are working from home then they need to be able to remotely access it to fix things. When people make working from home a large part of their work time the issue of who owns peripherals and how they are tracked will get interesting. In a previous blog post I suggested that keyboards and mice not be treated as assets [5]. But what about monitors, 4G/Wifi access points, etc?
Some people have suggested that there will be business sectors benefiting from the pandemic, such as telecoms and e-commerce. If you have a bunch of people forced to stay home who aren t broke (IE a large portion of the middle class in Australia) they will probably order delivery of stuff for entertainment. But in the long term e-commerce seems unlikely to change much, people will spend less due to economic uncertainty so while they may shift some purchasing to e-commerce apart from home delivery of groceries e-commerce probably won t go up overall. Generally telecoms won t gain anything from teleworking, the Internet access you need for good Netflix viewing is generally greater than that needed for good video-conferencing.
Money
I previously wrote about a Basic Income for Australia [6]. One of the most cited reasons for a Basic Income is to deal with robots replacing people. Now we are at the start of what could be a long term economic contraction caused by the pandemic which could reduce the scale of the economy by a similar degree while also improving the economic case for a robotic workforce. We should implement a Universal Basic Income now.
I previously wrote about the make-work jobs and how we could optimise society to achieve the worthwhile things with less work [7]. My ideas about optimising public transport and using more car share services may not work so well after the pandemic, but the rest should work well.
Business
There are a number of big companies that are not aiming for profitability in the short term. WeWork and Uber are well documented examples. Some of those companies will hopefully go bankrupt and make room for more responsible companies.
The co-working thing was always a precarious business. The companies renting out office space usually did so on a monthly basis as flexibility was one of their selling points, but they presumably rented buildings on an annual basis. As the profit margins weren t particularly high having to pay rent on mostly empty buildings for a few months will hurt them badly. The long term trend in co-working spaces might be some sort of collaborative arrangement between the people who run them and the landlords similar to the way some of the hotel chains have profit sharing agreements with land owners to avoid both the capital outlay for buying land and the risk involved in renting. Also city hotels are very well equipped to run office space, they have the staff and the procedures for running such a business, most hotels also make significant profits from conventions and conferences.
The way the economy has been working in first world countries has been about being as competitive as possible. Just in time delivery to avoid using storage space and machines to package things in exactly the way that customers need and no more machines than needed for regular capacity. This means that there s no spare capacity when things go wrong. A few years ago a company making bolts for the car industry went bankrupt because the car companies forced the prices down, then car manufacture stopped due to lack of bolts this could have been a wake up call but was ignored. Now we have had problems with toilet paper shortages due to it being packaged in wholesale quantities for offices and schools not retail quantities for home use. Food was destroyed because it was created for restaurant packaging and couldn t be packaged for home use in a reasonable amount of time.
Farmer s markets alleviate some of the problems with packaging food etc. But they aren t a good option when there s a pandemic as disease risk makes them less appealing to customers and therefore less profitable for vendors.
Religion
Many religious groups have supported social distancing. Could this be the start of more decentralised religion? Maybe have people read the holy book of their religion and pray at home instead of being programmed at church? We can always hope.
I have been convinced by the arguments that it's not nice to keep using the word master for the default git branch. Regardless of the etymology (which is unclear), some people say they have negative associations for this word, Changing this upstream in git is complicated on a technical level and, sadly, contested.
But git is flexible enough that I can make this change in my own repositories. Doing so is not even so difficult.
So:
Announcement
I intend to rename master to trunk in all repositories owned by my personal hat. To avoid making things very complicated for myself I will just delete refs/heads/master when I make this change. So there may be a little disruption to downstreams.
I intend make this change everywhere eventually. But rather than front-loading the effort, I'm going to do this to repositories as I come across them anyway. That will allow me to update all the docs references, any automation, etc., at a point when I have those things in mind anyway. Also, doing it this way will allow me to focus my effort on the most active projects, and avoids me committing to a sudden large pile of fiddly clerical work. But: if you have an interest in any repository in particular that you want updated, please let me know so I can prioritise it.
Bikeshed
Why "trunk"? "Main" has been suggested elswewhere, and it is often a good replacement for "master" (for example, we can talk very sensibly about a disk's Main Boot Record, MBR). But "main" isn't quite right for the VCS case; for example a "main" branch ought to have better quality than is typical for the primary development branch.
Conversely, there is much precedent for "trunk". "Trunk" was used to refer to this concept by at least SVN, CVS, RCS and CSSC (and therefore probably SCCS) - at least in the documentation, although in some of these cases the command line API didn't have a name for it.
So "trunk" it is.
Aside: two other words - passlist, blocklist
People are (finally!) starting to replace "blacklist" and "whitelist". Seriously, why has it taken everyone this long?
I have been using "blocklist" and "passlist" for these concepts for some time. They are drop-in replacements. I have also heard "allowlist" and "denylist" suggested, but they are cumbersome and cacophonous.
Also "allow" and "deny" seem to more strongly imply an access control function than merely "pass" and "block", and the usefulness of passlists and blocklists extends well beyond access control: protocol compatibility and ABI filtering are a couple of other use cases.
Welcome to the 27th post in the rationally regularized R revelations series, or R4 for short. This is a edited / updated version of yesterday s T^4 post #7 as it really fits the R4 series as well as it fits the T4 series.
A new video in both our T^4 series of video lightning talks with tips, tricks, tools, and toys is also a video in the R^4 series as it revisits a topic previously covered in the latter: how to (more easily) get (binary) packages onto your Ubuntu system. In fact, we show it in three different ways.
The slides are here.
This repo at GitHub support the series: use it to open issues for comments, criticism, suggestions, or feedback.
Thanks to I aki Ucar who followed up on twitter with a Fedora version. We exchanged some more message, and concluded that complete comparison (from an empty Ubuntu or Fedora container) to a full system takes about the same time on either system.
If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.
Italianization (Italian: Italianizzazione; Croatian: talijanizacija; Slovene: poitaljan evanje; German: Italianisierung; Greek: ) is the spread of Italian culture and language, either by integration or assimilation.[1][2]
In 1919, at the time of its annexation, the middle part of the County of Tyrol which is today called South Tyrol (in Italian Alto Adige) was inhabited by almost 90% German speakers.[1] Under the 1939 South Tyrol Option Agreement, Adolf Hitler and Benito Mussolini determined the status of the German and Ladin (Rhaeto-Romanic) ethnic groups living in the region. They could emigrate to Germany, or stay in Italy and accept their complete Italianization. As a consequence of this, the society of South Tyrol was deeply riven. Those who wanted to stay, the so-called Dableiber, were condemned as traitors while those who left (Optanten) were defamed as Nazis. Because of the outbreak of World War II, this agreement was never fully implemented. Illegal Katakombenschulen ("Catacomb schools") were set up to teach children the German language.
The Prontuario dei nomi locali dell'Alto Adige (Italian for Reference Work of Place Names of Alto Adige) is a list of Italianized toponyms for mostly German place names in South Tyrol (Alto Adige in Italian) which was published in 1916 by the Royal Italian Geographic Society (Reale Societ Geografica Italiana). The list was called the Prontuario in short and later formed an important part of the Italianization campaign initiated by the fascist regime, as it became the basis for the official place and district names in the Italian-annexed southern part of the County of Tyrol.
Ettore Tolomei (16 August 1865, in Rovereto 25 May 1952, in Rome) was an Italian nationalist and fascist. He was designated a Member of the Italian Senate in 1923, and ennobled as Conte della Vetta in 1937.
The South Tyrol Option Agreement (German: Option in S dtirol; Italian: Opzioni in Alto Adige) was an agreement in effect between 1939 and 1943, when the native German speaking people in South Tyrol and three communes in the province of Belluno were given the option of either emigrating to neighboring Nazi Germany (of which Austria was a part after the 1938 Anschluss) or remaining in Fascist Italy and being forcibly integrated into the mainstream Italian culture, losing their language and cultural heritage. Over 80% opted to move to Germany.
A new video in both our T^4 series of video lightning talks with tips, tricks, tools, and toys is also a video in the R^4 series as it revisits a topic previously covered in the latter: how to (more easily) get (binary) packages onto your Ubuntu system. In fact, we show it in three different ways.
Limesurvey is an online survey tool. It is very powerful and commonly used in academic environments because it is Free Software (GPLv2+), allows for local installations protecting the data of participants and allowing to comply with data protection regulations. This also means there are typically no load-balanced multi-server szenarios with HA databases. But simple VMs where Limesurvey runs and needs upgrading in place.
There's an LTS branch (currently 3.x) and a stable branch (currently 4.x). There's also a 2.06 LTS branch that is restricted to paying customers. The main developers behind Limesurvey offer many services from template design to custom development to support to hosting ("Cloud", "Limesurvey Pro"). Unfortunately they also charge for easy updates called "ComfortUpdate" (currently 39 for three months) and the manual process is made a bit cumbersome to make the "ComfortUpdate" offer more attractive.
Due to Limesurvey being an old code base and UI elements not being clearly separated, most serious use cases will end up patching files and symlinking logos around template directories. That conflicts a bit with the opaque "ComfortUpdate" process where you push a button and then magic happens. Or you have downtime and a recovery case while surveys are running.
If you do not intend to use the "ComfortUpdate" offering, you can prevent Limesurvey from connecting to http://comfortupdate.limesurvey.org daily by adding the updatable stanza as in line 14 to limesurvey/application/config/config.php:
// debug: Set this to 1 if you are looking for errors. If you still get no errors after enabling this
// then please check your error-logs - either in your hosting provider admin panel or in some /logs directory
// on your webspace.
// LimeSurvey developers: Set this to 2 to additionally display STRICT PHP error messages and get full access to standard templates
'debug'=>0,
'debugsql'=>0,// Set this to 1 to enanble sql logging, only active when debug = 2
// Mysql database engine (INNODB MYISAM):
'mysqlEngine'=>'MYISAM'
,// Update default LimeSurvey config here
'updatable'=>false,
)
);
The comma on line 13 is placed like that in the current default limesurvey config.php, don't let yourself get confused.
Every item in a php array must end with a comma. It can be on the next line.
The basic principle of low risk, near-zero downtime, in-place upgrades is:
Create a diff between the current release and the target release
Inspect the diff
Make backups of the application webroot
Patch a copy of the application in-place
(optional) stop the web server
Make a backup of the production database
Move the patched application to the production webroot
Release 0.3.8 of RcppGSL is now getting onto CRAN. The RcppGSL package provides an interface from R to the GNU GSL using the Rcpp package.
Peter Carbonetto let us know in issue #25 that the included example now showed linker errors on (everybody s favourite CRAN platform) Slowlaris. Kidding aside, the added compiler variety really has benefits because we were indeed missing a good handful or two of inline statements in the headers which our good friends g++ and clang++ apparently let us get away with. This has been fixed, and a little bit of the usual package polish and cleanup has been added; see the list of detailed changes below.
Changes in version 0.3.8 (2020-06-21)
A few missing inline statements were added to the headers fixing a (genuine) error that was seen only on Solaris (Dirk).
The nice colNorm example is now in a file by itself, the previous versions are off in a new file colNorm_old.cpp (Dirk).
The README.me now sports two new badges (Dirk).
Travis CI was updated to 'bionic' and R 4.0 (Dirk).
In considering storage trends for the consumer side I m looking at the current prices from MSY (where I usually buy computer parts). I know that other stores will have slightly different prices but they should be very similar as they all have low margins and wholesale prices are the main factor.
Small Hard Drives Aren t Viable
The cheapest hard drive that MSY sells is $68 for 500G of storage. The cheapest SSD is $49 for 120G and the second cheapest is $59 for 240G. SSD is cheaper at the low end and significantly faster. If someone needed about 500G of storage there s a 480G SSD for $97 which costs $29 more than a hard drive. With a modern PC if you have no hard drives you will notice that it s quieter. For anyone who s buying a new PC spending an extra $29 is definitely worthwhile for the performance, low power use, and silence.
The cheapest 1TB disk is $69 and the cheapest 1TB SSD is $159. Saving $90 on the cost of a new PC probably isn t worth while.
For 2TB of storage the cheapest options are Samsung NVMe for $339, Crucial SSD for $335, or a hard drive for $95. Some people would choose to save $244 by getting a hard drive instead of NVMe, but if you are getting a whole system then allocating $244 to NVMe instead of a faster CPU would probably give more benefits overall.
Computer stores typically have small margins and computer parts tend to quickly either become cheaper or be obsoleted by better parts. So stores don t want to stock parts unless they will sell quickly. Disks smaller than 2TB probably aren t going to be profitable for stores for very long. The trend of SSD and NVMe becoming cheaper is going to make 2TB disks non-viable in the near future.
NVMe vs SSD
M.2 NVMe devices are at comparable prices to SATA SSDs. For some combinations of quality and capacity NVMe is about 50% more expensive and for some it s slightly cheaper (EG Intel 1TB NVMe being cheaper than Samsung EVO 1TB SSD). Last time I checked about half the motherboards on sale had a single M.2 socket so for a new workstation that doesn t need more than 2TB of storage (the largest NVMe that MSY sells) it wouldn t make sense to use anything other than NVMe.
The benefit of NVMe is NOT throughput (even though NVMe devices can often sustain over 4GB/s), it s low latency. Workstations can t properly take advantage of this because RAM is so cheap ($198 for 32G of DDR4) that compiles etc mostly come from cache and because most filesystem writes on workstations aren t synchronous. For servers a large portion of writes are synchronous, for example a mail server can t acknowledge receiving mail until it knows that it s really on disk, so there s a lot of small writes that block server processes and the low latency of NVMe really improves performance. If you are doing a big compile on a workstation (the most common workstation task that uses a lot of disk IO) then the writes aren t synchronised to disk and if the system crashes you will just do all the compilation again. While NVMe doesn t give a lot of benefit over SSD for workstation use (I ve uses laptops with SSD and NVMe and not noticed a great difference) of course I still want better performance. ;)
Last time I checked I couldn t easily buy a PCIe card that supported 2*NVMe cards, I m sure they are available somewhere but it would take longer to get and probably cost significantly more than twice as much. That means a RAID-1 of NVMe takes 2 PCIe slots if you don t have an M.2 socket on the motherboard. This was OK when I installed 2*NVMe devices on a server that had 18 disks and lots of spare PCIe slots. But for some systems PCIe slots are an issue.
My home server has all PCIe slots used by a video card and Ethernet cards and the BIOS probably won t support booting from NVMe. It s a Dell server so I can t just replace the motherboard with one that has more PCIe slots and M.2 on the motherboard. As it s running nicely and doesn t need replacing any time soon I won t be using NVMe for home server stuff.
Small Servers
Most servers that I am responsible for have less than 2TB of storage. For my clients I now only recommend SSD storage for small servers and am recommending SSD for replacing any failed disks.
My home server has 2*500G SSDs in a BTRFS RAID-1 for the root filesystem, and 3*4TB disks in a BTRFS RAID-1 for storing big files. I bought the SSDs when 500G SSDs were about $250 each and bought 2*4TB disks when they were about $350 each. Currently that server has about 3.3TB of space used and I could probably get it down to about 2.5TB if I deleted things I don t really need. If I was getting storage for that server now I d use 2*2TB SSDs and 3*1TB hard drives for the stuff that doesn t fit on SSDs (I have some spare 1TB disks that came with servers). If I didn t have spare hard drives I d get 3*2TB SSDs for that sort of server which would give 3TB of BTRFS RAID-1 storage.
Last time I checked Dell servers had a card for supporting M.2 as an optional extra so Dells probably won t boot from NVMe without extra expense.
Ars Technica has an informative article about WD selling SMR disks as NAS disks [1]. The Shingled Magnetic Recording technology allows greater storage density on a platter which leads to either larger capacity or cheaper disks but at the cost of lower write performance and apparently extremely bad latency in some situations. NAS disks are supposed to be low latency as the expectation is that they will be used in a RAID array and kicked out of the array if they have problems. There are reports of ZFS kicking SMR disks from RAID sets. I think this will end the use of hard drives for small servers. For a server you don t want to deal with this sort of thing, by definition when a server goes down multiple people will stop work (small server implies no clustering). Spending extra to get SSDs just to avoid the risk of unexpected SMR would be a good plan.
Medium Servers
The largest SSD and NVMe devices that are readily available are 2TB but 10TB disks are commodity items, there are reports of 20TB hard drives being available but I can t find anyone in Australia selling them.
If you need to store dozens or hundreds of terabytes than hard drives have to be part of the mix at this time. There s no technical reason why SSDs larger than 10TB can t be made (the 2.5 SATA form factor has more than 5* the volume of a 2TB M.2 card) and it s likely that someone sells them outside the channels I buy from, but probably at a price higher than what my clients are willing to pay. If you want 100TB of affordable storage then a mid range server like the Dell PowerEdge T640 which can have up to 18*3.5 disks is good. One of my clients has a PowerEdge T630 with 18*3.5 disks in the 8TB-10TB range (we replace failed disks with the largest new commodity disks available, it used to have 6TB disks). ZFS version 0.8 introduced a Special VDEV Class which stores metadata and possibly small data blocks on faster media. So you could have some RAID-Z groups on hard drives for large storage and the metadata on a RAID-1 on NVMe for fast performance. For medium size arrays on hard drives having a find / operation take hours is not uncommon, for large arrays having it take days isn t that uncommon. So far it seems that ZFS is the only filesystem to have taken the obvious step of storing metadata on SSD/NVMe while bulk data is on cheap large disks.
One problem with large arrays is that the vibration of disks can affect the performance and reliability of nearby disks. The ZFS server I run with 18 disks was originally setup with disks from smaller servers that never had ZFS checksum errors, but when disks from 2 small servers were put in one medium size server they started getting checksum errors presumably due to vibration. This alone is a sufficient reason for paying a premium for SSD storage.
Currently the cost of 2TB of SSD or NVMe is between the prices of 6TB and 8TB hard drives, and the ratio of price/capacity for SSD and NVMe is improving dramatically while the increase in hard drive capacity is slow. 4TB SSDs are available for $895 compared to a 10TB hard drive for $549, so it s 4* more expensive on a price per TB. This is probably good for Windows systems, but for Linux systems where ZFS and special VDEVs is an option it s probably not worth considering. Most Linux user cases where 4TB SSDs would work well would be better served by smaller NVMe and 10TB disks running ZFS. I don t think that 4TB SSDs are at all popular at the moment (MSY doesn t stock them), but prices will come down and they will become common soon enough. Probably by the end of the year SSDs will halve in price and no hard drives less than 4TB will be viable.
For rack mounted servers 2.5 disks have been popular for a long time. It s common for vendors to offer 2 versions of a rack mount server for 2.5 and 3.5 disks where the 2.5 version takes twice as many disks. If the issue is total storage in a server 4TB SSDs can give the same capacity as 8TB HDDs.
SMR vs Regular Hard Drives
Rumour has it that you can buy 20TB SMR disks, I haven t been able to find a reference to anyone who s selling them in Australia (please comment if you know who sells them and especially if you know the price). I expect that the ZFS developers will soon develop a work-around to solve the problems with SMR disks. Then arrays of 20TB SMR disks with NVMe for special VDEVs will be an interesting possibility for storage. I expect that SMR disks will be the majority of the hard drive market by 2023 if hard drives are still on the market. SSDs will be large enough and cheap enough that only SMR disks will offer enough capacity to be worth using.
I think that it is a possibility that hard drives won t be manufactured in a few years. The volume of a 3.5 disk is significantly greater than that of 10 M.2 devices so current technology obviously allows 20TB of NVMe or SSD storage in the space of a 3.5 disk. If the price of 16TB NVMe and SSD devices comes down enough (to perhaps 3* the price of a 20TB hard drive) almost no-one would want the hard drive and it wouldn t be viable to manufacture them.
It s not impossible that in a few years time 3D XPoint and similar fast NVM technologies occupy the first level of storage (the ZFS special VDEV , OS swap device, log device for database servers, etc) and NVMe occupies the level for bulk storage with no space left in the market for spinning media.
Computer Cases
For servers I expect that models supporting 3.5 storage devices will disappear. A 1RU server with 8*2.5 storage devices or a 2RU server with 16*2.5 storage devices will probably be of use to more people than a 1RU server with 4*3.5 or a 2RU server with 8*3.5 .
My first IBM PC compatible system had a 5.25 hard drive, a 5.25 floppy drive, and a 3.5 floppy drive in 1988. My current PC is almost a similar size and has a DVD drive (that I almost never use) 5 other 5.25 drive bays that have never been used, and 5*3.5 drive bays that I have never used (I have only used 2.5 SSDs). It would make more sense to have PC cases designed around 2.5 and maybe 3.5 drives with no more than one 5.25 drive bay.
The Intel NUC SFF PCs are going in the right direction. Many of them only have a single storage device but some of them have 2*M.2 sockets allowing RAID-1 of NVMe and some of them support ECC RAM so they could be used as small servers.
A USB DVD drive costs $36, it doesn t make sense to have every PC designed around the size of an internal DVD drive that will probably only be used to install the OS when a $36 USB DVD drive can be used for every PC you own.
The only reason I don t have a NUC for my personal workstation is that I get my workstations from e-waste. If I was going to pay for a PC then a NUC is the sort of thing I d pay to have on my desk.
The diffoscope maintainers are pleased to announce the release of diffoscope
version 148. This version includes the following changes:
[ Daniel Fullmer ]
* Fix a regression in the CBFS comparator due to changes in our_check_output.
[ Chris Lamb ]
* Add a remark in the deb822 handling re. potential security issue in the
.changes, .dsc, .buildinfo comparator.
LWN published today yet another great piece of writing, Loaded terms
in free software. I am sorry, the
content will not be immediately available to anybody following at
home, as LWN is based on a subscription model But a week from now,
the article will be open for anybody to read. Or you can ask me (you
most likely can find my contact addresses, as they are basically
everywhere) for a subscriber link, I will happily provide it.
In consonance with the current mood that started with the killing of
George Floyd
and sparked worldwide revolts against police brutality, racism (mostly
related to police and law enforcement forces, but social as well) and
the like, the debate that already started some months ago in technical
communities has re-sparked:
We have many terms that come with long histories attached to them, and
we are usually oblivious to their obvious meaning. We? Yes, we, the
main users and creators of technology. I never felt using master and
slave to refer to different points of a protocol, bus, clock or
whatever (do refer to the Wikipedia article for a fuller
explanation)
had any negative connotations but then again, those terms have never
tainted my personal family. That is, I understand I speak from a
position of privilege.
A similar although less heated issue goes around the blacklist and
whitelist terms, or other uses that use white to refer to good,
law-abiding citizens, and black to refer to somewhat antisocial uses
(i.e. the white hat and black hat hackers).
For several years, this debate has been sparking and dying off. Some
important changes have been made Particularly, in 2017 the Internet
Software Consortium started recommending Primary and
Secondary,
Python dropped master/slave pairs after a quite thorough and deep
review throughout 2018, GitHub
changed the default branch from master to main earlier this
week. The
Internet Engineering Task Force has a draft (that lapsed and thus
sadly didn t become an RFC, but still, is archived), Terminology,
Power and Oppressive
Language
that lists suggested alternatives:
There are also many other relationships that can be used as
metaphors, Eglash s research calls into question the accuracy of the
master-slave metaphor. Fortunately, there are ample alternatives for
the master-slave relationship. Several options are suggested here
and should be chosen based on the pairing that is most clear in
context:
Primary-secondary
Leader-follower
Active-standby
Primary-replica
Writer-reader
Coordinator-worker
Parent-helper
I ll add that I think we Spanish-speakers are not fully aware of the
issue s importance, because the most common translation I have seen
for master/slave is maestro/esclavo: Maestro is the word for
teacher (although we do keep our slaves in place). But think
whether it sounds any worse if you refer to device pairs, or members
of a database high-availability cluster, or whatever as Amo and
Esclavo. It does sound much worse
I cannot add much of value to this debate. I am just happy issues like
this are being recognized and dealt with. If the topic interests you,
do refer to the LWN article! Some excrepts:
I consider the following to be the core of Jonathan Corbet s writeup:
Recent events, though, have made it clear even to those of us who
were happy to not question this view that the story of slavery and
the wider racist systems around it is not yet finished. There are
many people who are still living in the middle of it, and it is not
a nice place to be. We are not so enlightened as we like to think we
are.
If there is no other lesson from the events of the last few weeks,
we should certainly take to heart the point that we need to be
listening to the people who have been saying, for many years, that
they are still suffering. If there are people who are telling us
that terms like slave or blacklist are a hurtful reminder of the
inequities that persist in our society, we need to accept that as
the truth and act upon it. Etymological discussions on what, say,
master really means may be interesting, but they miss the point
and are irrelevant to this discussion.
Part of a comment by user yokem_55:
Often, it seems to me that the replacement words are much more
descriptive and precise than the old language. Allowlist is far
more obviously a list of explicitly authorized entities than
whitelist . Mainline has a more obvious meaning of a core stream
of development than master .
The benefit of moving past this language is more than just changing
cultural norms, it s better, more precise communication across the
board.
Another spot-on comment, by user alan:
From my perspective as a Black American male, I think that it s nice
to see people willing to see and address racism in various
spheres. I am concerned that some of these steps will be more
performative than substantial. Terminology changes in software so as
to be more welcoming is a nice thing. Ensuring that oppressed
minorities have access to the tools and resources to help reduce
inequity and ensuring equal protection under the laws is
better. We ll get there one day I m sure. The current ask is much
simpler, its just to stop randomly killing and terrorizing
us. Please and thank you.
So Maybe the protests of this year caught special notoriety because
the society is reacting after (or during, for many of us) the
lockdown. In any case, I hope for their success in changing the
planet s culture of oppression.
Comments
Tomas Janousek 2020-06-19 10:04:32 +0200
In the blog post On masters and slaves, whitelists and blacklists
you claim that GitHub changed the default branch from master to main
earlier this week but I don t think that change is in effect
yet. When you create a repo, the default branch is still named
master .
Gunnar Wolf 2020-06-19 11:52:30 -0500
Umh, seems you are right. Well, what can I say? I m reporting only
what I have been able to find / read
Now, given that said master branch does not carry any Git-specific
meaning and is just a commonly used configuration I hope people
start picking it up.
No, I have not renamed master branches in any of my repos but
intend to do so soonish.
Tomas Janousek 2020-06-19 20:01:52 +0200
Yeah, don t worry. I just find it sad that so much inaccurate news is
spreading from a single CEO tweet, and I wanted to help stop that. I m
sure some change will happen eventually, but until it does, we
shouldn t speak about it in the past tense. :-)
Setting up a Sony PS3 Blu-Ray Disc Remote Controller with Kodi
TLDR; Since most of the articles on the internet were either obsolete or broken, I ve chosen to write these notes down in the form of a blog post so that it helps me now and in future, and hopefully others too.
Raspberry Pi
All this time, I have been using the Raspberry Pi for my HTPC needs. The first RPi I acquired was in 2014 and I have been very very happy with the amount of support in the community and quality of the HTPC offering it has. I also appreciate the RPi s form factor and the power consumption limits. And then, to add more sugar to it, it uses a derivative of Debian, Raspbian, which was very familiar and feel good to me.
Raspberry Pi Issues
So primarily, I use my RPi with Kodi. There are a bunch of other (daemon) services but the primary use case is HTPC only. RPi + Kodi has a very very annoying issue wherein it loses its audio pitch during video playback. The loss is so bad that the audio is barely audible. The workaround is to seek the video playback either way and then it comes back to its actual audio level, just to fade again in a while.
My suspicion was that it may be a problem with Kodi. Or at least, Kodi would have a workaround in software. But unfortunately, I wasted a lot of time in dealing with my suspicion with no fruitful result.
This started becoming a PITA over time.
And it seems the issue is with the hardware itself because after I moved my setup to a regular laptop, the audio loss is gone.
Laptop with Kodi
Since I had my old Lenovo Yoga 2 13 lying on all the time, it made sense to make some more use of it, using as the HTPC. This machine comes with a Micro-HDMI Out port, so it felt ideal for my High Definition video rendering needs.
It comes stock with just Intel HD Video with good driver support in Linux, so it was quite quick and easy getting Kodi charged up and running on it. And as I mentioned above, the sound issues are not seen on this setup.
Some added benefits are that I get to run stock Debian on this machine. And I must say a big THANK YOU to the Debian Multimedia Maintainers, who ve done a pretty good job maintaining Kodi under Debian.
HDMI CEC
Only after I decommissioned my RPi, I came to notice how convenient the HDMI CEC functionality is. Turns out no standard laptops ship CEC functionality onto them. Even the case of my laptop, which has a Micro HDMI Out port, but still no CEC capabilities. As far as I know, the RPi came with the Pulse-Eight CEC module, so obvious first thought was to opt for a compatible external module of the same; but it comes with a nice price tag, me not willing to spend.
WiFi Remotes
Kodi has very well implemented network interface for almost all its features. One could take the Yatse or Music Pump Kodi Remote Android applications that work very very well with Kodi.
But wifi can be flaky some times. Especially, my experience with the Realtek network devices hasn t been very good. The driver support in Linux is okay but there are many firmware bugs to deal with. In my case, the machine will lose wifi signal/network every once in a while. And it turns out, for this machine, with this network device type, I m not the only one running into such problems.
And to add to that, this is an UltraBook, which means it doesn t have an Ethernet port. So I ve had not much choice other than to live and suffer deal with it.
The WiFi chip also provides the Bluetooth module, which so far I had not used much. From my /etc/modprobe.d/blacklist-memstick.conf, all relevant BT modules were added to the blacklist, all this time.
rrs@lenovo:~$ cat /etc/modprobe.d/blacklist-memstick.conf
blacklist memstick
blacklist rtsx_usb_ms
# And bluetooth too
#blacklist btusb
#blacklist btrtl
#blacklist btbcm
#blacklist btintel
#blacklist bluetooth
21:21
Also to keep in mind is that the driver for my card gives a very misleading kernel message, which is one of the many reasons for this blog post, so that I don t forget it a couple of months later. The missing firmware error message is okay to ignore, as per this upstream comment.
Jun 14 17:17:08 lenovo kernel: usbcore: registered new interface driver btusb
Jun 14 17:17:08 lenovo systemd[1]: Mounted /boot/efi.
Jun 14 17:17:08 lenovo kernel: Bluetooth: hci0: RTL: examining hci_ver=06 hci_rev=000b lmp_ver=06 lmp_subver=8723
Jun 14 17:17:08 lenovo kernel: Bluetooth: hci0: RTL: rom_version status=0 version=1
Jun 14 17:17:08 lenovo kernel: Bluetooth: hci0: RTL: loading rtl_bt/rtl8723b_fw.bin
Jun 14 17:17:08 lenovo kernel: bluetooth hci0: firmware: direct-loading firmware rtl_bt/rtl8723b_fw.bin
Jun 14 17:17:08 lenovo kernel: Bluetooth: hci0: RTL: loading rtl_bt/rtl8723b_config.bin
Jun 14 17:17:08 lenovo kernel: bluetooth hci0: firmware: failed to load rtl_bt/rtl8723b_config.bin (-2)
Jun 14 17:17:08 lenovo kernel: firmware_class: See https://wiki.debian.org/Firmware for information about missing firmware
Jun 14 17:17:08 lenovo kernel: bluetooth hci0: Direct firmware load for rtl_bt/rtl8723b_config.bin failed with error -2
Jun 14 17:17:08 lenovo kernel: Bluetooth: hci0: RTL: cfg_sz -2, total sz 22496
And then, when the btusb module is initialed (along with the misleading driver message), you ll get the following in your USB device listing
Bus 002 Device 005: ID 0bda:b728 Realtek Semiconductor Corp. Bluetooth Radio
Sony PlayStation 3 BD Remote
Almost 10 years ago, I bought the PS3 and many of its accessories. The remote has just been rotting in the shelf. It had rusted so bad that it is better described with these pics.
Rusted inside
Rusted inside and cover
Rusted spring
The rust was so much that the battery holding spring gave up.
A little bit scrubbing and cleaning has gotten it working. I hope it lasts for some time before I find time to open it up and give it a full clean-up.
Pairing the BD Remote to laptop
Honestly, with the condition of the hardware and software on both ends, I did not have much hopes of getting this to work. And in all the years on my computer usage, I hardly recollect much days when I ve made use of BT. Probably, because the full BT stack wasn t that well integrated in Linux, earlier. And I mostly used to disable them in hardware and software to save on battery.
All yielded results from the internet talked about tools/scripts that were either not working, pointing to broken links etc.
These days, bluez comes with a nice utility, bluetoothctl. It was a nice experience using it.
First, start your bluetooth service and ensure that the device talks well with the kernel
rrs@lenovo:~$ systemctl status bluetooth
bluetooth.service - Bluetooth service
Loaded: loaded (/lib/systemd/system/bluetooth.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-06-15 12:54:58 IST; 3s ago
Docs: man:bluetoothd(8)
Main PID: 310197 (bluetoothd)
Status: "Running"
Tasks: 1 (limit: 9424)
Memory: 1.3M
CGroup: /system.slice/bluetooth.service
310197 /usr/lib/bluetooth/bluetoothd
Jun 15 12:54:58 lenovo systemd[1]: Starting Bluetooth service...
Jun 15 12:54:58 lenovo bluetoothd[310197]: Bluetooth daemon 5.50
Jun 15 12:54:58 lenovo systemd[1]: Started Bluetooth service.
Jun 15 12:54:58 lenovo bluetoothd[310197]: Starting SDP server
Jun 15 12:54:58 lenovo bluetoothd[310197]: Bluetooth management interface 1.15 initialized
Jun 15 12:54:58 lenovo bluetoothd[310197]: Sap driver initialization failed.
Jun 15 12:54:58 lenovo bluetoothd[310197]: sap-server: Operation not permitted (1)
12:55
Next, then is to discover and connect to your device
rrs@lenovo:~$ bluetoothctl
Agent registered
[bluetooth]# devices
Device E6:3A:32:A4:31:8F MI Band 2
Device D4:B8:FF:43:AB:47 MI RC
Device 00:1E:3D:10:29:0F BD Remote Control
[CHG] Device 00:1E:3D:10:29:0F Connected: yes
[BD Remote Control]# info 00:1E:3D:10:29:0F
Device 00:1E:3D:10:29:0F (public)
Name: BD Remote Control
Alias: BD Remote Control
Class: 0x0000250c
Paired: no
Trusted: yes
Blocked: no
Connected: yes
LegacyPairing: no
UUID: Human Interface Device... (00001124-0000-1000-8000-00805f9b34fb)
UUID: PnP Information (00001200-0000-1000-8000-00805f9b34fb)
Modalias: usb:v054Cp0306d0100
[bluetooth]#
In case of the Sony BD Remote, there s no need to pair. In fact, trying to pair fails. It prompts for the PIN code, but neither 0000 or 1234 are accepted.
So, the working steps so far are to Trust the device and then Connect the device.
For the sake of future use, I also populated /etc/bluetooth/input.conf based on suggestions on the internet.
Note: The advertised keymappings in this config file do not work.
Note: I m only using it for the power saving measures in instructing the BT connection to sleep after 3 minutes.
rrs@priyasi:/tmp$ cat input.conf
# Configuration file for the input service
# This section contains options which are not specific to any
# particular interface
[General]
# Set idle timeout (in minutes) before the connection will
# be disconnect (defaults to 0 for no timeout)
IdleTimeout=3
# Enable HID protocol handling in userspace input profile
# Defaults to false (HIDP handled in HIDP kernel module)
#UserspaceHID=true
# Limit HID connections to bonded devices
# The HID Profile does not specify that devices must be bonded, however some
# platforms may want to make sure that input connections only come from bonded
# device connections. Several older mice have been known for not supporting
# pairing/encryption.
# Defaults to false to maximize device compatibility.
#ClassicBondedOnly=true
# LE upgrade security
# Enables upgrades of security automatically if required.
# Defaults to true to maximize device compatibility.
#LEAutoSecurity=true
#
#[00:1E:3D:10:29:0F]
[2c:33:7a:8e:d6:30]
[PS3 Remote Map]
# When the 'OverlayBuiltin' option is TRUE (the default), the keymap uses
# the built-in keymap as a starting point. When FALSE, an empty keymap is
# the starting point.
#OverlayBuiltin = TRUE
#buttoncode = keypress # Button label = action with default key mappings
#OverlayBuiltin = FALSE
0x16 = KEY_ESC # EJECT = exit
0x64 = KEY_MINUS # AUDIO = cycle audio tracks
0x65 = KEY_W # ANGLE = cycle zoom mode
0x63 = KEY_T # SUBTITLE = toggle subtitles
0x0f = KEY_DELETE # CLEAR = delete key
0x28 = KEY_F8 # /TIME = toggle through sleep
0x00 = KEY_1 # NUM-1
0x01 = KEY_2 # NUM-2
0x02 = KEY_3 # NUM-3
0x03 = KEY_4 # NUM-4
0x04 = KEY_5 # NUM-5
0x05 = KEY_6 # NUM-6
0x06 = KEY_7 # NUM-7
0x07 = KEY_8 # NUM-8
0x08 = KEY_9 # NUM-9
0x09 = KEY_0 # NUM-0
0x81 = KEY_F2 # RED = red
0x82 = KEY_F3 # GREEN = green
0x80 = KEY_F4 # BLUE = blue
0x83 = KEY_F5 # YELLOW = yellow
0x70 = KEY_I # DISPLAY = show information
0x1a = KEY_S # TOP MENU = show guide
0x40 = KEY_M # POP UP/MENU = menu
0x0e = KEY_ESC # RETURN = back/escape/cancel
0x5c = KEY_R # TRIANGLE/OPTIONS = cycle through recording options
0x5d = KEY_ESC # CIRCLE/BACK = back/escape/cancel
0x5f = KEY_A # SQUARE/VIEW = Adjust Playback timestretch
0x5e = KEY_ENTER # CROSS = select
0x54 = KEY_UP # UP = Up/Skip forward 10 minutes
0x56 = KEY_DOWN # DOWN = Down/Skip back 10 minutes
0x57 = KEY_LEFT # LEFT = Left/Skip back 5 seconds
0x55 = KEY_RIGHT # RIGHT = Right/Skip forward 30 seconds
0x0b = KEY_ENTER # ENTER = select
0x5a = KEY_F10 # L1 = volume down
0x58 = KEY_J # L2 = decrease the play speed
0x51 = KEY_HOME # L3 = commercial skip previous
0x5b = KEY_F11 # R1 = volume up
0x59 = KEY_U # R2 = increase the play speed
0x52 = KEY_END # R3 = commercial skip next
0x43 = KEY_F9 # PS button = mute
0x50 = KEY_M # SELECT = menu (as per PS convention)
0x53 = KEY_ENTER # START = select / Enter (matches terminology in mythwelcome)
0x30 = KEY_PAGEUP # PREV = jump back (default 10 minutes)
0x76 = KEY_J # INSTANT BACK (newer RCs only) = decrease the play speed
0x75 = KEY_U # INSTANT FORWARD (newer RCs only) = increase the play speed
0x31 = KEY_PAGEDOWN # NEXT = jump forward (default 10 minutes)
0x33 = KEY_COMMA # SCAN BACK = decrease scan forward speed / play
0x32 = KEY_P # PLAY = play/pause
0x34 = KEY_DOT # SCAN FORWARD decrease scan backard speed / increase playback speed; 3x, 5, 10, 20, 30, 60, 120, 180
0x60 = KEY_LEFT # FRAMEBACK = Left/Skip back 5 seconds/rewind one frame
0x39 = KEY_P # PAUSE = play/pause
0x38 = KEY_P # STOP = play/pause
0x61 = KEY_RIGHT # FRAMEFORWARD = Right/Skip forward 30 seconds/advance one frame
0xff = KEY_MAX
21:48
I have not spent much time finding out why not all the key presses work. Especially, given that most places on the internet mention these mappings. For me, some of the key scan codes aren t even reported. For keys like L1, L2, L3, R1, R2, R3, Next_Item, Prev_Item, they generate no codes in the kernel.
If anyone has suggestions, ideas or fixes, I d appreciate if you can drop a comment or email me privately.
But given my limited use to get a simple remote ready, to be usable with Kodi, I was apt with only some of the keys working.
Mapping the keys in Kodi
With the limited number of keys detected, mapping those keys to what Kodi could use was the next step. Kodi has a very nice and easy to use module, Keymap Editor. It is very simple to use and map detected keys to functionalities you want. With it, I was able to get a functioning remote to use with my Kodi HTPC setup.
Update: Wed Jun 17 11:38:20 2020
One annoying problem that breaks the overall experience is the following bug on the driver side, that results in connections not being established instantly.
Once the device goes into sleep mode, in random attempts, waking up and re-establishing a BT connection can be multi-poll affair. This can last from a couple of seconds to well over minute.
Random suggestions on the internet mention disabling the autosuspend functionality for the device in the driver with btusb.enable_autosuspend=n, but that did not help in this case.
Given that this device is enumberated over the USB Bus, it probably needs this feature applied to the whole USB tree of the device s chain. Something to investigate over the weekend.
Jun 16 20:41:23 lenovo kernel: Bluetooth: hci0: ACL packet for unknown connection handle 7
Jun 16 20:41:43 lenovo kernel: Bluetooth: hci0: ACL packet for unknown connection handle 8
Jun 16 20:41:59 lenovo kernel: Bluetooth: hci0: ACL packet for unknown connection handle 9
Jun 16 20:42:18 lenovo kernel: input: BD Remote Control as /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0/hci0:10/0005:054C:030>
Jun 16 20:42:18 lenovo kernel: sony 0005:054C:0306.0006: input,hidraw1: BLUETOOTH HID v1.00 Gamepad [BD Remote Control] on 2c:33:7a:8e:d6:30
Jun 16 20:51:59 lenovo kernel: input: BD Remote Control as /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0/hci0:11/0005:054C:030>
Jun 16 20:51:59 lenovo kernel: sony 0005:054C:0306.0007: input,hidraw1: BLUETOOTH HID v1.00 Gamepad [BD Remote Control] on 2c:33:7a:8e:d6:30
Jun 16 21:05:55 lenovo rtkit-daemon[1723]: Supervising 3 threads of 1 processes of 1 users.
Jun 16 21:05:55 lenovo rtkit-daemon[1723]: Successfully made thread 32747 of process 1646 owned by '1000' RT at priority 5.
Jun 16 21:05:55 lenovo rtkit-daemon[1723]: Supervising 4 threads of 1 processes of 1 users.
Jun 16 21:05:56 lenovo kernel: Bluetooth: hci0: ACL packet for unknown connection handle 12
Jun 16 21:06:12 lenovo kernel: Bluetooth: hci0: ACL packet for unknown connection handle 1
Jun 16 21:06:34 lenovo kernel: Bluetooth: hci0: ACL packet for unknown connection handle 2
Jun 16 21:06:59 lenovo kernel: input: BD Remote Control as /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0/hci0:3/0005:054C:0306>
Jun 16 21:06:59 lenovo kernel: sony 0005:054C:0306.0008: input,hidraw1: BLUETOOTH HID v1.00 Gamepad [BD Remote Control] on 2c:33:7a:8e:d6:30
Others
There s a package, kodi-eventclients-ps3, which can be used to talk to the BD Remote. Unfortunately, it isn t up-to-date. When trying to make use of it, I ran into a couple of problems.
First, the easy one is:
rrs@lenovo:~/ps3pair$ kodi-ps3remote localhost 9777
usr/share/pixmaps/kodi//bluetooth.png
Traceback (most recent call last):
File "/usr/bin/kodi-ps3remote", line 220, in <module>
File "/usr/bin/kodi-ps3remote", line 208, in main
xbmc.connect(host, port)
packet = PacketHELO(self.name, self.icon_type, self.icon_file)
File "/usr/lib/python3/dist-packages/kodi/xbmcclient.py", line 285, in __init__
with open(icon_file, 'rb') as f:
11:16 => 1
This one was simple as it was just a broken path.
The second issue with the tool is a leftover from python2 to python3 conversion.
rrs@lenovo:/etc/bluetooth$ kodi-ps3remote localhost
/usr/share/pixmaps/kodi//bluetooth.png
Searching for BD Remote Control
(Hold Start + Enter on remote to make it discoverable)
Redmi (E8:5A:8B:73:57:44) in range
Living Room TV (E4:DB:6D:24:23:E9) in range
Could not find BD Remote Control. Trying again...
Searching for BD Remote Control
(Hold Start + Enter on remote to make it discoverable)
Living Room TV (E4:DB:6D:24:23:E9) in range
Redmi (E8:5A:8B:73:57:44) in range
Could not find BD Remote Control. Trying again...
Searching for BD Remote Control
(Hold Start + Enter on remote to make it discoverable)
BD Remote Control (00:1E:3D:10:29:0F) in range
Found BD Remote Control with address 00:1E:3D:10:29:0F
Attempting to pair with remote
Remote Paired.
Traceback (most recent call last):
File "/usr/bin/kodi-ps3remote", line 221, in <module>
main()
File "/usr/bin/kodi-ps3remote", line 212, in main
if process_keys(remote, xbmc):
File "/usr/bin/kodi-ps3remote", line 164, in process_keys
keycode = data.encode("hex")[10:12]
AttributeError: 'bytes' object has no attribute 'encode'
11:24 => 1
Fixing that too did not give me the desired result on using the BD Remote in the way I want. So eventually, I gave up and used Kodi s Keymap Editor instead.
Next
Next in line, when I can manage to get some free time, is to improve the Kodi Video Scraper to have a fallback mode. Currently, for files where it cannot determine the content, it reject the file resulting in those files not showing up in your collection at all. A better approach would have been to have a fallback mode, that when the scraper cannot determine the content, it should fallback to using the filename scraper
With the current four-month release cycle, the next Rcpp release is due in July following the 1.0.4 release in March. Just prior to the 1.0.4 release I had asked this:
It would be particularly beneficial if those with unsual build dependencies tested it as we would increase overall coverage beyond what I get from testing against 1800+ CRAN packages. BioConductor would also be welcome.
but only on the rcpp-devel list, and only about a good week prior to the release.
I remain rather disappointed and disillusioned about what happened after 1.0.4 was released. Two PRs in that release were soon seen to have side effects on more marginal test systems, precisely what added testing could have revealed. An additional issue arose from changes in R s make system, which is harder to anticipate or test. Each and every infelicity was fixed within a day or so, and we always make candidate releases available the current Rcpp as of this writing is 1.0.4.12 meaning twelve microreleases were made since 1.0.4. And those microreleases are always available for normal download and install.packages use via the Rcpp drat repository accessible to all. So it was truly troubling to see some, especially those with experience in setting up or running testing / ci platforms, pretend to be unable to access, install, and provide these for their own tests, or the tests of their users. It just doesn t pass a basic logic test: it takes a single call to install.packages(), or, even more easily, a single assignment of an auxiliary repo. All told this was a rather sad experience.
So let s try to not repeat this. If you, or maybe users of a build or ci system you maintain, rely on Rcpp, and especially if you do so on systems outside the standard CRAN grid of three OSs and the triplet of previous, current, next releases of R, then please help by testing. I maitain these release as a volunteer, unpaid at that, and I simply cannot expand to more systesm. We take reverse dependency check seriously (and I just run two taking about a day each) but if you insist on building on stranger hardware or much older releases it will be up to you to ensure Rcpp passes. We prep for CRAN, and try our best to pass at CRAN. For nearly a dozen years.
To install the current microrelease from the Rcpp drat repository, just do
That is all there is to it. You could even add the Rcpp drat repository to your repository list.
Rcpp has become successful because so many people help with suggestions, documentation, and code. It is used by (as of today) 1958 CRAN packages, 205 BioConductor packages, and downloaded around a million times per month. So if you can, please help now with some more testing.
If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.
This post was originally published in the Wikimedia Tech blog, and
is authored by Arturo Borrero Gonzalez and Brooke Storm.
In the previous post, we shared the context on the recent Kubernetes upgrade that we
introduced in the Toolforge service. Today we would like to dive a bit more in the technical
details.
Custom admission controllers
One of the key components of the Toolforge Kubernetes are our custom admission controllers. We use
them to validate and enforce that the usage of the service is what we intended for. Basically, we
have 2 of them:
The source code is written in Golang, which is pretty convenient for natively working in a
Kubernetes environment. Both code repositories include extensive documentation: how to develop,
test, use, and deploy them. We decided to go with custom admission controllers because we couldn t
find any native (or built-in) Kubernetes mechanism to accomplish the same sort of checks on user
activity.
With the Ingress controller, we want to ensure that Ingress objects only handle traffic to our
internal domains, which by the time of this writing, are toolforge.org
(our new domain) and tools.wmflabs.org (legacy). We safe-list the kube-system
namespace and the tool-fourohfour namespace because both need special consideration. More on the
Ingress setup later.
The registry controller is pretty simple as well. It ensures that only our
internal docker registry is used for user-scheduled containers running in Kubernetes.
Again, we exclude from the checks containers running in the kube-system namespace (those used by
Kubernetes itself). Other than that, the validation itself is pretty easy. For some extra
containers we run (like those related to Prometheus metrics) what we do is simply upload those
docker images to our internal registry. The controls provided by this admission controller helps us
validate that only FLOSS software is run in our environment, which is
one of the core rules of Toolforge.
RBAC and Pod Security Policy setup
I would like to comment next on our RBAC and Pod Security Policy setup. Using the Pod
Security Policies (or PSP) we establish a set of constraints on what containers can and can t do in
our cluster. We have many PSP configured in our setup:
Privileged policy: used by Kubernetes containers themselves basically a very relaxed set of
constraints that are required for the system itself to work.
Default policy: a bit more restricted than the privileged policy, is intended for admins to
deploy services, but it isn t currently in use..
Toolforge user policies: this applies to user-scheduled containers, and there are some obvious
restrictions here: we only allow unprivileged pods, we control which HostPath is available for
pods, use only default Linux capabilities, etc.
Each user can interact with their own namespace (this is how we achieve multi-tenancy in the
cluster). Kubernetes knows about each user by means of TLS certs, and for that we have RBAC. Each
user has a rolebinding to a shared cluster-role that defines how Toolforge tools can use the
Kubernetes API. The following diagram shows the design of our RBAC and PSP in our cluster:
RBAC and PSP for Toolforge, original image in wikitech
I mentioned that we know about each user by means of TLS certificates. This is true, and in fact,
there is a key component in our setup called maintain-kubeusers. This custom piece
of Python software is run as a pod inside the cluster and is responsible for reading our external
user database (LDAP) and generating the required credentials, namespaces, and other configuration
bits for them. With the TLS cert, we basically create a kubeconfig file that is then written into
the homes NFS share, so each Toolforge user has it in their shell home directory.
Networking and Ingress setup
With the basic security controls in place, we can move on to explaining our networking and Ingress
setup. Yes, the Ingress word might be a bit overloaded already, but we refer here to
Ingress as the path that end-users follow from their web browser in their local machine to a
webservice running in the Toolforge cluster.
Some additional context here. Toolforge is not only Kubernetes, but we also have a Son of
GridEngine deployment, a job scheduler that covers some features not available in Kubernetes. The
grid can also run webservices, although we are encouraging users to migrate them to Kubernetes. For
compatibility reasons, we needed to adapt our Ingress setup to accommodate the old web grid.
Deciding the layout of the network and Ingress was definitely something that took us some time to
figure out because there is not a single way to do it right.
The following diagram can be used to explain the different steps involved in serving a web service
running in the new Toolforge Kubernetes.
Toolforge k8s network topology, original image in Wikitech
The end-user HTTP/HTTPs request first hits our front proxy in (1). Running here is
NGINX with a custom piece of LUA code that is able to decide whether to contact the web grid
or the new Kubernetes cluster. TLS termination happens here as well, for both domains
(toolforge.org and tools.wmflabs.org). Note this proxy is reachable from the internet, as it
uses a public IPv4 address, a floating IP from CloudVPS, the infrastructure service we
provide based on Openstack. Remember that our Kubernetes is directly built in virtual machines a
bare-metal type deployment.
If the request is directed to a webservice running in Kubernetes, the request now reaches
haproxy in (2), which knows the cluster nodes that are available for Ingress. The
original 80/TCP packet is now translated to 30000/TCP; this is the TCP port we use internally for
the Ingress traffic. This haproxy instance provides load-balancing also for the Kubernetes API as
well, using 6443/TCP. It s worth mentioning that unlike the Ingress, the API is only reachable from
within the cluster and not from the internet.
We have a NGINX-Ingress NodePort service listening in 30000/TCP in every Kubernetes worker node in
(3); this helps the request to eventually reach the actual NGINX-Ingress pod in (4), which is
listening in 8080/TCP. You can see in the diagram how in the API server (5) we hook the Ingress
admission controller (6) to validate Kubernetes Ingress configuration objects before allowing them
in for processing by NGINX-Ingress (7).
The NGINX-Ingress process knows which tools webservices are online and how to contact them by means
of an intermediate Service object in (8). This last Service object means the request finally
reaches the actual tool pod in (9). At this point, it is worth noting that our Kubernetes cluster
uses internally kube-proxy and Calico, both using Netfilter components to handle traffic.
tools-webservice
Most user-facing operations are simplified by means of another custom piece of Python code:
tools-webservice. This package provides users with the webservice command line
utility in our shell bastion hosts. Typical usage is to just run webservice start stop status. This
utility creates all the required Kubernetes objects on-demand like Deployment, ReplicaSet, Ingress
and Service to ease deploying web apps in Toolforge. Of course, advanced users can interact
directly with Kubernetes API and create their custom configuration objects. This utility is just a
wrapper, a shortcut.
tool-fourohfour and tool-k8s-status
The last couple of custom components we would like to mention are the tool-fourohfour
and tool-k8s-status web services. These two utilities run inside the cluster as if
they were any other user-created tool. The fourohfour tool allows for a controlled handling of HTTP
404 errors, and it works as the default NGINX-Ingress backend. The k8s-status tool shows plenty of
information about the cluster itself and each tool running in the cluster, including links to the
Server Admin Log, an auto-generated grafana dashboard for metrics, and more.
For metrics, we use an external Prometheus server that contacts the Kubernetes cluster to scrape
metrics. We created a custom metrics namespace in which we deploy all the different
components we use to observe the behavior of the system:
metrics-server: used by some utilities like kubectl top.
kube-state-metrics: provides advanced metrics about the state of the cluster.
cadvisor: to obtain fine-grained metrics about pods, deployments, nodes, etc.
All the Prometheus data we collect is used in several different Grafana dashboards, some of them
directed for user information like the ones linked by the k8s-status tool and some others for
internal use by us the engineers. These are for internal use but are still public, like the
Ingress specific dashboard, or the cluster state dashboard. Working
publicly, in a transparent way, is key for the success of CloudVPS in general and Toolforge in
particular. Like we commented in the previous post, all the engineering work that was done here was
shared by community members.
By the community, for the community
We think this post sheds some light on how the Toolforge Kubernetes service works, and we hope it
could inspire others when trying to build similar services or, even better, help us improve
Toolforge itself. Since this was first put into production some months ago we detected already some
margin for improvement in a couple of the components. As in many other engineering products, we
will follow an iterative approach for evolving the service. Mind that Toolforge is maintained by
the Wikimedia Foundation, but you can think of it as a service by the community for the community.
We will keep an eye on it and
have a list of feature requests and things to improve in the future. We are looking forward
to it!
This post was originally published in the Wikimedia Tech blog, and
is authored by Arturo Borrero Gonzalez and Brooke Storm.
Video Essay Catalog No. 91 by Kevin B. Lee. Featured on the New York Times and other outlets. Originally published December 13, 2011 on Fandor. https://carpetbagger.blogs.nytimes.com/2011/12/19/staring-in-awe-its-the-spielberg-face/?_r=0
Bandicoot Cabbagepatch, Bandersnatch Cumberbund, and even Wimbledon
Tennismatch: there seem to be endless variations on the name of Benedict
Cumberbatch. [...] But how is a normal internet citizen supposed to know,
when they hear someone say I just can t stop looking at gifs of Bombadil
Rivendell that this person isn t talking about some other actor with a
name and a voice and cheekbones? Or in other words, what makes for a
reasonable variation of the name Bendandsnap Calldispatch?
Last month, I got selected as a Google Summer of Code student for Debian again! \o/
Here is my monthly update covering what I have been doing in the free software world during June 2020 (
The twelveth release of 
Limesurvey is an online survey tool. It is very powerful and commonly used in academic environments because it is Free Software (GPLv2+), allows for local installations protecting the data of participants and allowing to comply with data protection regulations. This also means there are typically no load-balanced multi-server szenarios with HA databases. But simple VMs where Limesurvey runs and needs upgrading in place.
There's an LTS branch (currently 3.x) and a stable branch (currently 4.x). There's also a 2.06 LTS branch that is restricted to paying customers. The main developers behind Limesurvey offer many services from template design to custom development to support to hosting ("Cloud", "Limesurvey Pro"). Unfortunately they also charge for easy updates called "ComfortUpdate" (currently 39 for three months) and the manual process is made a bit cumbersome to make the "ComfortUpdate" offer more attractive.
Due to Limesurvey being an old code base and UI elements not being clearly separated, most serious use cases will end up patching files and symlinking logos around template directories. That conflicts a bit with the opaque "ComfortUpdate" process where you push a button and then magic happens. Or you have downtime and a recovery case while surveys are running.
If you do not intend to use the "ComfortUpdate" offering, you can prevent Limesurvey from connecting to 
LWN published today yet another great piece of writing, 
With the current four-month release cycle, the next 
This post was originally published in the 
