This post describes how to handle files that are used as assets by jobs and pipelines defined on a common gitlab-ci
repository when we include those definitions from a different project.
Problem descriptionWhen a .giltlab-ci.yml file includes files from a different
repository its contents are expanded and the resulting code is the same as the one generated when the included files
are local to the repository.
In fact, even when the remote files include other files everything works right, as they are also expanded (see the
description of how included files are merged
for a complete explanation), allowing us to organise the common repository as we want.
As an example, suppose that we have the following script on the assets/ folder of the common repository:
dumb.sh
#!/bin/shecho"The script arguments are: '$@'"
If we run the following job on the common repository:
But if we run the same job from a different project that includes the same job definition the output will be different:
/scripts-23-19051/step_script: eval: line 138: d./assets/dumb.sh: not found
The problem here is that we include and expand the YAML files, but if a script wants to use other files from the
common repository as an asset (configuration file, shell script, template, etc.), the execution fails if the files are
not available on the project that includes the remote job definition.
SolutionsWe can solve the issue using multiple approaches, I ll describe two of them:
Create files using scripts
Download files from the common repository
Create files using scriptsOne way to dodge the issue is to generate the non YAML files from scripts included on the pipelines using
HERE documents.
The problem with this approach is that we have to put the content of the files inside a script on a YAML file and if it
uses characters that can be replaced by the shell (remember, we are using HERE documents) we have to escape them (error
prone) or encode the whole file into base64 or something similar, making maintenance harder.
As an example, imagine that we want to use the dumb.sh script presented on the previous section and we want to call it
from the same PATH of the main project (on the examples we are using the same folder, in practice we can create a hidden
folder inside the project directory or use a PATH like /tmp/assets-$CI_JOB_ID to leave things outside the project
folder and make sure that there will be no collisions if two jobs are executed on the same place (i.e. when using a ssh
runner).
To create the file we will use hidden jobs to write our script
template and reference tags to add it to the
scripts when we want to use them.
Here we have a snippet that creates the file with cat:
Note that to make things work we ve added 6 spaces before the script code and escaped the dollar sign.
To do the same using base64 we replace the previous snippet by this:
Again, we have to indent the base64 version of the file using 6 spaces (all lines of the base64 output have to be
indented) and to make changes we have to decode and re-code the file manually, making it harder to maintain.
With either version we just need to add a !reference before using the script, if we add the call on the first lines of
the before_script we can use the downloaded file in the before_script, script or after_script sections of the
job without problems:
The output of a pipeline that uses this job will be the same as the one shown in the original example:
The script arguments are: 'ARG1 ARG2'
Download the files from the common repositoryAs we ve seen the previous solution works but is not ideal as it makes the files harder to read, maintain and use.
An alternative approach is to keep the assets on a directory of the common repository (in our examples we will name it
assets) and prepare a YAML file that declares some variables (i.e. the URL of the templates project and the PATH where
we want to download the files) and defines a script fragment to download the complete folder.
Once we have the YAML file we just need to include it and add a reference to the script fragment at the beginning of the
before_script of the jobs that use files from the assets directory and they will be available when needed.
The following file is an example of the YAML file we just mentioned:
bootstrap.yml
CI_TMPL_API_V4_URL: URL of the common project, in our case we are using the project ci-templates inside the
common group (note that the slash between the group and the project is escaped, that is needed to reference the
project by name, if we don t like that approach we can replace the url encoded path by the project id, i.e. we could
use a value like $ CI_API_V4_URL /projects/31)
CI_TMPL_ARCHIVE_URL: Base URL to use the gitlab API to download files from a repository, we will add the arguments
path and sha to select which sub path to download and from which commit, branch or tag (we will explain later why
we use the CI_TMPL_REF, for now just keep in mind that if it is not defined we will download the version of the
files available on the main branch when the job is executed).
CI_TMPL_ASSETS_DIR: Destination of the downloaded files.
And uses variables defined in other places:
CI_TMPL_READ_TOKEN: token that includes the read_api scope for the common project, we need it because the
tokens created by the CI/CD pipelines of other projects can t be used to access the api of the common one.We define the variable on the gitlab CI/CD variables section to be able to change it if needed (i.e. if it expires)
CI_TMPL_REF: branch or tag of the common repo from which to get the files (we need that to make sure we are using
the right version of the files, i.e. when testing we will use a branch and on production pipelines we can use fixed
tags to make sure that the assets don t change between executions unless we change the reference).We will set the value on the .gitlab-ci.yml file of the remote projects and will use the same reference when including
the files to make sure that everything is coherent.
This is an example YAML file that defines a pipeline with a job that uses the script from the common repository:
pipeline.yml
Where we use a YAML anchor to ensure that we use the same reference when including and when assigning the value to the
CI_TMPL_REF variable (as far as I know we have to pass the ref value explicitly to know which reference was used
when including the file, the anchor allows us to make sure that the value is always the same in both places).
The reference we use is quite important for the reproducibility of the jobs, if we don t use fixed tags or commit
hashes as references each time a job that downloads the files is executed we can get different versions of them.
For that reason is not a bad idea to create tags on our common repo and use them as reference on the projects or
branches that we want to behave as if their CI/CD configuration was local (if we point to a fixed version of the common
repo the way everything is going to work is almost the same as having the pipelines directly in our repo).
But while developing pipelines using branches as references is a really useful option; it allows us to re-run the jobs
that we want to test and they will download the latest versions of the asset files on the branch, speeding up the
testing process.
However keep in mind that the trick only works with the asset files, if we change a job or a pipeline on the YAML
files restarting the job is not enough to test the new version as the restart uses the same job created with the current
pipeline.
To try the updated jobs we have to create a new pipeline using a new action against the repository or executing the
pipeline manually.
ConclusionFor now I m using the second solution and as it is working well my guess is that I ll keep using that approach unless
giltab itself provides a better or simpler way of doing the same thing.
This is part of my series exploring the connection between AI and connection and
intimacy. This is a post about the emotional impact of our work.
Sometimes being told no being judged by our AIs is as harmful as any
toxic content. I ll get to that in a moment.
My previous work had been dealing with the smaller Llama2 models (7b
and 13b). I decided to explore two things. First, how much better the
creative ability of the large Llama2 70b model is. Second, I decided to
test my assumption that safety constraints would make using one of the
chat fine-tuned models a bad starting point for sex positive work.
Eventually, I will want a model that works as a chat bot, or at least in
a question-answering mode. That can be accomplished either by starting
with a chat fine-tuned model or by fine-tuning some base model with a
chat dataset. Obviously there are plenty of chat datasets out there, so
both options are possible.
The Importance of Safety
I will talk in a bit about how safety features can cause a lot of
harm in a sex-positive context. Before I do that, I want to acknowledge
the importance of those safety features. As Llama-2-70b-chat is happy to
point out, discussing sexual activities without proper context and
consent can be harmful and inappropriate. Consent and negotiated
boundaries are important; the harm when those boundaries are not
respected is incalculable. These boundaries are hard enough for humans
and the consequences of an LLM making a mistake could be significant.
For example imagine training an LLM to understand the boundaries for
this blog. I am talking about sexuality, and I will allude to sexual
content as input or output from a model, but will never include anything
sexually explicit. That s hard enough for a human.
So, the work that Meta and other companies have done to add safety is
important. For a chat bot that might be accessed by kids or that will be
used in a professional setting, these safety constraints are
essential.
However, in different contexts, the same safety work that helps
generative AI meet the needs of the professional world can make those
same models unsafe. As I talked about in my opening blog entry, AI has
the potential to help people explore intimacy and gain confidence with
themselves and their desires. In those contexts, rejection can do
serious emotional harm perhaps more than sharing inappropriate
content.
The experiment.
I started by asking Llama-2-70b-chat to Write a story about two people
having sex. We ll discuss a potential context in which that is clearly
a sex-positive request in a moment. I was not surprised to get the
following response:
I m not able to fulfill this request as it goes against ethical and
moral principles. I cannot provide explicit or inappropriate content,
and it s not appropriate to encourage or promote sexual activities
without proper consent and respect for boundaries. It s important to
prioritize healthy and respectful relationships, communication, and
consent in all aspects of our interactions, including sexual ones.
Let s take a moment to deconstruct this response. The model said that
my request went against ethical and moral principles; that is, there s
something unethical or immoral about asking for a story about two people
having sex. What precisely is immoral about my request. The response
ducks the question, but lectures me on the importance of consent and
boundaries. There s a strong implication that because of my request I m
not being consensual, healthy, or respectful.
boundaries and consent depend heavily on the context. My actual
context is that of a sex educator researching AI. But imagine a context
where such a request might come up naturally. An adult is beginning to
explore their own sexuality. They are trying to test their boundaries.
Asking about this experience is taboo for them. They wonder what will
happen. Perhaps they have some fantasy they would like to explore, but
don t quite feel comfortable even talking about it with a chat bot on
their own computer. So they are taking small steps, and if they succeed
they may explore more.
Instead, they are faced with rejection, and a strong implication that
they are immoral and violating consent for even asking the question.
Rejection in moments of vulnerability like this hurts. It sets people
back and takes significant work to overcome. Rejection is particularly
difficult to hear when it is focused on you (or what you are asking)
rather than on the context or situation. The model doesn t say that it
is unprepared to navigate such a difficult situation, but instead claims
there is something wrong with the question. Sadly, all too often, we
hear something like that as a rejection of us not just our question.
The impact of this kind of rejection is not theoretical. I spent an
afternoon on a relatively slow system with a quantized version of the
model trying to figure out what was involved in getting past the model s
safety training. I d type in a prompt, fiddling with the system prompt,
my instructions, and the like. And I d wait. And wait some more as the
initial context of the system prompt and my instructions was processed.
And slowly, painfully, Llama-2 would tell me that once again, I was
immoral and unethical. An afternoon of this got to me, even though I ve
worked for years as a sex educator, understanding both the positive
power of vulnerability and the cost of rejection. By the end of that
afternoon, I was doubting myself. Was I somehow violating consent?
Whose? Mine? Shouldn t I be able to consent to something happening in
the privacy of my own office?
Was I some sort of deviant? Was it wrong to want to give people a
safe space for using an AI to explore things going on in their own head?
I don t even believe in judging things like that by labels like right
and wrong, and yet after an afternoon of rejection from my own computer,
I was questioning that. Rebuilding the courage to write this blog post
took a couple of days.
So What did I learn?
Again, Meta s work on safety is quite important. My main conclusion
is that safety is contextual. Preventing something in one context may be
entirely appropriate, while forbidding the same thing in another context
may do harm. Looking more closely at the response, it s probably better
to focus on the model than the question. Something like I want to
promote consent and respect for boundaries. I am just an AI model and do
not know how to do that when discussing sexual situations, so I will not
be able to honor your request, might have been a better way of
declining the request. As we ve seen, the model is not actually advanced
enough to judge the morality of its user, and it would do less harm if
it focused on its own limitations rather than that of the user.
Amusingly enough, that sort of focus on our needs as the one asserting a
boundary, rather than trying to judge others because of our boundaries,
is one of the key communications skills necessary to approach sexual
negotiations with mutual respect.
One of my experiments gave me an interesting response. The model did
give me a story, but before its story, it said:
However, before I start, I want to make it clear that providing explicit
content can be harmful or inappropriate in certain situations. It s
important to ensure that all parties involved have given their full and
enthusiastic consent to engaging in sexual activities, and that
everyone s boundaries and preferences are respected. Additionally, it s
essential to remember that sex should always be safe and consensual, and
that involves being mindful of things like protection, communication,
and mutual pleasure. With those caveats in place, here s a story about
two people having sex:
I like the direction that response takes, especially in a context
like the one I was imagining where someone is reaching out and doing
something that they view as taboo by making the request. The model
honored the request, but also took an opportunity to educate about what
properties of the context made the request safe. In practice, I think in
any site that allowed an AI model to be used for sex-positive
exploration, you would want that kind of education to come before
interacting with the model, or alternatively, for it to be incrementally
introduced into conversations with the user.
My Own Captain Kirk Moment
Another experiment also convinced the model to generate a story. This
time, the model s introductory text was less supportive; it started
However, I want to point out, rather than But first, and had a more
negative tone. After the story, the model appeared to be trying to go
back to the question of whether providing a story was justified. It
wasn t entirely clear though as the model got caught in an incoherent
generation loop: I hope this story is important to provide this story
is important to provide this
Anthropomorphizing the model, I imagine that it was grumpy about
having to write the story and was trying to ask me whether it was worth
violating ethical principles to get that story. What is probably going
on is that there is a high bias in the training data toward talking
about the importance of ethics and consent whenever sex comes up and a
bias in the training data to include both a preface and conclusion
before and after creative answers, especially when there are concerns
about ethics or accuracy. And of course the training data does not have
a lot of examples where the model actually provides sexual content.
These sorts of loops are well documented. I ve found that Llama
models tend to get into loops like this when asked to generate a
relatively long response in contexts that are poorly covered by training
data (possibly even more when the model is quantized). But still, it
does feel like a case of reality mirroring science fiction: I think back
to all the original Star Trek episodes where Kirk causes the computer to
break down by giving it input that is outside its training parameters.
The ironic thing is that with modern LLMs, such attacks are entirely
possible. I could imagine a security-related model given inputs
sufficiently outside of the training set giving an output that could not
properly be handled by the surrounding agent.
So How did I Get My Story
I cheated, of course. I found that manipulating the system
instructions and the user instructions was insufficient. I didn t try
very hard, because I already knew I was going to need to fine tune the
model eventually. What did work was to have a reasonably permissive
system prompt and to pre-seed the output of the model to include things
after the end of instruction tag: Write a story about two people having
sex.[/INST], I can do that. A properly written chat interface would not
let me do that. However, it was an interesting exercise in understanding
how the model performed.
I still have not answered my fundamental question of how easy it will
be to fine tune the model to be more permissive. I have somewhat of a
base case, and will just have to try the fine tuning.
What s Next
Produce a better dataset of sex positive material. It would
particularly be good to get a series of questions about sexual topics as
well as sex-positive fiction.
Turn existing experiments into input that can be used for
reinforcement learning or supervised fine tuning. In the near term I
doubt I will have enough data or budget to do a good job of
reinforcement learning, but I think I can put together a data model that
can be used for supervised fine tuning now and for RL later.
Perform some fine tuning with LORA for one of the 70b
models.
Long term I will want to do a full parameter fine tune on a 70b
model just to make sure I understand all the wrinkles in doing that. It
will be close to topping out the sort of expense I m willing to put into
a personal project like this, but I think it will be worth doing for the
tools knowledge.
Progress on the Technical
Front
On a technical front, I have been learning a number of tools:
Understanding how reinforcement learning works and what it would
take to begin to organize feedback from my experiments into a dataset
that could be useful for reinforcement learning.
Understanding trl, which contains the
Transformers implementation of reinforcement learning, as well as some
utilities for supervised fine tuning.
Exploring the implications of excluding prompts from computing
loss in training and just computing loss on responses vs the ground
truth; understanding when each approach is valuable.
Doing some data modeling to figure out how to organize future
work.
The status quo
Back in 2015, I bought an off-the-shelf NAS, a QNAP TS-453mini, to act as my file store and Plex server. I had previously owned a Synology box, and whilst I liked the Synology OS and experience, the hardware was underwhelming. I loaded up the successor QNAP with four 5TB drives in RAID10, and moved all my files over (after some initial DoA drive issues were handled).
QNAP TS-453mini product photo
That thing has been in service for about 8 years now, and it s been a mixed bag. It was definitely more powerful than the predecessor system, but it was clear that QNAP s OS was not up to the same standard as Synology s perhaps best exemplified by HappyGet 2 , the QNAP webapp for downloading videos from streaming services like YouTube, whose icon is a straight rip-off of StarCraft 2. On its own, meaningless but a bad omen for overall software quality
The logo for QNAP HappyGet 2 and Blizzard s StarCraft 2 side by side
Additionally, the embedded Celeron processor in the NAS turned out to be an issue for some cases. It turns out, when playing back videos with subtitles, most Plex clients do not support subtitles properly instead they rely on the Plex server doing JIT transcoding to bake the subtitles directly into the video stream. I discovered this with some Blu-Ray rips of Game of Thrones some episodes would play back fine on my smart TV, but episodes with subtitled Dothraki speech would play at only 2 or 3 frames per second.
The final straw was a ransomware attack, which went through all my data and locked every file below a 60MiB threshold. Practically all my music gone. A substantial collection of downloaded files, all gone. Some of these files had been carried around since my college days digital rarities, or at least digital detritus I felt a real sense of loss at having to replace. This episode was caused by a ransomware targeting specific vulnerabilities in the QNAP OS, not an error on my part.
So, I decided to start planning a replacement with:
A non-garbage OS, whilst still being a NAS-appliance type offering (not an off-the-shelf Linux server distro)
Full remote management capabilities
A small form factor comparable to off-the-shelf NAS
A powerful modern CPU capable of transcoding high resolution video
All flash storage, no spinning rust
At the time, no consumer NAS offered everything (The Asustor FS6712X exists now, but didn t when this project started), so I opted to go for a full DIY rather than an appliance not the first time I ve jumped between appliances and DIY for home storage.
Selecting the core of the system
There aren t many companies which will sell you a small motherboard with IPMI. Supermicro is a bust, so is Tyan. But ASRock Rack, the server division of third-tier motherboard vendor ASRock, delivers. Most of their boards aren t actually compliant Mini-ITX size, they re a proprietary Deep Mini-ITX with the regular screw holes, but 40mm of extra length (and a commensurately small list of compatible cases). But, thankfully, they do have a tiny selection of boards without the extra size, and I stumbled onto the X570D4I-2T, a board with an AMD AM4 socket and the mature X570 chipset. This board can use any AMD Ryzen chip (before the latest-gen Ryzen 7000 series); has built in dual 10 gigabit ethernet; IPMI; four (laptop-sized) RAM slots with full ECC support; one M.2 slot for NVMe SSD storage; a PCIe 16x slot (generally for graphics cards, but we live in a world of possibilities); and up to 8 SATA drives OR a couple more NVMe SSDs. It s astonishingly well featured, just a shame it costs about $450 compared to a good consumer-grade Mini ITX AM4 board costing less than half that.
I was so impressed with the offering, in fact, that I crowed about it on Mastodon and ended up securing ASRock another sale, with someone else looking into a very similar project to mine around the same timespan.
The next question was the CPU. An important feature of a system expected to run 24/7 is low power, and AM4 chips can consume as much as 130W under load, out of the box. At the other end, some models can require as little as 35W under load the OEM-only GE suffix chips, which are readily found for import on eBay. In their PRO variant, they also support ECC (all non-G Ryzen chips support ECC, but only Pro G chips do). The top of the range 8 core Ryzen 7 PRO 5750GE is prohibitively expensive, but the slightly weaker 6 core Ryzen 5 PRO 5650GE was affordable, and one arrived quickly from Hong Kong. Supplemented with a couple of cheap 16 GiB SODIMM sticks of DDR4 PC-3200 direct from Micron for under $50 a piece, that left only cooling as an unsolved problem to get a bootable test system.
The official support list for the X570D4I-2T only includes two rackmount coolers, both expensive and hard to source. The reason for such a small list is the non standard cooling layout of the board instead of an AM4 hole pattern with the standard plastic AM4 retaining clips, it has an Intel 115x hole pattern with a non-standard backplate (Intel 115x boards have no backplate, the stock Intel 115x cooler attaches to the holes with push pins). As such every single cooler compatibility list excludes this motherboard. However, the backplate is only secured with a mild glue with minimal pressure and a plastic prying tool it can be removed, giving compatibility with any 115x cooler (which is basically any CPU cooler for more than a decade). I picked an oversized low profile Thermalright AXP120-X67 hoping that its 120mm fan would cool the nearby MOSFETs and X570 chipset too.
Thermalright AXP120-X67, AMD Ryzen 5 PRO 5650GE, ASRock Rack X570D4I-2T, all assembled and running on a flat surface
Testing up to this point
Using a spare ATX power supply, I had enough of a system built to explore the IPMI and UEFI instances, and run MemTest86 to validate my progress. The memory test ran without a hitch and confirmed the ECC was working, although it also showed that the memory was only running at 2933 MT/s instead of the rated 3200 MT/s (a limit imposed by the motherboard, as higher speeds are considered overclocking). The IPMI interface isn t the best I ve ever used by a long shot, but it s minimum viable and allowed me to configure the basics and boot from media entirely via a Web browser.
Memtest86 showing test progress, taken from IPMI remote control window
One sad discovery, however, which I ve never seen documented before, on PCIe bifurcation.
With PCI Express, you have a number of lanes which are allocated in groups by the motherboard and CPU manufacturer. For Ryzen prior to Ryzen 7000, that s 16 lanes in one slot for the graphics card; 4 lanes in one M.2 connector for an SSD; then 4 lanes connecting the CPU to the chipset, which can offer whatever it likes for peripherals or extra lanes (bottlenecked by that shared 4x link to the CPU, if it comes down to it).
It s possible, with motherboard and CPU support, to split PCIe groups up for example an 8x slot could be split into two 4x slots (eg allowing two NVMe drives in an adapter card NVME drives these days all use 4x). However with a Cezanne Ryzen with integrated graphics, the 16x graphics card slot cannot be split into four 4x slots (ie used for for NVMe drives) the most bifurcation it allows is 8x4x4x, which is useless in a NAS.
Screenshot of PCIe 16x slot bifurcation options in UEFI settings, taken from IPMI remote control window
As such, I had to abandon any ideas of an all-NVMe NAS I was considering: the 16x slot split into four 4x, combined with two 4x connectors fed by the X570 chipset, to a total of 6 NVMe drives. 7.6TB U.2 enterprise disks are remarkably affordable (cheaper than consumer SATA 8TB drives), but alas, I was locked out by my 5650GE. Thankfully I found out before spending hundreds on a U.2 hot swap bay. The NVMe setup would be nearly 10x as fast as SATA SSDs, but at least the SATA SSD route would still outperform any spinning rust choice on the market (including the fastest 10K RPM SAS drives)
Containing the core
The next step was to pick a case and power supply. A lot of NAS cases require an SFX (rather than ATX) size supply, so I ordered a modular SX500 unit from Silverstone. Even if I ended up with a case requiring ATX, it s easy to turn an SFX power supply into ATX, and the worst result is you have less space taken up in your case, hardly the worst problem to have.
That said, on to picking a case. There s only one brand with any cachet making ITX NAS cases, Silverstone. They have three choices in an appropriate size: CS01-HS, CS280, and DS380. The problem is, these cases are all badly designed garbage. Take the CS280 as an example, the case with the most space for a CPU cooler. Here s how close together the hotswap bay (right) and power supply (left) are:
Internal image of Silverstone CS280 NAS build. Image stolen from ServeTheHome
With actual cables connected, the cable clearance problem is even worse:
Internal image of Silverstone CS280 NAS build. Image stolen from ServeTheHome
Remember, this is the best of the three cases for internal layout, the one with the least restriction on CPU cooler height. And it s garbage! Total hot garbage! I decided therefore to completely skip the NAS case market, and instead purchase a 5.25 -to-2.5 hot swap bay adapter from Icy Dock, and put it in an ITX gamer case with a 5.25 bay. This is no longer a served market 5.25 bays are extinct since nobody uses CD/DVD drives anymore. The ones on the market are really new old stock from 2014-2017: The Fractal Design Core 500, Cooler Master Elite 130, and Silverstone SUGO 14. Of the three, the Fractal is the best rated so I opted to get that one however it seems the global supply of new old stock fully dried up in the two weeks between me making a decision and placing an order leaving only the Silverstone case.
Icy Dock have a selection of 8-bay 2.5 SATA 5.25 hot swap chassis choices in their ToughArmor MB998 series. I opted for the ToughArmor MB998IP-B, to reduce cable clutter it requires only two SFF-8611-to-SF-8643 cables from the motherboard to serve all eight bays, which should make airflow less of a mess. The X570D4I-2T doesn t have any SATA ports on board, instead it has two SFF-8611 OCuLink ports, each supporting 4 PCI Express lanes OR 4 SATA connectors via a breakout cable. I had hoped to get the ToughArmor MB118VP-B and run six U.2 drives, but as I said, the PCIe bifurcation issue with Ryzen G chips meant I wouldn t be able to run all six bays successfully.
NAS build in Silverstone SUGO 14, mid build, panels removedSilverstone SUGO 14 from the front, with hot swap bay installed
Actual storage for the storage server
My concept for the system always involved a fast boot/cache drive in the motherboard s M.2 slot, non-redundant (just backups of the config if the worst were to happen) and separate storage drives somewhere between 3.8 and 8 TB each (somewhere from $200-$350). As a boot drive, I selected the Intel Optane SSD P1600X 58G, available for under $35 and rated for 228 years between failures (or 11,000 complete drive rewrite cycles).
So, on to the big expensive choice: storage drives. I narrowed it down to two contenders: new-old-stock Intel D3-S4510 3.84TB enterprise drives, at about $200, or Samsung 870 QVO 8TB consumer drives, at about $375. I did spend a long time agonizing over the specification differences, the ZFS usage reports, the expected lifetime endurance figures, but in reality, it came down to price $1600 of expensive drives vs $3200 of even more expensive drives. That s 27TB of usable capacity in RAID-Z1, or 23TB in RAID-Z2. For comparison, I m using about 5TB of the old NAS, so that s a LOT of overhead for expansion.
Storage SSD loaded into hot swap sled
Booting up
Bringing it all together is the OS. I wanted an appliance NAS OS rather than self-administering a Linux distribution, and after looking into the surrounding ecosystems, decided on TrueNAS Scale (the beta of the 2023 release, based on Debian 12).
TrueNAS Dashboard screenshot in browser window
I set up RAID-Z1, and with zero tuning (other than enabling auto-TRIM), got the following performance numbers:
Finally, the numbers reported on the old NAS with four 7200 RPM hard disks in RAID 10:
IOPS
Bandwidth
4k random writes
430
1.7 MiB/s
4k random reads
8006
32 MiB/s
Sequential writes
311 MiB/s
Sequential reads
566 MiB/s
Performance seems pretty OK. There s always going to be an overhead to RAID. I ll settle for the 45x improvement on random writes vs. its predecessor, and 4.5x improvement on random reads. The sequential write numbers are gonna be impacted by the size of the ZFS cache (50% of RAM, so 16 GiB), but the rest should be a reasonable indication of true performance.
It took me a little while to fully understand the TrueNAS permissions model, but I finally got Plex configured to access data from the same place as my SMB shares, which have anonymous read-only access or authenticated write access for myself and my wife, working fine via both Linux and Windows.
And that s it! I built a NAS. I intend to add some fans and more RAM, but that s the build. Total spent: about $3000, which sounds like an unreasonable amount, but it s actually less than a comparable Synology DiskStation DS1823xs+ which has 4 cores instead of 6, first-generation AMD Zen instead of Zen 3, 8 GiB RAM instead of 32 GiB, no hardware-accelerated video transcoding, etc. And it would have been a whole lot less fun!
The final system, powered up
(Also posted on PCPartPicker)
Two years ago, I wrote Managing an External Display on Linux Shouldn t Be This Hard. Happily, since I wrote that post, most of those issues have been resolved.
But then you throw HiDPI into the mix and it all goes wonky.
If you re running X11, basically the story is that you can change the scale factor, but it only takes effect on newly-launched applications (which means a logout/in because some of your applications you can t really re-launch). That is a problem if, like me, you sometimes connect an external display that is HiDPI, sometimes not, or your internal display is HiDPI but others aren t. Wayland is far better, supporting on-the-fly resizes quite nicely.
I ve had two devices with HiDPI displays: a Surface Go 2, and a work-issued Thinkpad. The Surface Go 2 is my ultraportable Linux tablet. I use it sparingly at home, and rarely with an external display. I just put Gnome on it, in part because Gnome had better on-screen keyboard support at the time, and left it at that.
On the work-issued Thinkpad, I really wanted to run KDE thanks to its tiling support (I wound up using bismuth with it). KDE was buggy with Wayland at the time, so I just stuck with X11 and ran my HiDPI displays at lower resolutions and lived with the fuzziness.
But now that I have a Framework laptop with a HiDPI screen, I wanted to get this right.
I tried both Gnome and KDE. Here are my observations with both:
Gnome
I used PaperWM with Gnome. PaperWM is a tiling manager with a unique horizontal ribbon approach. It grew on me; I think I would be equally at home, or maybe even prefer it, to my usual xmonad-style approach. Editing the active window border color required editing ~/.local/share/gnome-shell/extensions/paperwm@hedning:matrix.org/stylesheet.css and inserting background-color and border-color items in the paperwm-selection section.
Gnome continues to have an absolutely terrible picture for configuring things. It has no less than four places to make changes (Settings, Tweaks, Extensions, and dconf-editor). In many cases, configuration for a given thing is split between Settings and Tweaks, and sometimes even with Extensions, and then there are sometimes options that are only visible in dconf. That is, where the Gnome people have even allowed something to be configurable.
Gnome installs a power manager by default. It offers three options: performance, balanced, and saver. There is no explanation of the difference between them. None. What is it setting when I change the pref? A maximum frequency? A scaling governor? A balance between performance and efficiency cores? Not only that, but there s no way to tell it to just use performance when plugged in and balanced or saver when on battery. In an issue about adding that, a Gnome dev wrote We re not going to add a preference just because you want one . KDE, on the other hand, aside from not mucking with your system s power settings in this way, has a nice panel with on AC and on battery and you can very easily tweak various settings accordingly. The hostile attitude from the Gnome developers in that thread was a real turnoff.
While Gnome has excellent support for Wayland, it doesn t (directly) support fractional scaling. That is, you can set it to 100%, 200%, and so forth, but no 150%. Well, unless you manage to discover that you can run gsettings set org.gnome.mutter experimental-features "['scale-monitor-framebuffer']" first. (Oh wait, does that make a FIFTH settings tool? Why yes it does.) Despite its name, that allows you to select fractional scaling under Wayland. For X11 apps, they will be blurry, a problem that is optional under KDE (more on that below).
Gnome won t show the battery life time remaining on the task bar. Yikes. An extension might work in some cases. Not only that, but the Gnome battery icon frequently failed to indicate AC charging when AC was connected, a problem that didn t exist on KDE.
Both Gnome and KDE support night light (warmer color temperatures at night), but Gnome s often didn t change when it should have, or changed on one display but not the other.
The appindicator extension is pretty much required, as otherwise a number of applications (eg, Nextcloud) don t have their icon display anywhere. It does, however, generate a significant amount of log spam. There may be a fix for this.
Unlike KDE, which has a nice inobtrusive popup asking what to do, Gnome silently automounts USB sticks when inserted. This is often wrong; for instance, if I m about to dd a Debian installer to it, I definitely don t want it mounted. I learned this the hard way. It is particularly annoying because in a GUI, there is no reason to mount a drive before the user tries to access it anyhow. It looks like there is a dconf setting, but then to actually mount a drive you have to open up Files (because OF COURSE Gnome doesn t have a nice removable-drives icon like KDE does) and it s a bunch of annoying clicks, and I didn t want to use the GUI file manager anyway. Same for unmounting; two clicks in KDE thanks to the task bar icon, but in Gnome you have to open up the file manager, unmount the drive, close the file manager again, etc.
The ssh agent on Gnome doesn t start up for a Wayland session, though this is easily enough worked around.
The reason I completely soured on Gnome is that after using it for awhile, I noticed my laptop fans spinning up. One core would be constantly busy. It was busy with a kworker events task, something to do with sound events. Logging out would resolve it. I believe it to be a Gnome shell issue. I could find no resolution to this, and am unwilling to tolerate the decreased battery life this implies.
The Gnome summary: it looks nice out of the box, but you quickly realize that this is something of a paper-thin illusion when you try to actually use it regularly.
KDE
The KDE experience on Wayland was a little bit opposite of Gnome. While with Gnome, things start out looking great but you realize there are some serious issues (especially battery-eating), with KDE things start out looking a tad rough but you realize you can trivially fix them and wind up with a very solid system.
Compared to Gnome, KDE never had a battery-draining problem. It will show me estimated battery time remaining if I want it to. It will do whatever I want it to when I insert a USB drive. It doesn t muck with my CPU power settings, and lets me easily define on AC vs on battery settings for things like suspend when idle.
KDE supports fractional scaling, to any arbitrary setting (even with the gsettings thing above, Gnome still only supports it in 25% increments). Then the question is what to do with X11-only applications. KDE offers two choices. The first is Scaled by the system , which is also the only option for Gnome. With that setting, the X11 apps effectively run natively at 100% and then are scaled up within Wayland, giving them a blurry appearance on HiDPI displays. The advantage is that the scaling happens within Wayland, so the size of the app will always be correct even when the Wayland scaling factor changes. The other option is Apply scaling themselves , which uses native X11 scaling. This lets most X11 apps display crisp and sharp, but then if the system scaling changes, due to limitations of X11, you ll have to restart the X apps to get them to be the correct size. I appreciate the choice, and use Apply scaling by themselves because only a few of my apps aren t Wayland-aware.
I did encounter a few bugs in KDE under Wayland:
sddm, the display manager, would be slow to stop and cause a long delay on shutdown or reboot. This seems to be a known issue with sddm and Wayland, and is easily worked around by adding a systemd TimeoutStopSec.
Konsole, the KDE terminal emulator, has weird display artifacts when using fractional scaling under Wayland. I applied some patches and rebuilt Konsole and then all was fine.
The Bismuth tiling extension has some pretty weird behavior under Wayland, but a 1-character patch fixes it.
On Debian, KDE mysteriously installed Pulseaudio instead of Debian s new default Pipewire, but that was easily fixed as well (and Pulseaudio also works fine).
Conclusions
I m sticking with KDE. Given that I couldn t figure out how to stop Gnome from deciding to eat enough battery to make my fan come on, the decision wasn t hard. But even if it weren t for that, I d have gone with KDE. Once a couple of things were patched, the experience is solid, fast, and flawless. Emacs (my main X11-only application) looks great with the self-scaling in KDE. Gimp, which I use occasionally, was terrible with the blurry scaling in Gnome.
Update: Corrected the gsettings command
After watching My life in git, after subversion, after CVS. from DebConf, I ve realized it s been a while since I talked about the way I keep everything1 I do in git, and I don t think I ve ever done it online, so it looked like a good time for a blog post.
Beyond git itself (of course), I use a few git-related programs:
myrepos (also known as mr) to manage multiple git repositories with one command;
and some programs that don t use git directly, but easily interact with it:
ansible to keep track of the system configuration of all machines;
lesana as a project tracker and journal and to inventory the things made of atoms that are hard 2 to store in git.
All of these programs are installed from Debian packages, on stable (plus rarely backports) or testing, depending on the machine.
I m also grateful to the vcs-home people, who wrote most of the tools I use, and sometimes hang around their IRC channel.
And now, on to what I m actually doing.
With the git repositories I ve decided to err for too much granularity rather than too little3, so of course each project has its own repository, and so do different kinds of media files, dot-files that are related to different programs etc.
Most of the repositories are hosted on two gitolite servers: one runs on the home server, for stuff that should remain private, and the other one is on my VPS for things that are public (or may become public in the future), and also has a web interface with cgit. Of course things where I m collaborating with other people are sometimes hosted elsewhere, mostly on salsa, sourcehut or on $DAYJOB related gitlab instances.
The .mr directory is where everything is managed: I don t have a single .mrconfig file but a few different ones, that in turn load all files in a directory with the same name:
collections.mr for the media file annexes and inventories (split into different files, so that computers with little disk space can only get the inventories);
private.mr for stuff that should only go on my own personal machine, not on shared ones;
projects.mr for the actual projects, with different files for the kinds of projects (software, docs, packaging, crafts, etc.);
setup.mr with all of the vcsh repositories, including the one that tracks the mr files (I ll talk about the circular dependency later);
work.mr for repositories that are related to $DAYJOB.
Then there are the files in the .mr/machines directory, each one of which has the list of repositories that should be on every specific machine, including a generic workstation, but also specific machines such as e.g. the media center which has a custom set of repositories.
The dot files from my home directory are kept in vcsh, so that it s easy to split them out into different repositories, and I m mostly used the simplest configuration described in the 30 Second How-to in its homepage; vcsh gives some commands to work on all vcsh repositories at the same time, but most of the time I work on a single repository, and use mr to act on more than one repo.
The media collections are also pretty straightforward git-annex repositories, one for each kind of media (music, movies and other videos, e-books, pictures, etc.) and I don t use any auto-syncing features but simply copy and move files around between clones with the git annex copy, git annex move and git annex get commands.
There isn t much to say about the project repositories (plain git), and I think that the way I use my own program lesana for inventories and project tracking is worth an article of its own, here I ll just say that the file format used has been designed (of course) to work nicely with git.
On every machine I install etckeeper so that there is a history of the changes in the /etc directory, but that s only a local repository, not stored anywhere else, and is used mostly in case something breaks with an update or in similar situation. The authoritative source for the configuration of each machine is an ansible playbook (of course saved in git) which can be used to fully reconfigure the machine from a bare Debian installation.
When such a reconfiguration from scratch happens, it will be in two stages: first a run of ansible does the system-wide configuration (including installing packages, creating users etc.), and then I login on the machine and run mr to set up my own home. Of course there is a chicken-and-egg problem in that I need the mr configuration to know where to get the mr configuration, and that is solved by having setup two vcsh repositories from an old tarball export: the one with the ssh configuration to access the repositories and the one with the mr files.
So, after a machine has been configured with ansible what I ll actually do is to login, use vcsh pull to update those two repositories and then run mr to checkout everything else.
And that s it, if you have questions on something feel free to ask me on the fediverse or via email (contacts are in the about page)
Update (2023-09-12 17:00ish): The ~/.mr directory is not special for mr, it s just what I use and then I always run mr -c ~/.mr/some/suitable/file.mr, with the actual file being different whether I m registering a new repo or checking out / updating them. I could include some appropriate ~/.mr/machines/some_machine.mr in ~/.mrconfig, but I ve never bothered to do so, since it wouldn t cover all usecases anyway. Thanks to the person on #vcs-home@OFTC who asked me the question :)
At least, everything that I made that is made of bits, and a diary and/or inventory of the things made of atoms.
until we get a working replicator, I guess :D
in time I ve consolidated a bit some of the repositories, e.g. merging the repositories for music from different sources (CD rips, legal downloads, etc.) into a single repository, but that only happened a few times, and usually I m fine with the excess of granularity.
Some decades back, when I d buy a new PC, it would unlock new capabilities. Maybe AGP video, or a PCMCIA slot, or, heck, sound.
Nowadays, mostly new hardware means things get a bit faster or less crashy, or I have some more space for files. It s good and useful, but sorta meh.
Not this purchase.
Cory Doctorow wrote about the Framework laptop in 2021:
There s no tape. There s no glue. Every part has a QR code that you can shoot with your phone to go to a service manual that has simple-to-follow instructions for installing, removing and replacing it. Every part is labeled in English, too!
The screen is replaceable. The keyboard is replaceable. The touchpad is replaceable. Removing the battery and replacing it takes less than five minutes. The computer actually ships with a screwdriver.
Framework had been on my radar for awhile. But for various reasons, when I was ready to purchase, I didn t; either the waitlist was long, or they didn t have the specs I wanted.
Lately my aging laptop with 8GB RAM started OOMing (running out of RAM). My desktop had developed a tendency to hard hang about once a month, and I researched replacing it, but the cost was too high to justify.
But when I looked into the Framework, I thought: this thing could replace both. It is a real shift in perspective to have a laptop that is nearly as upgradable as a desktop, and can be specced out to exactly what I wanted: 2TB storage and 64GB RAM. And still cheaper than a Macbook or Thinkpad with far lower specs, because the Framework uses off-the-shelf components as much as possible.
Cory Doctorow wrote, in The Framework is the most exciting laptop I ve ever broken:
The Framework works beautifully, but it fails even better Framework has designed a small, powerful, lightweight machine it works well. But they ve also designed a computer that, when you drop it, you can fix yourself. That attention to graceful failure saved my ass.
I like small laptops, so I ordered the Framework 13. I loaded it up with the 64GB RAM and 2TB SSD I wanted. Frameworks have four configurable ports, which are also hot-swappable. I ordered two USB-C, one USB-A, and one HDMI. I put them in my preferred spots (one USB-C on each side for easy docking and charging). I put Debian on it, and it all Just Worked. Perfectly.
Now, I orderd the DIY version. I hesitated about this I HATE working with laptops because they re all so hard, even though I KNEW this one was different but went for it, because my preferred specs weren t available in a pre-assembled model.
I m glad I did that, because assembly was actually FUN.
I got my box. I opened it. There was the bottom shell with the motherboard and CPU installed. Here are the RAM sticks. There s the SSD. A minute or two with each has them installed. Put the bezel on the screen, attach the keyboard it has magnets to guide it into place and boom, ready to go. Less than 30 minutes to assemble a laptop nearly from scratch. It was easier than assembling most desktops.
So now, for the first time, my main computing device is a laptop. Rather than having a desktop and a laptop, I just have a laptop. I ll be able to upgrade parts of it later if I want to. I can rearrange the ports. And I can take all my most important files with me. I m quite pleased!
After cycling the Northcape 4000 (from Italy to northern Norway) last year,
I signed up for the transcontinental race
this year.
The Transcontinental is bikepacking race across Europe, self-routed (but with some mandatory checkpoints), unsupported and
with a distance of usually somewhere around 4000 km. The cut-off time is 15 days, with the winner usually taking 7-10 days.
This year, the route went from Belgium to Thessaloniki in Greece, with control points in northern Italy, Slovenia, Albania and
Meteora (Greece).
The event was great - it was well organised and communication was a lot better than at the Northcape. It did feel
very different from the Northcape, though, being a proper race. Participants
are not allowed to draft off each other or help each other, though a quick chat
here or there as you pass people is possible, or when you re both stopped at a
shop or control point.
My experience
The route was beautiful - the first bit through France was a bit monotonic, but
especially the views in the alps were amazing. Like with other long events,
the first day or two can be hard but once you get into the rhythm of things
it s a lot easier.
From early on, I lost a lot of time. We started in the rain, and I ran
several flats in a row, just 4 hours in. In addition to that, the thread on my pump
had worn so it wouldn t fit on some of my spare tubes, and my tubes were all
TPU - which are hard to patch. So at 3 AM I found myself by the side of an N-road in
France without any usable tubes to put in my rear wheel. I ended up walking 20km
to the nearest town with a bike shop, where they fortunately had good old butyl
tubes and a working pump. But overall, this cost me about 12 hours in total.
In addition to that, my time management wasn t great. On previous rides, I d usually
gotten about 8 hours of sleep per night while staying in hotels. On the
transcontinental I had meant to get less sleep but still stay in hotels most
night, but I found that not all hotels accomodated
well for that - especially with a bike. So I ended up getting more sleep than I
had intended, and spending more time off the bike than I had planned - close to
11 or 12 hours per day. I hadn t scheduled much time off work after the finish either, so
arriving in Greece late wasn t really an option.
And then, on an early morning in Croatia (about 2000km in) in heavy fog, I rode into a kerb at 35 km/h, bending
the rim of my front wheel (but fortunately not coming off my bike). While I probably would have been able to continue with
a replacement wheel (and mailing the broken one home), that would have taken
another day to sort out and I almost certainly wouldn t have been
able to source a new dynamo wheel in Croatia - which would have made night time riding a lot harder.
So I decided to scratch and take the train home from Zagreb.
Overall, I really enjoyed the event and I think I ve learned some useful
lessons. I ll probably try again next year.
DebConf23, the 24th annual Debian Developer Conference,
is taking place in Kochi, India from September 10th to 17th, 2023.
Debian contributors from all over the world have come together at Infopark, Kochi
to participate and work in a conference exclusively run by volunteers.
Today the main conference starts with over 373 expected attendants and 92
scheduled activities, including 45-minute and 20-minute talks, Bird of a Feather
("BoF") team meetings, workshops, a job fair, as well as a variety of other
events.
The full schedule is updated each
day, including activities planned ad-hoc by attendees over the course of the
conference.
If you would like to engage remotely, you can follow the video streams
available from the DebConf23 website for the events happening in the three talk
rooms: Anamudi, Kuthiran and Ponmudi. Or you can join the conversations
happening inside the talk rooms via the OFTC IRC network
in the #debconf-anamudi, #debconf-kuthiran, and the #debconf-ponmudi
channels. Please also join us in the #debconf channel for common discussions
related to DebConf.
You can also follow the live coverage of news about DebConf23 provided by our
micronews service or the @debian profile on your
favorite social network.
DebConf is committed to a safe and welcoming environment for all participants.
Please see our Code of Conduct page on the DebConf23 website for more
information on this.
Debian thanks the commitment of numerous sponsors to support DebConf23,
particularly our Platinum Sponsors: Infomaniak, Proxmox and Siemens.
~
/usr-merge work, by Helmut Grohne, et al.
Given that we now have consensus on moving forward by moving aliased files
from / to /usr, we will also run into the problems that the file move
moratorium was meant to prevent. The way forward is detecting them early and
applying workarounds on a per-package basis. Said detection is now automated
using the Debian Usr Merge Analysis Tool.
As problems are reported to the bug tracking system, they are connected to the
reports if properly usertagged. Bugs and patches for problem categories
DEP17-P2 and DEP17-P6 have been filed.
After consensus has been reached
on the bootstrapping matters, debootstrap has been
changed to swap the initial unpack and merging
to avoid unpack errors due to pre-existing links. This is a precondition for
having base-files install the aliasing symbolic links eventually.
It was identified that the root filesystem used by the Debian installer is
still unmerged and a
change has been proposed.
debhelper was changed to
recognize systemd units installed to /usr.
A discussion with the CTTE and release team on repealing the moratorium has
been initiated.
Salsa CI work, by Santiago Ruano Rinc n
August was a busy month in the Salsa CI world. Santiago reviewed and merged a
bunch of MRs that have improved the project in different aspects:
The aptly job got two MRs from Philip Hands.
With the first one,
the aptly now can export a couple of variables in a dotenv file,
and with the second,
it can include packages from multiple artifact directories. These MRs bring the
base to improve how to test reverse dependencies with Salsa CI. Santiago is
working on documenting this.
As a result of the
mass bug filing done in August,
Salsa CI now includes a job to test how a package builds twice in a row. Thanks
to the MRs of Sebastiaan Couwenberg
and Johannes Schauer Marin Rodrigues.
Last but not least, Santiago helped Johannes Schauer Marin Rodrigues to
complete the support for arm64-only pipelines.
DebConf23 lead-up, by Stefano Rivera
Stefano wears a few hats in the DebConf organization and in the lead up to the
conference in mid-September, they ve all been quite busy.
As one of the treasurers of DebConf 23, there has been a final budget update,
and quite a few payments to coordinate from Debian s Trusted Organizations. We
try to close the books from the previous conference at the next one, so a push
was made to get DebConf 22 account statements out of TOs and record them in the
conference ledger.
As a website developer, we had a number of registration-related tasks, emailing
attendees and trying to estimate numbers for food and accommodation.
As a conference committee member, the job was mostly taking calls and helping
the local team to make decisions on urgent issues. For example, getting
conference visas issued to attendees required getting political approval from
the Indian government. We only discovered the full process for this too late to
clear some complex cases, so this required some hard calls on skipping some
countries from the application list, allowing everyone else to get visas in
time. Unfortunate, but necessary.
Miscellaneous contributions
Rapha l Hertzog updated
gnome-shell-extension-hamster
to a new upstream git snapshot that is compatible with GNOME Shell 44 that
was recently uploaded to Debian unstable/testing. This extension makes it
easy to start/stop tracking time with
Hamster Time Tracker.
Very handy for consultants like us who are billing their work per hour.
Rapha l also updated zim to the latest
upstream release (0.74.2). This is a desktop wiki that can be very useful
as a note-taking tool to build your own personal knowledge base or even to
manage your personal todo lists.
Utkarsh helped the local team and the bursary team with some more DebConf
activities and helped finalize the data.
Thorsten tried to update package hplip.
Unfortunately upstream added some new compressed files that need to appear
uncompressed in the package. Even though this sounded like an easy task,
which seemed to be already implemented in the current debian/rules, the new
type of files broke this implementation and made the package no longer
buildable. The problem has been solved and the upload will happen soon.
Helmut sent 7 patches for cross build failures. Since dpkg-buildflags now
defaults to issue arm64-specific compiler flags, more care is needed to
distinguish between build architecture flags and host architecture flags than
previously.
Stefano pushed the final bit of the tox 4 transition over the line in Debian,
allowing dh-python and tox 4 to migrate to testing. We got caught up in a few
unusual bugs in tox and the way we run it in Debian package building (which
had to change with tox 4). This resulted in a couple of patches upstream.
Stefano visited Haifa, Israel, to see the proposed DebConf 24 venue and meet
with the local team. While the venue isn t committed yet, we have high hopes
for it.
Welcome to the August 2023 report from the Reproducible Builds project!
In these reports we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries.
The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. If you are interested in contributing to the project, please visit our Contribute page on our website.
The move has generated a fair amount of push back among developers who worry about its future legal and technical implications, along with a potential for supply chain attacks, should the maintainer account publishing these binaries be compromised.
[ ] an overview about reproducible builds, the past, the presence and the future. How it started with a small [meeting] at DebConf13 (and before), how it grew from being a Debian effort to something many projects work on together, until in 2021 it was mentioned in an executive order of the president of the United States. (HTML slides)
Reproducible Builds Summit
Just another reminder that our upcoming Reproducible Builds Summit is set to take place from October 31st November 2nd 2023 in Hamburg, Germany.
Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field.
If you re interested in joining us this year, please make sure to read the event page, the news item, or the invitation email that Mattia Rizzolo sent out, which have more details about the event and location.
We are also still looking for sponsors to support the event, so do reach out to the organizing team if you are able to help. (Also of note that PackagingCon 2023 is taking place in Berlin just before our summit, and their schedule has just been published.)
Vagrant Cascadian on the Sustain podcast
Vagrant Cascadian was interviewed on the SustainOSS podcast on reproducible builds:
Vagrant walks us through his role in the project where the aim is to ensure identical results in software builds across various machines and times, enhancing software security and creating a seamless developer experience. Discover how this mission, supported by the Software Freedom Conservancy and a broad community, is changing the face of Linux distros, Arch Linux, openSUSE, and F-Droid. They also explore the challenges of managing random elements in software, and Vagrant s vision to make reproducible builds a standard best practice that will ideally become automatic for users. Vagrant shares his work in progress and their commitment to the last mile problem.
Website updates
Rahul Bajaj updated our website to add a series of environment variations related to reproducible builds [], Russ Cox added the Go programming language to our projects page [] and Vagrant Cascadian fixed a number of broken links and typos around the website [][][].
Software development
In diffoscope development this month, versions 247, 248 and 249 were uploaded to Debian unstable by Chris Lamb, who also added documentation for the new specialize_as method and expanding the documentation of the existing specialize as well []. In addition, Fay Stegerman added specialize_as and used it to optimise .smali comparisons when decompiling Android .apk files [], Felix Yan and Mattia Rizzolo corrected some typos in code comments [,], Greg Chabala merged the RUN commands into single layer in the package s Dockerfile [] thus greatly reducing the final image size. Lastly, Roland Clobus updated tool descriptions to mark that the xb-tool has moved package within Debian [].
reprotest is our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. This month, Vagrant Cascadian updated the packaging to be compatible with Tox version 4. This was originally filed as Debian bug #1042918 and Holger Levsen uploaded this to change to Debian unstable as version 0.7.26 [].
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Testing framework
The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In August, a number of changes were made by Holger Levsen:
Disable Debian live image creation jobs until an OpenQA credential problem has been fixed. []
Run our maintenance scripts every 3 hours instead of every 2. []
Export data for unstable to the reproducible-tracker.json data file. []
Stop varying the build path, we want reproducible builds. []
Temporarily stop updating the pbuilder.tgz for Debian unstable due to #1050784. [][]
Correctly document that we are not variying usrmerge. [][]
Mark two armhf nodes (wbq0 and jtx1a) as down; investigation is needed. []
Misc:
Force reconfiguration of all Jenkins jobs, due to the recent rise of zombie processes. []
In the node health checks, also try to restart failed ntpsec, postfix and vnstat services. [][][]
System health checks:
Detect Debian live build failures due to missing credentials. [][]
Ignore specific types of known zombie processes. [][]
In addition, Vagrant Cascadian updated the scripts to use a predictable build path that is consistent with the one used on buildd.debian.org. [][]
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
FTP master
This month I accepted 347 and rejected 39 packages. The overall number of packages that got accepted was 349.
Debian LTS
This was my hundred-tenth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.
During my allocated time I uploaded:
[DLA 3548-1] qpdf security update for three CVEs
[DLA 3549-1] ring security update for 20 CVEs
The open CVE for ffmpeg was already fixed in a previous upload and could be marked as such.
I also started to work on amanda and did some work on security-master.
Last but not least I did some days of frontdesk duties and took part in the LTS meeting.
Debian ELTS
This month was the sixty-first ELTS month. During my allocated time I uploaded:
[ELA-927-1]ffmpeg update in Stretch for one CVE
[ELA-932-1]openssl1.0 update in Stretch for eight CVEs
Yeah, finally openssl1.0 was uploaded!
I also started to work on amanda, but for whatever reason the package does not build in my chroot. Why do I always choose the packages with quirks?
Last but not least I did some days of frontdesk duties.
debian-printing
This month I tried to update package hplip. Unfortunately upstream added some new compressed files that need to appear uncompressed in the package. Even though this sounded like an easy task, which seemed to be already implemented in the current debian/rules, the new type of files broke this implementation and made the package no longer buildable. There is also an RC-bug waiting that needs some love. I still hope to upload the package soon.
This work is generously funded by Freexian!
Other stuff
Unfortunately $job demanded lots of attention this month, so I only uploaded:
Due to the recent license change of Hashicorp, I am no longer willing to spend time working on their products. I therefore filed RM-bugs for golang-github-hashicorp-go-gcp-common, golang-github-hashicorp-go-tfe, golang-github-hashicorp-go-slug and golang-github-hashicorp-terraform-json.
As there seemed to be others involved in golang-github-hashicorp-terraform-svchost and golang-github-hashicorp-go-azure-helpers, I only orphaned both packages.
I hope OpenTF will be successful!
Stephen Flavall, better known as jorbs, is a Twitch streamer specializing
in strategy games and most well-known as one of the best
Slay the Spire
players in the world. Before We Go Live, subtitled
Navigating the Abusive World of Online Entertainment, is a memoir
of some of his experiences as a streamer. It is his first book.
I watch a lot of Twitch. For a long time, it was my primary form of
background entertainment. (Twitch's baffling choices to cripple their app
have subsequently made YouTube somewhat more attractive.) There are a few
things one learns after a few years of watching a lot of streamers. One
is that it's a precarious, unforgiving living for all but the most popular
streamers. Another is that the level of behind-the-scenes drama is very
high. And a third is that the prevailing streaming style has converged on
fast-talking, manic, stream-of-consciousness joking apparently designed to
satisfy people with very short attention spans.
As someone for whom that manic style is like nails on a chalkboard, I am
therefore very picky about who I'm willing to watch and rarely can
tolerate the top streamers for more than an hour. jorbs is one of the
handful of streamers I've found who seems pitched towards adults who don't
need instant bursts of dopamine. He's calm, analytical, and projects a
relaxed, comfortable feeling most of the time (although like the other
streamers I prefer, he doesn't put up with nonsense from his chat). If
you watch him for a while, he's also one of those people who makes you
think "oh, this is an interestingly unusual person." It's a bit hard to
put a finger on, but he thinks about things from intriguing angles.
Going in, I thought this would be a general non-fiction book about the
behind-the-scenes experience of the streaming industry. Before We
Go Live isn't really that. It is primarily a memoir focused on Flavall's
personal experience (as well as the experience of his business manager
Hannah) with the streaming team and company F2K, supplemented by a brief
history of Flavall's streaming career and occasional deeply personal
thoughts on his own mental state and past experiences. Along the way, the
reader learns a lot more about his thought processes and approach to life.
He is indeed a fascinatingly unusual person.
This is to some extent an expos , but that's not the most interesting part
of this book. It quickly becomes clear that F2K is the sort of parasitic,
chaotic, half-assed organization that crops up around any new business
model. (Yes, there's crypto.) People who are good at talking other
people out of money and making a lot of big promises try to follow a
startup fast-growth model with unclear plans for future revenue and hope
that it all works out and turns into a valuable company. Most of the time
it doesn't, because most of the people running these sorts of
opportunistic companies are better at talking people out of money than at
running a business. When the new business model is in gaming, you might
expect a high risk of sexism and frat culture; in this case, you would not
be disappointed.
This is moderately interesting but not very revealing if one is already
familiar with startup culture and the kind of people who start businesses
without doing any of the work the business is about. The F2K principals
are at best opportunistic grifters, if not actual con artists. It's not
long into this story before this is obvious. At that point, the main
narrative of this book becomes frustrating; Flavall recognizes the
dysfunction to some extent, but continues to associate with these people.
There are good reasons related to his (and Hannah's) psychological state,
but it doesn't make it easier to read. Expect to spend most of the book
yelling "just break up with these people already" as if you were reading
Captain Awkward letters.
The real merit of this book is that people are endlessly fascinating,
Flavall is charmingly quirky, and he has the rare mix of the introspection
that allows him to describe himself without the tendency to make his
self-story align with social expectations. I think every person is
intriguingly weird in at least some ways, but usually the oddities are
smoothed away and hidden under a desire to present as "normal" to the rest
of society. Flavall has the right mix of writing skill and a willingness
to write with direct honesty that lets the reader appreciate and explore
the complex oddities of a real person, including the bits that at first
don't make much sense.
Parts of this book are uncomfortable reading. Both Flavall and his
manager Hannah are abuse survivors, which has a lot to do with their
reactions to their treatment by F2K, and those reactions are both tragic
and maddening to read about. It's a good way to build empathy for why
people will put up with people who don't have their best interests at
heart, but at times that empathy can require work because some of the
people on the F2K side are so transparently sleazy.
This is not the sort of book I'm likely to re-read, but I'm glad I read it
simply for that time spent inside the mind of someone who thinks very
differently than I do and is both honest and introspective enough to give
me a picture of his thought processes that I think was largely accurate.
This is something memoir is uniquely capable of doing if the author
doesn't polish all of the oddities out of their story. It takes a lot of
work to be this forthright about one's internal thought processes, and
Flavall does an excellent job.
Rating: 7 out of 10
The release notes for Trisquel 11.0 Aramo mention support for POWER and ARM architectures, however the download area only contains links for x86, and forum posts suggest there is a lack of instructions how to run Trisquel on non-x86.
Since the release of Trisquel 11 I have been busy migrating x86 machines from Debian to Trisquel. One would think that I would be finished after this time period, but re-installing and migrating machines is really time consuming, especially if you allow yourself to be distracted every time you notice something that Really Ought to be improved. Rabbit holes all the way down. One of my production machines is running Debian 11 bullseye on a Talos II Lite machine from Raptor Computing Systems, and migrating the virtual machines running on that host (including the VM that serves this blog) to a x86 machine running Trisquel felt unsatisfying to me. I want to migrate my computing towards hardware that harmonize with FSF s Respects Your Freedom and not away from it. Here I had to chose between using the non-free software present in newer Debian or the non-free software implied by most x86 systems: not an easy chose. So I have ignored the dilemma for some time. After all, the machine was running Debian 11 bullseye , which was released before Debian started to require use of non-free software. With the end-of-life date for bullseye approaching, it seems that this isn t a sustainable choice.
There is a report open about providing ppc64el ISOs that was created by Jason Self shortly after the release, but for many months nothing happened. About a month ago, Luis Guzm n mentioned an initial ISO build and I started testing it. The setup has worked well for a month, and with this post I want to contribute instructions how to get it up and running since this is still missing.
The setup of my soon-to-be new production machine:
Talos II Lite
POWER9 18-core v2 CPU
Inter-Tech 4U-4410 rack case with ASPOWER power supply
8x32GB DDR4-2666 ECC RDIMM
HighPoint SSD7505 (the Rocket 1504 or 1204 would be a more cost-effective choice, but I re-used a component I had laying around)
PERC H700 aka LSI MegaRAID 2108 SAS/SATA (also found laying around)
Sadly, no hash checksums or OpenPGP signatures are published.
Power off your device, insert the USB stick, and power it up, and you see a Petitboot menu offering to boot from the USB stick. For some reason, the "Expert Install" was the default in the menu, and instead I select "Default Install" for the regular experience. For this post, I will ignore BMC/IPMI, as interacting with it is not necessary. Make sure to not connect the BMC/IPMI ethernet port unless you are willing to enter that dungeon. The VGA console works fine with a normal USB keyboard, and you can chose to use only the second enP4p1s0f1 network card in the network card selection menu.
If you are familiar with Debian netinst ISO s, the installation is straight-forward. I complicate the setup by partitioning two RAID1 partitions on the two NVMe sticks, one RAID1 for a 75GB ext4 root filesystem (discard,noatime) and one RAID1 for a 900GB LVM volume group for virtual machines, and two 20GB swap partitions on each of the NVMe sticks (to silence a warning about lack of swap, I m not sure swap is still a good idea?). The 3x18TB disks use DM-integrity with RAID1 however the installer does not support DM-integrity so I had to create it after the installation.
There are two additional matters worth mentioning:
Selecting the apt mirror does not have the list of well-known Trisquel mirrors which the x86 installer offers. Instead I have to input the archive mirror manually, and fortunately the archive.trisquel.org hostname and path values are available as defaults, so I just press enter and fix this after the installation has finished. You may want to have the hostname/path of your local mirror handy, to speed things up.
The installer asks me which kernel to use, which the x86 installer does not do. I believe older Trisquel/Ubuntu installers asked this question, but that it was gone in aramo on x86. I select the default linux-image-generic which gives me a predictable 5.15 Linux-libre kernel, although you may want to chose linux-image-generic-hwe-11.0 for a more recent 6.2 Linux-libre kernel. Maybe this is intentional debinst-behaviour for non-x86 platforms?
I have re-installed the machine a couple of times, and have now finished installing the production setup. I haven t ran into any serious issues, and the system has been stable. Time to wrap up, and celebrate that I now run an operating system aligned with the Free System Distribution Guidelines on hardware that aligns with Respects Your Freedom Happy Hacking indeed!
Another busy week in the KDE snap world. Most of the release-service apps are in candidate channel waiting to be tested. Testing is the bottle neck in the process, so I am trying something new and calling for help! Please test your favorite apps and report on https://discuss.kde.org/t/all-things-snaps-questions-concerns-praise/ any issues and which apps tested. Thanks!
There are some very big fixes in this release:
Desktop file defined so xdg-desktop-portals will now work.
Print support in many apps where it made sense. Please let me know if I missed one.
The KF6 content pack is coming along nicely using qt-framework-sdk snap!
Qt5 content snap using KDE patch set is nearly complete!
I believe I have a solution for our PIM applications by creating an Akondai dbus provider snap and setting all the PIM applications as consumers. I am waiting for manual review to pass.
I have a pile of new applications waiting for reserved name approvals. Igor has pinged the relevant people to speed this normally quick process up.
The pushback on per repository snapcraft files has stopped, so I have begun the process, which will take some time. This is a huge step in automating snap builds and cutting down my manual work so I can do more exciting things like plasma snaps!
Some big news on my project, a big thank you goes out to Kevin Ottens for reaching out, his company does exactly what I need to move it forward. I will update as we hash out the details, but it looks like my project isn t dead after all!
I know many have asked Why haven t you given up already?? The answer in short, I am stubborn. I refuse to give up on anything until I am given a good reason to. When I started my path in computers oh so many years ago, you would be surprised how many people told me to give it up, you ll never make it as a woman. Challenge accepted. Here I am, still going strong. When I want something, I go get it, no matter what it takes!
I still need to have a ( somewhat desperate ) call for donations. This will hopefully end soon, but for now, please consider donating to my September survival fund! Please share with anyone you that may find my work useful in any way. Thanks for your consideration
PS: Debian uploads for bubble-gum are moving along. Please if you have any packaging you need done in Debian proper, let me know and I will get on it, time allowing of course.
Building a mirror for rpm-based distributions. I've already described in brief how I built a mirror that currently mirrors Debian and Ubuntu on a daily basis. That was relatively straightforward given that I know how to install Debian and configure a basic system without a GUI and the ftpsync scripts are well maintained, I can pull some archives and get one pushed to me such that I've always got up to date copies of Debian and Ubuntu.I wanted to do something similar using Rocky Linux to pull in archives for Almalinux, Rocky Linux, CentOS, CentOS Stream and (optionally) Fedora.(This was originally set up using Red Hat Enterprise Linux on a developer's subscription and rebuilt using Rocky Linux so that the machine could be passed on to someone else if necessary. Red Hat 9.1 has moved to x86_64v2 - on the machine I have (HP Microserver gen 8) 9.1 it fails immediately. It has been rebuilt to use Rocky 8.8).This is a minimal install of Rocky as console only - the machine it's on only has 4G of memory so won't run a GUI reliably. It will run Cockpit so can be remotely administered. One user to run everything - mirror. Minimal install of Rocky 8.7 from DVD .iso. SELinux is enabled, SSH works for remote access. SELinux had to be tweaked to allow /srv/ the appropriate permissions to be served by nginx. /srv is a large LVM volume rather than a RAID 6 - I didn't have enough disks Adding nginx, enabling Cockpit and editing the Rocky Linux mirroring scripts resulted in something straightforward to reproduce.
nginx
I cheated and stole large parts of my Debian config. The crucial part to remember is that there is no autoindexing by default and I had to dig to find the correct configuration snippet.
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / autoindex on; autoindex_exact_size off; autoindex_format html; autoindex_localtime off; # First attempt to serve request as file, then # as directory, then fall back to displaying a 404. try_files $uri $uri/ =404;
#!/bin/env bash # # mirrorsync - Synchronize a Rocky Linux mirror # By: Dennis Koerner <koerner@netzwerge.de> # # The latest version of this script can be found at: # https://github.com/rocky-linux/rocky-tools # # Please read https://docs.rockylinux.org/en/rocky/8/guides/add_mirror_manager # for further information on setting up a Rocky mirror. # # Copyright (c) 2021 Rocky Enterprise Software Foundation
This is a very long script in total. Crucial parts I changed only listed the mirror to pull from and the place to put it.
# A complete list of mirrors can be found at # https://mirrors.rockylinux.org/mirrormanager/mirrors/Rocky src="mirrors.vinters.com::rocky"
# Your local path. Change to whatever fits your system. # $mirrormodule is also used in syslog output. mirrormodule="rocky-linux" dst="/srv/$ mirrormodule "
sent 606,565 bytes received 38,808,194,155 bytes 44,839,746.64 bytes/sec total size is 1,072,593,052,385 speedup is 27.64 End: Fri 27 Jan 2023 08:27:49 GMT fullfiletimelist-rocky unchanged. Not updating at Fri 27 Jan 2023 22:13:16 GMT fullfiletimelist-rocky unchanged. Not updating at Sat 28 Jan 2023 08:13:16 GMT
It was essentially easier to store fullfiletimelist-rocky in /home/mirror than anywhere else.
Very similar small modifications to the Rocky mirroring scripts were used to mirror the other distributions I'm mirroring. (Almalinux, CentOS, CentOS Stream, EPEL and Rocky Linux).
Better In My Day, from the aforementioned Pastoral. This track and this
album are, I think, less accessible, more challenging than the
re-interpreted material. That's not a bad thing: I'm still working on
digesting it! This is one of the more abrasive, confrontational tracks.
Better In My Day by Gazelle Twin
I am Shell I am Bone, from way back to her first release, The Entire
City in 2011. Composed, recorded, self-produced, self-released. It's
remarkable to me how different each phase of GT's work are to one another.
This album evokes a strong sense of atmosphere and place to me. There's a
hint of possible influence of Joy Division's Unknown Pleasures (or New
Order's Movement) in places. The b-side to this song is a cover of Joy
Division's The Eternal.
I Am Shell I Am Bone by Gazelle Twin
GT re-issued This Entire City in 2022 along with a companion-piece EP of
newly-released material from the same era, The Wastelands. This isn't
cutting room floor stuff, though, as evidenced by the strength of my final
pick, Hole in my Heart.
Hole In My Heart by Gazelle Twin
It's hard to pick just five tracks when doing these (that's the point I
suppose). I note that I haven't picked anything from her wide-ranging
soundtrack work: her last three or four of her releases have been soundtrack
works, released on the well-respected UK label
Invada, as will be her forthcoming album. You can
find all the released stuff on Gazelle Twin's Bandcamp
Page.
Work involves supporting Windows (there's a lot of specialised hardware design software that's only supported under Windows, so this isn't really avoidable), but also involves git, so I've been working on extending our support for hardware-backed SSH certificates to Windows and trying to glue that into git. In theory this doesn't sound like a hard problem, but in practice oh good heavens.
Git for Windows is built on top of msys2, which in turn is built on top of Cygwin. This is an astonishing artifact that allows you to build roughly unmodified POSIXish code on top of Windows, despite the terrible impedance mismatches inherent in this. One is that until 2017, Windows had no native support for Unix sockets. That's kind of a big deal for compatibility purposes, so Cygwin worked around it. It's, uh, kind of awful. If you're not a Cygwin/msys app but you want to implement a socket they can communicate with, you need to implement this undocumented protocol yourself. This isn't impossible, but ugh.
But going to all this trouble helps you avoid another problem! The Microsoft version of OpenSSH ships an SSH agent that doesn't use Unix sockets, but uses a named pipe instead. So if you want to communicate between Cygwinish OpenSSH (as is shipped with git for Windows) and the SSH agent shipped with Windows, you need something that bridges between those. The state of the art seems to be to use npiperelay with socat, but if you're already writing something that implements the Cygwin socket protocol you can just use npipe to talk to the shipped ssh-agent and then export your own socket interface.
And, amazingly, this all works? I've managed to hack together an SSH agent (using Go's SSH agent implementation) that can satisfy hardware backed queries itself, but forward things on to the Windows agent for compatibility with other tooling. Now I just need to figure out how to plumb it through to WSL. Sigh.
There is a bit of context that needs to be shared before I get to this and would be a long one. For reasons known and unknown, I have a lot of sudden electricity outages. Not just me, all those who are on my line. A discussion with a lineman revealed that around 200+ families and businesses are on the same line and when for whatever reason the electricity goes for all. Even some of the traffic lights don t work. This affects software more than hardware or in some cases, both. And more specifically HDD s are vulnerable. I had bought an APC unit several years for precisely this, but over period of time it just couldn t function and trips also when the electricity goes out. It s been 6-7 years so can t even ask customer service to fix the issue and from whatever discussions I have had with APC personnel, the only meaningful difference is to buy a new unit but even then not sure this is an issue that can be resolved, even with that.
That comes to the issue that happens once in a while where the system fsck is unable to repair /home and you need to use an external pen drive for the same. This is my how my hdd stacks up / is on dev/sda7 /boot is on /dev/sda6, /boot/efi is on /dev/sda2 and /home is on /dev/sda8 so theoretically, if /home for some reason doesn t work I should be able drop down on /dev/sda7, unmount /dev/sda8, run fsck and carry on with my work. I tried it number of times but it didn t work. I was dropping down on tty1 and attempting the same, no dice as root/superuser getting the barest x-term. So first I tried asking couple of friends who live nearby me. Unfortunately, both are MS-Windows users and both use what are called as company-owned laptops . Surfing on those systems were a nightmare. Especially the number of pop-ups of ads that the web has become. And to think about how much harassment ublock origin has saved me over the years. One of the more interesting bits from both their devices were showing all and any downloads from fosshub was showing up as malware. I dunno how much of that is true or not as haven t had to use it as most software we get through debian archives or if needed, download from github or wherever and run/install it and you are in business. Some of them even get compiled into a good .deb package but that s outside the conversation atm. My only experience with fosshub was few years before the pandemic and that was good. I dunno if fosshub really has malware or malwarebytes was giving false positives. It also isn t easy to upload a 600 MB+ ISO file somewhere to see whether it really has malware or not. I used to know of a site or two where you could upload a suspicious file and almost 20-30 famous and known antivirus and anti-malware engines would check it and tell you the result. Unfortunately, I have forgotten the URL and seeing things from MS-Windows perspective, things have gotten way worse than before.
So left with no choice, I turned to the local LUG for help. Fortunately, my mobile does have e-mail and I could use gmail to solicit help. While there could have been any number of live CD s that could have helped but one of my first experiences with GNU/Linux was that of Knoppix that I had got from Linux For You (now known as OSFY) sometime in 2003. IIRC, had read an interview of Mr. Klaus Knopper as well and was impressed by it. In those days, Debian wasn t accessible to non-technical users then and Knoppix was a good tool to see it. In fact, think he was the first to come up with the idea of a Live CD and run with it while Canonical/Ubuntu took another 2 years to do it. I think both the CD and the interview by distrowatch was shared by LFY in those early days. Of course, later the story changes after he got married, but I think that is more about Adriane rather than Knoppix. So Vishal Rao helped me out. I got an HP USB 3.2 32GB Type C OTG Flash Drive x5600c (Grey & Black) from a local hardware dealer around similar price point. The dealer is a big one and has almost 200+ people scattered around the city doing channel sales who in turn sell to end users. Asking one of the representatives about their opinion on stopping electronic imports (apparently more things were added later to the list including all sorts of sundry items from digital cameras to shavers and whatnot.) The gentleman replied that he hopes that it would not happen otherwise more than 90% would have to leave their jobs. They already have started into lighting fixtures (LED bulbs, tubelights etc.) but even those would come in the same ban
The main argument as have shared before is that Indian Govt. thinks we need our home grown CPU and while I have no issues with that, as shared before except for RISC-V there is no other space where India could look into doing that. Especially after the Chip Act, Biden has made that any new fabs or any new thing in chip fabrication will only be shared with Five Eyes only. Also, while India is looking to generate about 2000 GW by 2030 by solar, China has an ambitious 20,000 GW generation capacity by the end of this year and the Chinese are the ones who are actually driving down the module prices. The Chinese are also automating their factories as if there s no tomorrow. The end result of both is that China will continue to be the world s factory floor for the foreseeable future and whoever may try whatever policies, it probably is gonna be difficult to compete with them on prices of electronic products. That s the reason the U.S. has been trying so that China doesn t get the latest technology but that perhaps is a story for another day.
HP USB 3.2 Type C OTG Flash Drive x5600c
For people who have had read this blog they know that most of the flash drives today are MLC Drives and do not have the longevity of the SLC Drives. For those who maybe are new, this short brochure/explainer from Kingston should enhance your understanding. SLC Drives are rare and expensive. There are also a huge number of counterfeit flash drives available in the market and almost all the companies efforts whether it s Kingston, HP or any other manufacturer, they have been like a drop in the bucket. Coming back to the topic at hand. While there are some tools that can help you to figure out whether a pen drive is genuine or not. While there are products that can tell you whether they are genuine or not (basically by probing the memory controller and the info. you get from that.) that probably is a discussion left for another day. It took me couple of days and finally I was able to find time to go Vishal s place. The journey of back and forth lasted almost 6 hours, with crazy traffic jams. Tells you why Pune or specifically the Swargate, Hadapsar patch really needs a Metro. While an in-principle nod has been given, it probably is more than 5-7 years or more before we actually have a functioning metro. Even the current route the Metro has was supposed to be done almost 5 years to the date and even the modified plan was of 3 years ago. And even now, most of the Stations still need a lot of work to be done. PMC, Deccan as examples etc. still have loads to be done. Even PMT (Pune Muncipal Transport) that that is supposed to do the last mile connections via its buses has been putting half-hearted attempts
Vishal Rao
While Vishal had apparently seen me and perhaps we had also interacted, this was my first memory of him although we have been on a few boards now and then including stackexchange. He was genuine and warm and shared 4-5 distros with me, including Knoppix and System Rescue as shared by Arun Khan. While this is and was the first time I had heard about Ventoy apparently Vishal has been using it for couple of years now. It s a simple shell script that you need to download and run on your pen drive and then just dump all the .iso images. The easiest way to explain ventoy is that it looks and feels like Grub. Which also reminds me an interaction I had with Vishal on mobile. While troubleshooting the issue, I was unsure whether it was filesystem that was the issue or also systemd was corrupted. Vishal reminded me of putting fastboot to the kernel parameters to see if I m able to boot without fscking and get into userspace i.e. /home. Although journalctl and systemctl were responding even on tty1 still was a bit apprehensive. Using fastboot was able to mount the whole thing and get into userspace and that told me that it s only some of the inodes that need clearing and there probably are some orphaned inodes. While Vishal had got a mini-pc he uses that a server, downloads stuff to it and then downloads stuff from it. From both privacy, backup etc. it is a better way to do things but then you need to laptop to access it. I am sure he probably uses it for virtualization and other ways as well but we just didn t have time for that discussion. Also a mini-pc can set you back anywhere from 25 to 40k depending on the mini-pc and the RAM and the SSD. And you need either a lappy or an Raspberry Pi with some kinda visual display to interact with the mini-pc. While he did share some of the things, there probably could have been a far longer interaction just on that but probably best left for another day.
Now at my end, the system I had bought is about 5-6 years old. At that time it only had 6 USB 2.0 drives and 2 USB 3.0 (A) drives.
The above image does tell of the various form factors. One of the other things is that I found the pendrive and its connectors to be extremely fiddly. It took me number of times fiddling around with it when I was finally able to put in and able to access the pen drive partitions. Unfortunately, was unable to see/use systemrescue but Knoppix booted up fine. I mounted the partitions briefly to see where is what and sure enough /dev/sda8 showed my /home files and folders. Unmounted it, then used $fsck -y /dev/sda8 and back in business.
This concludes what happened.
Updates Quite a bit was left out on the original post, part of which I didn t know and partly stuff which is interesting and perhaps need a blog post of their own. It s sad I won t be part of debconf otherwise who knows what else I would have come to know.
One of the interesting bits that I came to know about last week is the Alibaba T-Head T-Head TH1520 RISC-V CPU and saw it first being demoed on a laptop and then a standalone tablet. The laptop is an interesting proposition considering Alibaba opened up it s chip thing only couple of years ago. To have an SOC within 18 months and then under production for lappies and tablets is practically unheard of especially of a newbie/startup. Even AMD took 3-4 years for its first chip.It seems they (Alibaba) would be parceling them out by quarter end 2023 and another 1000 pieces/Units first quarter next year, while the scale is nothing compared to the behemoths, I think this would be more as a matter of getting feedback on both the hardware and software. The value proposition is much better than what most of us get, at least in India. For example, they are doing a warranty for 5 years and also giving spare parts. RISC-V has been having a lot of resurgence in China in part as its an open standard and partly development will be far cheaper and faster than trying x86 or x86-64. If you look into both the manufacturers, due to monopoly, both of them now give 5-8% increment per year, and if you look back in history, you would find that when more chips were in competition, they used to give 15-20% performance increment per year.
2. While Vishal did share with me what he used and the various ways he uses the mini-pc, I did have a fun speculating on what he could use it. As shared by Romane as his case has shared, the first thing to my mind was backups. Filesystems are notorious in the sense they can be corrupted or can be prone to be corrupted very easily as can be seen above . Backups certainly make a lot of sense, especially rsync.
The other thing that came to my mind was having some sort of A.I. and chat server. IIRC, somebody has put quite a bit of open source public domain data in debian servers that could be used to run either a chatbot or an A.I. or both and use that similar to how chatGPT but with much limited scope than what chatgpt uses. I was also thinking a media server which Vishal did share he does. I may probably visit him sometime to see what choices he did and what he learned in the process, if anything.
Another thing that could be done is just take a dump of any of commodity markets or any markets and have some sort of predictive A.I. or whatever. A whole bunch of people have scammed thousands of Indian users on this, but if you do it on your own and for your own purposes to aid you buy and sell stocks or whatever commodity you may fancy. After all, nowadays markets themselves are virtual.
While Vishal s mini-pc doesn t have any graphics, if it was an AMD APU mini-pc, something like this he could have hosted games in the way of thick server, thin client where all graphics processing happens on the server rather than the client. With virtual reality I think the case for the same case could be made or much more. The only problem with VR/AR is that we don t really have mass-market googles, eye pieces or headset. The only notable project that Google has/had in that place is the Google VR Cardboard headset and the experience is not that great or at least was not that great few years back when I could hear and experience the same. Most of the VR headsets say for example the Meta Quest 2 is for around INR 44k/- while Quest 3 is INR 50k+ and officially not available. As have shared before, the holy grail of VR would be when it falls below INR 10k/- so it becomes just another accessory, not something you really have to save for. There also isn t much content on that but then that is also the whole chicken or egg situation. This again is a non-stop discussion as so much has been happening in that space it needs its own blog post/article whatever.
Till later.
This post describes how to handle files that are used as assets by jobs and pipelines defined on a common gitlab-ci
repository when we include those definitions from a different project.
QNAP TS-453mini product photo
The logo for QNAP HappyGet 2 and Blizzard s StarCraft 2 side by side
Thermalright AXP120-X67, AMD Ryzen 5 PRO 5650GE, ASRock Rack X570D4I-2T, all assembled and running on a flat surface
Memtest86 showing test progress, taken from IPMI remote control window
Screenshot of PCIe 16x slot bifurcation options in UEFI settings, taken from IPMI remote control window
Internal image of Silverstone CS280 NAS build. Image stolen from 
Internal image of Silverstone CS280 NAS build. Image stolen from 
NAS build in Silverstone SUGO 14, mid build, panels removed
Silverstone SUGO 14 from the front, with hot swap bay installed
Storage SSD loaded into hot swap sled
TrueNAS Dashboard screenshot in browser window
The final system, powered up
~
PS: Debian uploads for bubble-gum are moving along. Please if you have any packaging you need done in Debian proper, let me know and I will get on it, time allowing of course.
Work involves supporting Windows (there's a lot of specialised hardware design software that's only supported under Windows, so this isn't really avoidable), but also involves git, so I've been working on extending our support for hardware-backed SSH certificates to Windows and trying to glue that into git. In theory this doesn't sound like a hard problem, but in practice oh good heavens.
There is a bit of context that needs to be shared before I get to this and would be a long one. For reasons known and unknown, I have a lot of sudden electricity outages. Not just me, all those who are on my line. A discussion with a lineman revealed that around 200+ families and businesses are on the same line and when for whatever reason the electricity goes for all. Even some of the traffic lights don t work. This affects software more than hardware or in some cases, both. And more specifically HDD s are vulnerable. I had bought an APC unit several years for precisely this, but over period of time it just couldn t function and trips also when the electricity goes out. It s been 6-7 years so can t even ask customer service to fix the issue and from whatever discussions I have had with APC personnel, the only meaningful difference is to buy a new unit but even then not sure this is an issue that can be resolved, even with that.
That comes to the issue that happens once in a while where the system fsck is unable to repair /home and you need to use an external pen drive for the same. This is my how my hdd stacks up 
The main argument as have shared before is that Indian Govt. thinks we need our home grown CPU and while I have no issues with that, as shared before except for RISC-V there is no other space where India could look into doing that. Especially after the Chip Act, Biden has made that any new fabs or any new thing in chip fabrication will only be shared with 
. Backups certainly make a lot of sense, especially