Tarisai was raised alone in Bhekina House by an array of servants and
tutors who were not allowed to touch her. Glimpses of the world were
fleeting and sometimes ended by nailed-shut windows. Her life revolved
around her rarely-seen mother, The Lady, who treats her with deep
affection but rarely offers a word of praise, instead only pushing her to
study harder. The servants whispered behind her back (but still in her
hearing) that she was not human.
At the age of seven, in a child's attempt to locate her absent mother by
sneaking out of the house, she finds her father and is told a piece of the
truth: she is the daughter of the Lady and a captive ehru, a djinn. At
the age of eleven she's sent with two guardians to Oluwan City, the
capital of Aritsar, to enter a competition she knows nothing about, for
reasons no one has ever explained.
Raybearer is a young adult fantasy novel, the first of a duology.
Like a lot of young adult novels, it is a coming of age story that follows
Tarisai from the end of her highly manipulated childhood through her
introduction to a world she was carefully never taught about. Like a lot
of young adult fantasy novels, Tarisai has some unusual abilities. What
those are, and why she has them, is perhaps less obvious than it may
appear at first.
Unlike a lot of young adult fantasy novels, Raybearer is not set in
a facsimile of Western Europe, the structure of gods and religion is not
obviously derived from Christianity or Greek or Norse mythology, and
neither Tarisai nor most of the characters of this story are white. Some
of the characters are; Ifueko draws from a grab bag of cultures that does
include European as well as African, Middle Eastern, and Asian. But the
food, the physical descriptions, the landscape, and the hair and hair
styles feel primarily African not in the sense of specific identifiable
regions, but in the same way that most fantasy feels European even if the
map isn't recognizable.
That gives this story a freshness that I found delightful. The mythology
of this world shares some similarities to standard fantasy tropes,
including a bargain with the underworld that plays a similar role to fae
bargains in some European fantasy, but it also goes in different
directions and finds atypical balances, which gave the story room to catch
me by surprise.
The magical center of this book (and series), which Tarisai is carefully
not told about until the story starts, is a system for anointing and
protecting the emperor: selection of people who swear loyalty to him and
each other and become his innermost circle, and thereby grant him magical
protection. The emperor himself is the Raybearer, possessing an artifact
that makes him invulnerable to one form of death for each member of his
council he anoints. At eleven council members, he becomes invulnerable to
anything but old age, or an attack from one of the council themselves.
As the reader learns early in the book, that last part is important.
Tarisai is an assassin; her mother's goal is for her to be selected as a
member of the council for the prince, who will become the next emperor.
But there is rather more to this system of magic than it may first appear,
in a way that adds good depth to the mythology. And there is quite a bit
more to Tarisai herself than anyone expects.
Tarisai as a protagonist follows a more typical young adult pattern, but
it's a formula that works for me. Her upbringing isolated from any other
children has left her craving connection, but it also made her
self-reliant, stubborn, and good at keeping her own counsel. One of the
things that I loved about this book is that she's not thrown into a nest
of vipers and cynical politics. Some of that is happening in the
background, but the first step of her mother's plan is for her to earn the
trust of the prince in a competition with other potential council members,
all of whom are, well, kids. They fight (some), but they also make
friends, helped along by the goal and requirement that they join a
cooperative council or be sent home. That gives the plot a more
collaborative and social feel than one would otherwise expect from the
setup. Ifueko does a great job juggling a challenging cast size by
focusing on a few kids with whom Tarisai strikes up a friendship but
giving the others distinct-enough personalities that their presence is
still felt in the story.
There are two character dynamics that stand out: Tarisai's relationship
with Prince Ekundayo, and her friendship with Sanjeet. The first carries
much of the weight of the plot, of course; Tarisai is supposed to gain his
trust and then kill him, and the reader will be unsurprised that this
takes twists and turns no one expected. But Ifueko, refreshingly, does
not reach for the stock plot development of a romance to complicate
matters, even though many of the characters expect that. To the contrary,
this is a rare story that at least hints at an acknowledgment that some
people are not interested in romance at all, and there are other forms
that mutual respect can take.
Tarisai's relationship with Sanjeet is a different type of depth: two kids
with very different histories finding a common understanding in the ways
that they were both abused, and create space for each other. It's a great
friendship that includes some deeply touching moments.
It took me a bit to get into this book, but once Tarisai starts finding
her feet and navigating her new relationships, I was engrossed. The story
takes a sharp and nasty turn that was hard to read, but Ifueko chooses to
turn it into a story of resiliency rather than survival, which makes it
much easier to read than it could have been. She also pulls off the kind
of plot that complicates and deepens the motives of the obvious villains
in a way that gives the story much greater heft, but without disregarding
the damage that they have done. I think the plot did fall apart a bit at
the end of the book, with too much quick travel and world-building
revelations at the cost of development of the relationships that were
otherwise at the center of the book, but I'm hoping the sequel will pull
those threads back together.
And it's so refreshing to read a fantasy novel of this type with a
different setting. It's not perfect: Ifueko falls back on
Planet of
the Hats regional characterization in a few places, and Songland is so
obviously Korea that it felt jarring and out of place. Christianity also
snuck its nose into the world-building tent near the end in ways that
bugged me a bit, although it was subtle enough that I think most readers
won't notice. But compared to most fantasy settings, it feels original
and fresh. More of this!
Ifueko starts this book with a wonderfully memorable dedication:
For the kid scanning fairy tales for a hero with a face like theirs.
And for the girls whose stories we compressed into pities and wonders,
triumphs and cautions, without asking, even once, for their names.
I think she was successful on both parts of that promise, and it makes for
some great reading. Recommended.
Followed by Redemptor.
Rating: 8 out of 10
Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 937 other packages on CRAN, and downloaded over 22 million times (per the partial logs from the cloud mirrors of CRAN).
This release brings another bug fix release 10.7.5 by Conrad in the long-term support 10.7.* series we started with 0.10.7.0 on September 30. As the bug fixes can come a little quicker than the desired monthly cadence CRAN aims for, we skipped a few of those release for CRAN only but of course still provide them via the Rcpp drat repo.
The full set of changes (since the last CRAN release 0.10.7.3.0) follows. It includes the nice fixes to the fields type mentioned right after the last release.
Changes in RcppArmadillo version 0.10.7.5.0 (2021-12-16)
Upgraded to Armadillo release 10.7.5
fix aliasing bug in diagmat()
fix detection of 2x2 triangular matrices
Changes in RcppArmadillo version 0.10.7.4.0 (2021-11-23)
Upgraded to Armadillo release 10.7.4
faster handling of diagonal matrices by inv_sympd(), pinv(), rank()
more robust detection of incorrect data format by .load()
Correct dimensions setting in import/export of arma::field types, protected by #define (Jonathan Berrisch in #352 fixing #351)
Add unit tests for fields both with and without new #define (Dirk)
A new release 0.4.18 of RProtoBuf arrived on CRAN earlier today. RProtoBuf provides R with bindings for the Google Protocol Buffers ( ProtoBuf ) data encoding and serialization library used and released by Google, and deployed very widely in numerous projects as a language and operating-system agnostic protocol.
This release, the first since March of last year, contains two contributed pull requests improving or extending the package, some internal maintance updating the CI setup as well as retiring an old-yet-unused stub interface for RPC, as well as an update for UCRT builds on Windows.
The following section from the NEWS.Rd file has more details.
Changes in RProtoBuf version 0.4.18 (2021-12-15)
Support string_view in FindMethodByName() (Adam Cozzette in #72).
CI use was updated first at Travis, later at GitHub and now uses r-ci (Dirk in #74 and (parts of) #76).
The (to the best of our knowledge) unused minimal RPC mechanism has been removed, retiring one method and one class as well as the import of the RCurl package (Dirk in #76).
The toJSON() method supports two (upstream) formatting toggles (Vitali Spinu in #79 with minor edit by Dirk).
Windows UCRT builds are now supported (Jeroen in #81, Dirk and Tomas Kalibera in #82).
Our project funding work continues with an active bid on the work of packaging a recent gradle in Debian. This month the bidder has been estimating the scope of the entire project.
The Grow Your Ideas project page also has some ambitious initiatives that may evolve into a funded project. The project ideas on that page range from a new wiki for Debian, a more efficient reimbursement process, and the implementation of PPAs for Debian.
We continue to looking forward to hearing about Debian project proposals from various Debian stakeholders. This month has seen work on a survey that will go out to Debian Developers to gather feedback on what they think should be the priorities for funding in the project.
Learn more about the rationale behind this initiative in this article.
Debian LTS contributors
In November 13 contributors were paid to work on Debian LTS, their reports are available below. If you re interested in participating in the LTS or ELTS teams, we welcome participation from the Debian community. Simply get in touch with Jeremiah if you are interested in participating.
Adrian Bunk did 62h out of 56h assigned for November and 6h from October.
Jeremiah Foster is coordinating/managing the LTS team did 29h (out of 10h assigned and 10h from October for LTS administration), and spent 9 hours on Projects funded directly through the project funding program.
Lee Garrett did 9 hours out 60 assigned and carried over 51h into December
Utkarsh Gupta did 30 (out of 40h assigned), thus carrying over 10h to December.
Evolution of the situation
In November we released 31 DLAs.
The security tracker currently lists 23 packages with a known CVE and the dla-needed.txt file has 16 packages needing an update.
Thanks to our sponsors
Sponsors that joined recently are in bold.
Debian security is pretty good, but there s always scope for improvement. Here are some ideas that I think could be used to improve things.
A security wizard , basically a set of scripts with support for plugins that will investigate your system and look for things that can be improved. It could give suggestions on LSMs that could be used, sysctl settings, lists of daemons running as root that possibly don t need root privs, etc. Plugins could be for different daemons, so there could be a plugin for Apache that looks for potential issues with Apache configuration. It wouldn t be possible to cover everything, but it would be possible to cover many common cases. It appears that we used to have a harden package to do some of these things which disappeared. It appears that the only remnant of that is the hardening-runtime package.
Kali Linux [1] is a distribution designed for penetration testing, I recently tried out many of it s features and I was very impressed. While I don t think that the aim should be to copy all Kali features into Debian there are probably some that are worthy of inclusion. Most Kali features run well in a VM, but the Wifi penetration testing tools need access to the hardware, so they could be a good candidate for inclusion in Debian (license permitting).
We have a Securing Debian Manual [2] that is really good. It s a little out of date and needs some contributions, it also needs to be better known.
The Security Management page of the Debian wiki [3] has links to a number of pages about improving system security. It needs some updates, it doesn t have a link to a page about SE Linux so there s some work for me to do there.
Can training help people? I would be happy to run some Debian SE Linux training sessions over Matrix or Jitsi. We can probably find people to offer training on other aspects of Linux security that are implemented in Debian if there is an audience. I don t think that I and other DDs (Debian Developers) can train everyone, but we could train people who then go on to run other training sessions and make the session notes etc available under the GPL.
There would also be some benefits to training other DDs as probably no-one has a good overview of all the security features that are supported.
Any other ideas? Feel free to comment here or start a thread on a public mailing list. If you start a mailing list discussion please email me or comment here with the URL if it s a list that I m not on so I can track it via the archives. This post was inspired by a discussion on a private list of a related topic. I think it s better to have a public discussion instead.
The first time I applied for grad school it was a bit of a lark. I was serious about it, I wanted to go, I had goals, and I am happy to be here now. However, it was a little fun. I took a white paper and crisped it up with the help of some friends who read it and provided comments. I wrote a personal statement, got some letters of recommendation, and generally felt pretty good about the process. It wasn t stressful.
The second time I applied to grad school it was a slog. For several weeks I gave up most of my free time to application writing and when I wasn t working on applications I thought about how I ought to be doing something about it. Applications are due while you re coming up on the end of your semester or end of the year push at work or both! There just isn t enough time for everything. I think even applying for grad school qualifies you to attend.
Here are some things I learned in the application process. Note, I haven t gotten in anywhere (yet), so we ll see how it pans out.
You have to do a lot of research.
This changes by field, school, and even individual faculty preference. Some possible mentors / advisors like to talk with prospective students ahead of time. On two of my applications I was explicitly asked who I spoke with in the department. On others I was asked to list faculty (in order of preference even!) that I would be interested in working with. Some schools requested in essays to list faculty. In general, it s considered important to explain why a particular school: what about that school attracts you? What resources do they have that you want to take advantage of? What research that they are doing interests you?
Faculty profiles and department websites are useful for this, but far from complete. A co-worker suggested I contact someone who, based on their profile, I shared no research interests with. However, we talked and it turns out we had things in common! In general, colleagues, peers, other academics, and friends were the best resource I had for investigating schools and potential faculty.
I also went to some Q&A sessions / office hours run by schools / departments / labs. Some really cool people wrote blog posts and / or Tweets soliciting grad students. These are great.
Sometimes the people you want to work with aren t taking grad students.
Most schools I looked at did not maintain lists of who is taking on grad students starting in the fall, but I think two did. Sometimes I contacted people and got back Sorry, not taking on anyone this year. Try $OtherProfessor. This was actually really helpful! But, it s a downer to learn you re not going to get to work with one of your academic heroes because the timing is wrong.
It s expensive and waivers might not help.
Most applications in the US cost about $100 (USD). In some fields it s recommended you apply to 8-10 schools. That s a lot of money. I think everywhere I applied had a waiver you could apply for, but I generally didn t qualify. Some waivers require you to submit days to weeks ahead of the deadline. Others require that you participate in certain professional organizations, student organizations, or live locally to the school. These might be less useful than you were hoping.
Everyone says your writing sample matters, but only five people will read it.
This is what I was told anyway. I was told that my writing sample (which was 20 pages!) will probably only be read by faculty who are thinking of taking me on as an advisee. Someone else might pick it up and read it out of curiosity. I was told stories about how the right writing sample turned eh, maybe candidates into enthusiastic yeses.
Everyone says letters of recommendation matter and it s extremely frustrating.
My recommenders are all amazing people I have so much respect and admiration for it s wild. They re people I want to be like in my academic career. They also all got their letters in the day they were due. I had a moment where I genuinely thought I was just not going to go to grad school because of how close the deadline was.
I ve been told that a great letter matters a lot, but meh letters don t hold you back as much as it sounds like they will. I ve been told that where a recommender is affiliated matters a lot.
My personal statement radically changed based on school / program.
Two of the schools I applied to had very specific requirements for their personal statements. Other schools wanted me to mostly talk about my research interests. One school had a 350 word personal statement, but a 1,000 word research statement. I wrote a research statement and a personal statement and then used them as the basis for what I submitted to different schools. Each application ended up with something pretty different, and not just because I talked about why each program was exciting to me in different ways.
I also applied to different kinds of programs, and presented myself a differently depending on the program s focus and the faculty. For the more social science heavy programs I emphasized the empirical research I did, while for others I leaned more heavily on my philosophical background and theoretical interests.
Oh, also, I was told to not use contractions. Getting rid of them in all my materials was super tedious.
Diversity Statements are becoming more common.
A few of my schools had diversity statements either required or optional. The purpose of these, I was told, is not to play some sort of marginalization Olympics, but to show that you can talk about diversity (and equity and inclusion, etc) without being offensive. It s just as valid to write about your own disability as the disability of a parent (though, ask their permission first).
Triple check what s required.
I made spreadsheets about what each school required. For example, some people wanted official transcripts, some unofficial. Some places required essays rather than generic personal statements, and I needed to build in enough time to manage that.
Ask your friends to help.
I hate asking for help, but applying for grad school made me shameless. I asked anyone I could to read over materials, proofread, or just talk with me about what I was writing. People picked up on things like you used the word technology five times in this one sentence and it looks real weird.
Some schools have projects where current grad students will even look over materials!
As a follow on to a previous blog entry of mine, Free and Open, I feel it
worthwhile to do my bit to dismantle the pseudo-science and over
simplification in the idea that gender is binary at a biological level.
TL;DR: Science simply does not support binary sexes or binary genders.
Truth is a bit more complicated.
There is certainty and there are binary answers in mathematics. Things
get less definitive in physics, certainly as soon as quantum is
broached. Processes become more of an equilibrium between states in
chemistry, never wholly one or the other. Yes, there is the oddity of
absolute zero but no experiment has yet achieved that fully. It is
accurate to describe physics as a development of applied mathematics
and to view chemistry as applied physics. Biology, at the biochemical
level, is applied chemistry. The sciences build on each other, "on the
shoulders of giants", but at each level, some certainty is lost, some
amount of uncertainty is expanded and measurements become
probabilities, proportions and percentages.
Biology is dependent on biochemistry - chemistry is how a biological
change results in a different organism. Physics is how that chemical
change occurs - temperature, pressure and physical states are inherent
to all chemical changes.
Outside laboratory constraints, few chemical reactions, especially in
organic chemistry, produce one and only one result from two or more
known reagents.
In biology, everyone is familiar with genetic mutations but a genetic
mutation only happens because a biochemical reaction (hydrogen bonding
of nucleobases) does not always produce the expected result. Every cell
division, every viral infection, there is a finite probability that a
change will occur. It might be a small number but it is never zero and
can never be dismissed. This is obvious in the current Covid pandemic -
genetic mutations result in new variants. Some variants are inviable,
some variants produce no net change in the way that the viral particles
infect adjacent cells. Sometimes, a mutation happens that changes
everything. These mutations are not mistakes - these are simply changes
with undetermined outcomes. Genetic changes are the foundation of
biodiversity and variety is what allows lifeforms of all kinds to
survive changes in environmental factors and/or changes in prevalent
diseases.
It is precisely the same in humans, particularly in one of the
principle spheres of human life that involves replicating genetic
material - the creation of gametes for sexual reproduction. Every
single time any DNA is copied, there is a finite chance that a
different base will be put in place compared to the original. Copying
genetic material is therefore non-binary. Given precisely the same
initial conditions, the result is not always predictable and the range
of how the results vary from one to another increases with every
iteration.
Let me stress that - at the molecular level, no genetic operation in
any biological lifeform has a truly binary result. Repeat that
operation sufficiently often and an unexpected result WILL inevitably
occur. It is a mathematical certainty that genetic changes will arise
by attempting precisely the same genetic operation enough times.
Genetic changes are fundamental to how lifeforms survive changing
conditions. Life would likely have died out a long time ago on this
planet if every genetic operation was perfect. Diversity is life.
Similarity leads to extinction.
Viral load is interesting at this point. Someone can be infected with a
virus, including coronavirus, by encountering a small number of viral
particles. Some viruses, it may be a few hundred, some viruses may need
a few thousand particles to infect a vulnerable host. But here's the
thing, for that host to be at risk of infecting another host, the virus
needs the host to produce billions upon billions of copies of the virus
by taking over the genetic machinery within a huge number of cells in
the host. This, as is accepted with Covid, is before the virus has been
copied enough times to produce symptoms in the host. Before those
symptoms become serious, billions more copies will be made. The numbers
become unimaginable - and that is within a single host, let alone the
265 million (and counting) hosts in the current Covid19 pandemic. It's
also no wonder that viral infections cause tiredness, the infection is
diverting huge resources to propagating itself - before even
considering the activity of the immune system.
It is idiocy of the highest order to expect all those copies to be
identical. The rise of variants is inevitable - indeed essential - in
all spheres of biology. A single viral particle is absolutely no threat
of any kind - it must first get inside and then copy the genetic
information in a host cell. This is where the complexity lies in the
definition of life itself. A virus can be considered a lifeform but it
is only able to reproduce using another, more complex, lifeform. In
truth, a viral particle does not and cannot mutate. The infected host
mutates the virus. The longer it takes that host to clear the
infection, the more mutations that host will create and then
potentially spread to others.
Now apply this to the creation of gametes in humans. With seven billion
humans, the amount of copying of genetic material is not as large as
the pandemic but it is still easy for everyone to understand that
children do not merely combine the DNA of both parents. Changes happen.
Human sexual reproduction is not as simple as 1 + 1 = 2. Sometimes, the
copying of the genetic material produces an unexpected result. Sexual
reproduction itself is non-binary. Sexual reproduction is not easy or
simple for lifeforms to adopt - the diversity which results from the
non-binary operations are exactly why so many lifeforms invest so much
energy in reproducing in this way.
Whilst many genetic changes in humans will be benign or beneficial, I d
like to take an example of a genetic disorder that results from the
non-binary nature of sex. Humans can be born with the XY phenotype -
i.e. at a genetic level, the individual has the same combination of
chromosomes as another XY individual but there are changes within the
genes in those chromosomes. We accept this, some children of blonde
parents do not have blonde hair, etc. There are also genetic changes
where an XY phenotype is not binary. Some people, who at a genetic
level would be almost identical to another person who is genetically
male, have a genetic mutation which makes it impossible for the cells
of that individual to respond to androgens (testosterone). (See
Androgen insensitivity syndrome). Genetically, that individual has an X
and a Y chromosome, just like many other individuals. However, due to a
change in how the genes on those chromosomes were copied, that
individual is biologically incapable of constructing the secondary
sexual characteristics of a male. At a genetic level, the individual
has the XY phenotype of a male. At the physical level, the individual
has all the sexual characteristics of a female and none of the sexual
characteristics of a male. The gender of that individual is not binary.
Treatment is centred on supporting the individual and minimising some
risks from the inactive genes on the Y chromosome.
Human sexual reproduction is non-binary. The results of any sexual
reproduction in humans will not always produce the binary option of
male or female. It is a lie to claim that human gender is binary. The
science is in plain view and cannot be ignored.
Identifying as non-binary is not a "cop out" - it can be a biological,
genetic, scientific fact.
Human sexuality and gender are malleable. Where genetic changes result
in symptoms, these can be ameliorated by treatment with human sex
hormones, like oestrogen and testosterone. There are valid medical uses
for anabolic steroids and hormone replacement therapies to help
individuals who, at a genetic level, have non-binary gender. These
treatments can help align the physical outer signs with the personality
and identity of the individual, whether with or without surgery. It is
unacceptable to abandon such people to suffer life long discrimination
and harassment by imposing a binary definition that has no basis in
science.
When a human being has an XY phenotype, that human being is not
necessarily male. That individual will be on a spectrum from female
(left unaffected by sex hormones in the womb, the foetus will be
female, even with an X and a Y chromosome), to various degrees of male.
So, at a genetic, biological level, it is a scientific fact that human
beings do not have binary gender. There is no evidence that this is new
to the modern era, there is no scientific basis for thinking that
copying of genetic material was somehow perfectly reliable in earlier
history, or that such mutations are specific to homo sapiens. Changes
in genetic material provide the diversity to fight infections and adapt
to changing environmental factors. Species have and will continue to go
extinct if this diversity is absent.
With that out of the way, it is no longer a stretch to encompass other
aspects of human non-binary genders beyond the known genetic syndromes
based on changes in the XY phenotype. Science has not uncovered all of
the ways that genes affect personality, behaviour, or identity.
How other, less studied, genetic changes affect the much more subtle
human facets, especially anything to do with consciousness, identity,
personality, sexuality and behaviour, is guesswork. All of these facets
can and likely are being affected by genetic factors as well as
environmental factors in an endless range of permutations. Personality
traits are a beautiful and largely unknowable blend of genes and
environment. Genetic information has a finite probability of changes at
each and every iteration. Environmental factors are more akin to chaos
theory. The idea that the results will fit into binary constructs is
laughable.
Human society puts huge emphasis on societal norms. Individuals who do
not fit into those norms suffer discrimination. The norms themselves
have evolved over time as a response to various influences on human
civilisation but most are not based on science. It is up to all humans
in that society to call out discrimination, to call for changes in the
accepted norms and support those who are marginalised. It is a
precarious balance, one that humans rarely get right, but it must be
based on an acceptance that variation is the natural state. Artificial
constraints, like binary genders, must be dismantled because human
beings and human sexual reproduction are not binary.
To those who think, "well it is for 99%", think again about Covid. 99%
(or closer to 98%) of infected humans recover without notable after
effects. That has still crippled the nations of the globe and humbled
all those who tried to deny it. Five million human beings are dead
because "most infected people recover".
Just because something only affects a proportion of human beings does
not invalidate the suffering of those humans and the discrimination
that those humans will face.
Societal norms are not necessarily correct. Religious and other
influences typically obscure and ignore scientific fact and undermine
human kindness.
The scientific truth of life on this planet is that gender is not
binary.
The more complex the lifeform, the more factors will affect where on
the spectrum any one individual will appear.
Just because we do not yet fully understand how genes affect human
personality and sexuality, does not invalidate the science that
variation is the natural order.
My previous blog about diversity is not just about male vs female, one
nationality vs another, one ethnicity compared to another. Diversity is
diverse. Diversity requires accepting that every facet of humanity is
subject to variation. That leads to tension at times, it is inevitable.
Tension against societal norms, tension against discrimination, tension
around those individuals who would abuse the tolerance of others for
their own gratification or from their own ignorance.
None of us are perfect, none of us have any of this fully sorted and
all of us will make mistakes. Personally, I try to respect those around
me. I will use whatever pronouns and other conventions that the person
requests, from their perspective and not mine. To do otherwise is to
deny the natural order and to deny the science.
Celebrate all diversity, it is the very stuff of life.
Final comments:
The discussions around (typically female) bathroom facilities often
miss the point. The concern is not about individuals who describe
themselves as non-binary. The concern is about individuals who are
fully certain of their own sexuality and who act as sexual predators
for their own gratification. These people are acting out a lie for
their own ends. The problem people are the predators, so stop blaming
the victims who are just as at risk as anyone else who identifies as
female. Maybe the best people to spot such predators are those who are
non-binary, who have had to pretend to fit into societal norms. Just as
travel can be a good antidote to racism, openness and discussion can be
a tool to undermine the lies of sexual predators and reassure those who
are justifiably fearful. There can never be a biological binary test of
gender, there can never be any scientific justification for binary
division of facilities.
Humanity itself is not binary, even life itself has blurry borders
around comas, suspended animation and locked-in syndrome. Legal
definitions of human death vary around the world.
The only common thread I have ever found is: Be kind to each other.
If you find anything above objectionable, then I can only suggest that
you reconsider the science and learn to be kind to your fellow humans.
None of us are getting out of this alive.
My degree is in pharmaceutical sciences and I practised community and
hospital pharmacy for 20 years before moving into programming. I have
direct experience of supporting people who were prescribed hormones to
transition their physical characteristics to match their personal
identity. I had a Christian upbringing but my work showed me that those
religious norms were incompatible with being kind to others, so I
rejected religion and I now consider myself a secular humanist.
Advent of Code 2021Advent of Code, for those not in the know,
is a yearly Advent calendar (since 2015) of coding puzzles many people
participate in for a plenary of reasons ranging from speed coding to code
golf with stops at learning a new language or practicing already known
ones.
I usually write boring C++, but any language and then some can be used.
There are reports of people implementing it in hardware, solving them by
hand on paper or using Microsoft Excel
so, after solving a puzzle the easy way yesterday, this time
I thought: CHALLENGE ACCEPTED! as I somehow remembered an old 2008
article about solving Sudoku with aptitude
(Daniel Burrows via archive.org as the blog is long gone)
and the good same old a package management system that can solve [puzzles] based on package dependency rules is not something that I think would be useful or worth having (Russell Coker).
Day 8 has a rather lengthy
problem description and can reasonably be approached in a bunch of
different way. One unreasonable approach might be to massage the
problem description into Debian packages and let apt help me solve the
problem (specifically Part 2, which you unlock by solving Part 1. You
can do that now, I will wait here.)
Be warned: I am spoiling Part 2 in the following, so solve it yourself
first if you are interested.
I will try to be reasonable consistent in naming things in the following
and so have chosen: The input we get are lines like acedgfb cdfbe gcdfa fbcad dab cefabd cdfgeb eafb cagedb ab cdfeb fcadb cdfeb cdbaf.
The letters are wires mixed up and connected to the segments of the
displays: A group of these letters is hence a digit (the first 10)
which represent one of the digits 0 to 9 and (after the pipe) the four
displays which match (after sorting) one of the digits which means
this display shows this digit. We are interested in which digits are
displayed to solve the puzzle. To help us we also know which segments
form which digit, we just don't know the wiring in the back.
So we should identify which wire maps to which segment!
We are introducing the packages wire-X-connects-to-Y for this which
each provide & conflict1 with the virtual packages segment-Y and
wire-X-connects.
The later ensures that for a given wire we can only pick one segment and
the former ensures that not multiple wires map onto the same segment.
As an example: wire a's possible association with segment b is
described as:
Note that we do not know if this is true! We generate packages for
all possible (and then some) combinations and hope dependency resolution
will solve the problem for us. So don't worry, the hard part will be
done by apt, we just have to provide all (im)possibilities!
What we need now is to translate the 10 digits (and 4 outputs) from
something like acedgfb into digit-0-is-eight and not, say
digit-0-is-one. A clever solution might realize that a one consists
only of two segments so a digit wiring up seven segments can not be
a 1 (and must be 8 instead), but again we aren't here to be clever:
We want apt to figure that out for us! So what we do is simply making
every digit-0-is-N (im)possible choice available as a package and
apply constraints: A given digit-N can only display one number and
each N is unique as digit so for both we deploy Provides
& Conflicts again.
We also need to reason about the segments in the digits: Each of the
digit packages gets Depends on wire-X-connects-to-Y where X is each
possible wire (e.g. acedgfb) and Y each segment forming the digit
(e.g. cf for one). The different choices for X are or'ed together,
so that either of them satisfies the Y.
We know something else too through: The segments which are not used by
the digit can not be wired to any of the Xs. We model this with
Conflicts on wire-X-connects-to-Y.
As an example: If digit-0s acedgfb would be displaying a one
(remember, it can't) the following package would be installable:
Repeat such stanzas for all 10 possible digits for digit-0 and then
repeat this for all the other nine digit-N. We produce pretty much the
same stanzas for display-0(-is-one), just that we omit the second
Provides & Conflicts from above (digit-is-one) as in the display
digits can be repeated. The rest is the same (modulo using display
instead of digit as name of course).
Lastly we create a package dubbed solution which depends on all 10
digit-N and 4 display-N all of them virtual packages apt will have
to choose an installable provider from and we are nearly done!
The resulting Packages file2 we can give to apt while requesting to
install the package solution and it will spit out not only the display
values we are interested in but also which number each digit represents
and which wire is connected to which segment. Nifty!
$ ./skip-aoc 'acedgfb cdfbe gcdfa fbcad dab cefabd cdfgeb eafb cagedb ab cdfeb fcadb cdfeb cdbaf'
[ ]
The following additional packages will be installed:
digit-0-is-eight digit-1-is-five digit-2-is-two digit-3-is-three
digit-4-is-seven digit-5-is-nine digit-6-is-six digit-7-is-four
digit-8-is-zero digit-9-is-one display-1-is-five display-2-is-three
display-3-is-five display-4-is-three wire-a-connects-to-c
wire-b-connects-to-f wire-c-connects-to-g wire-d-connects-to-a
wire-e-connects-to-b wire-f-connects-to-d wire-g-connects-to-e
[ ]
0 upgraded, 22 newly installed, 0 to remove and 0 not upgraded.
We are only interested in the numbers on the display through, so grepping
the apt output (-V is our friend here) a bit should let us end up with
what we need as calculating3 is (unsurprisingly) not a strong suit of
our package relationship language so we need a few shell commands to
help us with the rest.
$ ./skip-aoc 'acedgfb cdfbe gcdfa fbcad dab cefabd cdfgeb eafb cagedb ab cdfeb fcadb cdfeb cdbaf' -qq
5353
I have written the skip-aoc script as a testcase for apt,
so to run it you need to place it in
/path/to/source/of/apt/test/integration and built apt first, but that
is only due to my laziness. We could write a standalone script
interfacing with the system installed apt directly and in any apt
version since
~2011.
To hand in the solution for the puzzle we just need to run this on each
line of the input (~200 lines) and add all numbers together. In other
words: Behold this beautiful shell one-liner:
parallel -I ' ' ./skip-aoc ' ' -qq < input.txt paste -s -d'+' - bc
(You may want to run parallel with -P to properly grill your CPU as
that process can take a while otherwise and it still does anyhow as
I haven't optimized it at all the testing framework does a lot of
pointless things wasting time here, but we aren't aiming for the
leaderboard so )
That might or even likely will fail through as I have so far omitted a
not unimportant detail: The default APT resolver is not able to solve
this puzzle with the given problem description we need another solver!
Thankfully that is as easy as installing apt-cudf (and with it aspcud)
which the script is using via --solver aspcud to make apt hand over the
puzzle to a "proper" solver (or better: A solver who is supposed to be
good at "answering set" questions). The buildds are using this for
experimental and/or backports builds and also for installability checks
via dose3 btw, so you might have encountered it before.
Be careful however: Just because aspcud can solve this puzzle doesn't
mean it is a good default resolver for your day to day apt. One of the
reasons the default resolver has such a hard time solving this here is
that or-groups have usually an order in which the first is preferred over
every later option and so fort. This is of no concern here as all these
alternatives will collapse to a single solution anyhow, but if there are
multiple viable solutions (which is often the case) picking the "wrong"
alternative can have bad consequences. A classic example would be
exim4 postfix nullmailer. They are all MTAs
but behave very different. The non-default solvers also tend to lack
certain features like keeping track of auto-installed packages or
installing Recommends/Suggests. That said, Julian is working on
another solver
as I write this which might deal with more of these issues.
And lastly: I am also relatively sure that with a bit of massaging the
default resolver could be made to understand the problem, but I can't
play all day with this maybe some other day.
Disclaimer: Originally posted in the daily megathread
on reddit, the version here is just slightly better understandable as
I have hopefully renamed all the packages to have more conventional
names and tried to explain what I am actually doing.
No cows were harmed in this improved version, either.
If you would upload those packages somewhere, it would be good
style to add Replaces as well, but it is of minor concern for apt
so I am leaving them out here for readability.
We have generated 49 wires, 100 digits, 40 display and 1 solution
package for a grant total of 190 packages. We are also making use of
a few purely virtual ones, but that doesn't add up to many packages
in total. So few packages are practically childs play for apt given it
usually deals with thousand times more. The instability for those
packages tends to be a lot better through as only 22 of 190 packages
we generated can (and will) be installed. Britney will hate you if
your uploads to Debian unstable are even remotely as bad as this.
What we could do is introduce 10.000 packages which denote every
possible display value from 0000 to 9999. We would then need to
duplicate our 10.190 packages for each line (namespace them) and then
add a bit more than a million packages with the correct dependencies
for summing up the individual packages for apt to be able to display
the final result all by itself. That would take a while through as at
that point we are looking at working with ~22 million packages with
a gazillion amount of dependencies probably overworking every solver
we would throw at it a bit of shell glue seems the better option for
now.
This article was written by David Kalnischkies on apt-get a life and republished here by pulling it from a syndication feed. You should check there for updates and more articles about apt and EDSP.
One might think, this blog is exclusively about weirdlanguage behavior and yelling at computers Well, welcome to another episode of Jackass!
Today's opponent is Ruby, or maybe minitest , or maybe Mocha. I'm not exactly sure, but it was a rather amusing exercise and I like to share my nightmares ;)
It all started with the classical "you're using old and unmaintained software, please switch to something new".
The first attempt was to switch from the ci_reporter_minitest plugin to the minitest-ci plugin. While the change worked great for Foreman itself, it broke the reporting in Katello - the tests would run but no junit.xml was generated and Jenkins rightfully complained that it got no test results.
While investigating what the hell was wrong, we realized that Katello was already using a minitest reporting plugin: minitest-reporters. Loading two different reporting plugins seemed like a good source for problems, so I tried using the same plugin for Foreman too.
Guess what? After a bit of massaging (mostly to disable the second minitest-reporters initialization in Katello) reporting of test results from Katello started to work like a charm. But now the Foreman tests started to fail. Not fail to report, fail to actually run. WTH
The failure was quite interesting too:
test/unit/parameter_filter_test.rb:5:in block in <class:ParameterFilterTest>':
Mocha methods cannot be used outside the context of a test (Mocha::NotInitializedError)
Yes, this is a single test file failing, all others were fine.
The failing code doesn't look problematic on first glance:
The failing line (5) is mock('Example').tap and for some reason Mocha thinks it's not initialized here.
This certainly has something to do with how the various reporting plugins inject themselves, but I really didn't want to debug how to run two reporting plugins in parallel (which, as you remember, didn't expose this behavior). So the only real path forward was to debug what's happening here.
Calling the test on its own, with one of the working reporter was the first step:
Wait, what? #<Mocha::Mock: >? Shouldn't this read more like ParameterFilterTest:: as it happens for every single other test in our test suite? It definitely should! That's actually great, as it tells us that there is really something wrong with the test and the change of the reporting plugin just makes it worse.
What comes next is sheer luck. Well, that, and years of experience in yelling at computers.
We use let(:klass) to define an object called klass and this object is a Mocha::Mock that we'll use in our tests later. Now klass is a very common term in Ruby when talking about classes and needing to store them mostly because one can't use class which is a keyword. Is something else in the stack using klass and our let is overriding that, making this whole thing explode?
It was! The moment we replaced klass with klass1 (silly, I know, but there also was a klass2 in that code, so it did fit), things started to work nicely.
I really liked Tomer's comment in the PR: "no idea why, but I am not going to dig into mocha to figure that out."
Turns out, I couldn't let (HAH!) the code rest and really wanted to understand what happened there.
What I didn't want to do is to debug the whole Foreman test stack, because it is massive.
So I started to write a minimal reproducer for the issue.
All starts with a Gemfile, as we need a few dependencies:
And a test! I took the liberty to replace ActiveSupport::TestCase with Minitest::Test, as the test won't be using any Rails features and I wanted to keep my environment minimal.
Wait, I didn't change anything and it's already failing?! Fuck! I mean, cool!
But the test isn't minimal yet. What can we reduce? let is just a fancy, lazy def, right? So instead of let(:klass) we should be able to write def class and achieve a similar outcome and drop that Minitest::Spec.
$ bundle exec rake testTESTOPTS=-v
/home/evgeni/Devel/minitest-wtf/test/parameter_filter_test.rb:5:in klass': Mocha methods cannot be used outside the context of a test (Mocha::NotInitializedError) from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/railties-6.1.4.1/lib/rails/test_unit/reporter.rb:68:in format_line' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/railties-6.1.4.1/lib/rails/test_unit/reporter.rb:15:in record' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:682:in block in record' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:681:in each' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:681:in record' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:324:in run_one_method' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:311:in block (2 levels) in run' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:310:in each' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:310:in block in run' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:350:in on_signal' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:337:in with_info_handler' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:309:in run' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:159:in block in __run' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:159:in map' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:159:in __run' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:136:in run' from /home/evgeni/Devel/minitest-wtf/vendor/bundle/ruby/3.0.0/gems/minitest-5.10.3/lib/minitest.rb:63:in block in autorun'rake aborted!
Oh nice, this is even better! Instead of the mangled class name, we now get the very same error the Foreman tests aborted with, plus a nice stack trace! But wait, why is it pointing at railties? We're not loading that! Anyways, lets look at railties-6.1.4.1/lib/rails/test_unit/reporter.rb, line 68
defformat_line(result)klass=result.respond_to?(:klass)?result.klass:result.class"%s#%s = %.2f s = %s"%[klass,result.name,result.time,result.result_code]end
Heh, this is touching result.klass, which we just messed up. Nice!
But quickly back to railties What if we only add that to the Gemfile, not full blown Rails?
Yepp, same failure. Also happens with require => false added to the line, so it seems railties somehow injects itself into rake even if nothing is using it?! "Cool"!
By the way, why are we still pinning minitest to < 5.11? Oh right, this was the original reason to look into that whole topic. And, uh, it's pointing at klass there already! 4 years ago!
So lets remove that boundary and funny enough, now tests are passing again, even if we use klass!
Minitest 5.11 changed how Minitest::Test is structured, and seems not to rely on klass at that point anymore. And I guess Rails also changed a bit since the original pin was put in place four years ago.
I didn't want to go another rabbit hole, finding out what changed in Rails, but I did try with 5.0 (well, 5.0.7.2) to be precise, and the output with newer (>= 5.11) Minitest was interesting:
It's leaking Minitest::Result as klass now, instead of Mocha::Mock. So probably something along these lines was broken 4 years ago and triggered this pin.
What do we learn from that?
klass is cursed and shouldn't be used in places where inheritance and tooling might decide to use it for some reason
inheritance is cursed - why the heck are implementation details of Minitest leaking inside my tests?!
tooling is cursed - why is railties injecting stuff when I didn't ask it to?!
dependency pinning is cursed - at least if you pin to avoid an issue and then forget about said issue for four years
On the road to AppStream 1.0, a lot of items from the long todo list have been done so far only one major feature is remaining, external release descriptions, which is a tricky one to implement and specify. For AppStream 1.0 it needs to be present or be rejected though, as it would be a major change in how release data is handled in AppStream.
Besides 1.0 preparation work, the recent 0.15 release and the releases before it come with their very own large set of changes, that are worth a look and may be interesting for your application to support. But first, for a change that affects the implementation and not the XML format:
1. Completely rewritten caching code
Keeping all AppStream data in memory is expensive, especially if the data is huge (as on Debian and Ubuntu with their large repositories generated from desktop-entry files as well) and if processes using AppStream are long-running. The latter is more and more the case, not only does GNOME Software run in the background, KDE uses AppStream in KRunner and Phosh will use it too for reading form factor information. Therefore, AppStream via libappstream provides an on-disk cache that is memory-mapped, so data is only consuming RAM if we are actually doing anything with it.
Previously, AppStream used an LMDB-based cache in the background, with indices for fulltext search and other common search operations. This was a very fast solution, but also came with limitations, LMDB s maximum key size of 511 bytes became a problem quite often, adjusting the maximum database size (since it has to be set at opening time) was annoyingly tricky, and building dedicated indices for each search operation was very inflexible. In addition to that, the caching code was changed multiple times in the past to allow system-wide metadata to be cached per-user, as some distributions didn t (want to) build a system-wide cache and therefore ran into performance issues when XML was parsed repeatedly for generation of a temporary cache. In addition to all that, the cache was designed around the concept of one cache for data from all sources , which meant that we had to rebuild it entirely if just a small aspect changed, like a MetaInfo file being added to /usr/share/metainfo, which was very inefficient.
To shorten a long story, the old caching code was rewritten with the new concepts of caches not necessarily being system-wide and caches existing for more fine-grained groups of files in mind. The new caching code uses Richard Hughes excellent libxmlb internally for memory-mapped data storage. Unlike LMDB, libxmlb knows about the XML document model, so queries can be much more powerful and we do not need to build indices manually. The library is also already used by GNOME Software and fwupd for parsing of (refined) AppStream metadata, so it works quite well for that usecase. As a result, search queries via libappstream are now a bit slower (very much depends on the query, roughly 20% on average), but can be mmuch more powerful. The caching code is a lot more robust, which should speed up startup time of applications. And in addition to all of that, the AsPool class has gained a flag to allow it to monitor AppStream source data for changes and refresh the cache fully automatically and transparently in the background.
All software written against the previous version of the libappstream library should continue to work with the new caching code, but to make use of some of the new features, software using it may need adjustments. A lot of methods have been deprecated too now.
2. Experimental compose support
Compiling MetaInfo and other metadata into AppStream collection metadata, extracting icons, language information, refining data and caching media is an involved process. The appstream-generator tool does this very well for data from Linux distribution sources, but the tool is also pretty heavyweight with lots of knobs to adjust, an underlying database and a complex algorithm for icon extraction. Embedding it into other tools via anything else but its command-line API is also not easy (due to D s GC initialization, and because it was never written with that feature in mind). Sometimes a simpler tool is all you need, so the libappstream-compose library as well as appstreamcli compose are being developed at the moment. The library contains building blocks for developing a tool like appstream-generator while the cli tool allows to simply extract metadata from any directory tree, which can be used by e.g. Flatpak. For this to work well, a lot of appstream-generator s D code is translated into plain C, so the implementation stays identical but the language changes.
Ultimately, the generator tool will use libappstream-compose for any general data refinement, and only implement things necessary to extract data from the archive of distributions. New applications (e.g. for new bundling systems and other purposes) can then use the same building blocks to implement new data generators similar to appstream-generator with ease, sharing much of the code that would be identical between implementations anyway.
2. Supporting user input controls
Want to advertise that your application supports touch input? Keyboard input? Has support for graphics tablets? Gamepads? Sure, nothing is easier than that with the new control relation item and supports relation kind (since 0.12.11 / 0.15.0, details):
3. Defining minimum display size requirements
Some applications are unusable below a certain window size, so you do not want to display them in a software center that is running on a device with a small screen, like a phone. In order to encode this information in a flexible way, AppStream now contains a display_length relation item to require or recommend a minimum (or maximum) display size that the described GUI application can work with. For example:
This will make the application require a display length greater or equal to 300 logical pixels. A logical pixel (also device independent pixel) is the amount of pixels that the application can draw in one direction. Since screens, especially phone screens but also screens on a desktop, can be rotated, the display_length value will be checked against the longest edge of a display by default (by explicitly specifying the shorter edge, this can be changed).
This feature is available since 0.13.0, details. See also Tobias Bernard s blog entry on this topic.
4. Tags
This is a feature that was originally requested for the LVFS/fwupd, but one of the great things about AppStream is that we can take very project-specific ideas and generalize them so something comes out of them that is useful for many. The new tags tag allows people to tag components with an arbitrary namespaced string. This can be useful for project-internal organization of applications, as well as to convey certain additional properties to a software center, e.g. an application could mark itself as featured in a specific software center only. Metadata generators may also add their own tags to components to improve organization. AppStream gives no recommendations as to how these tags are to be interpreted except for them being a strictly optional feature. So any meaning is something clients and metadata authors need to negotiate. It therefore is a more specialized usecase of the already existing custom tag, and I expect it to be primarily useful within larger organizations that produce a lot of software components that need sorting. For example:
This feature is available since 0.15.0, details.
5. MetaInfo Creator changes
The MetaInfo Creator (source) tool is a very simple web application that provides you with a form to fill out and will then generate MetaInfo XML to add to your project after you have answered all of its questions. It is an easy way for developers to add the required metadata without having to read the specification or any guides at all.
Recently, I added support for the new control and display_length tags, resolved a few minor issues and also added a button to instantly copy the generated output to clipboard so people can paste it into their project. If you want to create a new MetaInfo file, this tool is the best way to do it!
The creator tool will also not transfer any data out of your webbrowser, it is strictly a client-side application.
And that is about it for the most notable changes in AppStream land! Of course there is a lot more, additional tags for the LVFS and content rating have been added, lots of bugs have been squashed, the documentation has been refined a lot and the library has gained a lot of new API to make building software centers easier. Still, there is a lot to do and quite a few open feature requests too. Onwards to 1.0!
A new version 0.0.7 of RcppSpdlog is now on CRAN. RcppSpdlog bundles spdlog, a wonderful header-only C++ logging library with all the bells and whistles you would want that was written by Gabi Melman, and also includes fmt by Victor Zverovich.
This release brings upstream bugfix releases 1.9.1 and 1.9.2 of spdlog. We also removed the YAML file (and badge) for the disgraced former continuous integration service we shall not name (yet that we all used to use). And just like digest four days ago, drat three days ago, littler two days ago, and RcppAPT yesterday, we converted the vignettes from using the minidown package to the (fairly new) simplermarkdown package which is so much more appropriate for our use of the minimal water.css style.
The (minimal) NEWS entry for this release follows.
Changes in RcppSpdlog version 0.0.7 (2021-12-05)
Upgraded to upstream bug fix releases spdlog 1.9.1 and 1.9.2
Welcome to the November 2021 report from the Reproducible Builds project.
As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is therefore to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. If you are interested in contributing to our project, please visit our Contribute page on our website.
On November 6th, Vagrant Cascadian presented at this year s edition of the SeaGL conference, giving a talk titled Debugging Reproducible Builds One Day at a Time:
I ll explore how I go about identifying issues to work on, learn more about the specific issues, recreate the problem locally, isolate the potential causes, dissect the problem into identifiable parts, and adapt the packaging and/or source code to fix the issues.
In the role of a packager, updating packages is a recurring task. For some projects, a packager is involved in upstream maintenance, or well written release notes make it easy to figure out what changed between the releases. This isn t always the case, for instance with some small project maintained by one or two people somewhere on GitHub, and it can be useful to verify what exactly changed. Diffoscope can help determine the changes between package releases. []
diffoscopediffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb made the following changes, including preparing and uploading versions 190, 191, 192, 193 and 194 to Debian:
New features:
Continue loading a .changes file even if the referenced files do not exist, but include a comment in the returned diff. []
Log the reason if we cannot load a Debian .changes file. []
Bug fixes:
Detect XML files as XML files if file(1) claims if they are XML files or if they are named .xml. (#999438)
Don t duplicate file lists at each directory level. (#989192)
Don t raise a traceback when comparing nested directories with non-directories. []
Re-enable test_android_manifest. []
Don t reject Debian .changes files if they contain non-printable characters. []
Codebase improvements:
Avoid aliasing variables if we aren t going to use them. []
Use isinstance over type. []
Drop a number of unused imports. []
Update a bunch of %-style string interpolations into f-strings or str.format. []
When pretty-printing JSON, mark the difference as being reformatted, additionally avoiding including the full path. []
Import itertools top-level module directly. []
Chris Lamb also made an update to the command-line client to trydiffoscope, a web-based version of the diffoscope in-depth and content-aware diff utility, specifically only waiting for 2 minutes for try.diffoscope.org to respond in tests. (#998360)
In addition Brandon Maier corrected an issue where parts of large diffs were missing from the output [], Zbigniew J drzejewski-Szmek fixed some logic in the assert_diff_startswith method [] and Mattia Rizzolo updated the packaging metadata to denote that we support both Python 3.9 and 3.10 [] as well as a number of warning-related changes[][]. Vagrant Cascadian also updated the diffoscope package in GNU Guix [][].
Software development
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Elsewhere, in software development, Jonas Witschel updated strip-nondeterminism, our tool to remove specific non-deterministic results from a completed build so that it did not fail on JAR archives containing invalid members with a .jar extension []. This change was later uploaded to Debian by Chris Lamb.
reprotest is the Reproducible Build s project end-user tool to build the same source code twice in widely different environments and checking whether the binaries produced by the builds have any differences. This month, Mattia Rizzolo overhauled the Debian packaging [][][] and fixed a bug surrounding suffixes in the Debian package version [], whilst Stefano Rivera fixed an issue where the package tests were broken after the removal of diffoscope from the package s strict dependencies [].
Testing framework
The Reproducible Builds project runs a testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:
Holger Levsen:
Document the progress in setting up snapshot.reproducible-builds.org. []
Extend testing to include Debian bookworm too.. []
Automatically create the Jenkins view to display jobs related to building the Live images. []
Vagrant Cascadian:
Add a Debian package set group for the packages and tools maintained by the Reproducible Builds maintainers themselves. []
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
This post is part of a series called "PACKRAT". If this is the first post
you've found, it'd be worth reading the
intro post first and then looking over
all posts in the series.
In the last post, we we were able to build a functioning
Layer 1 PHY where we can encode symbols to transmit, and receive symbols on the
other end, we re now at the point where we can encode and decode those symbols
as bits and frame blocks of data, marking them with a Sender and a Destination
for routing to the right host(s). This is a Layer 2 scheme in the OSI model,
which is otherwise known as the Data Link
Layer. You re using one to
view this website right now I m willing to bet your data is going
through an Ethernet layer 2 as well as WiFi or maybe a cellular data
protocol like 5G or LTE.
Given that this entire exercise is hard enough without designing a complex
Layer 2 scheme, I opted for simplicity in the hopes this would free me from the
complexity and research that has gone into this field for the last 50 years. I
settled on stealing a few ideas from Ethernet Frames namely, the use of MAC
addresses to identify parties, and
the EtherType field to indicate the Payload type. I also stole the idea of
using a CRC at the end of the Frame to check for corruption, as well as the
specific CRC method (crc32 using 0xedb88320 as the polynomial).
Lastly, I added a callsign field to make life easier on ham radio frequencies
if I was ever to seriously attempt to use a variant of this protocol over the
air with multiple users. However, given this scheme is not a commonly used
scheme, it s best practice to use a nearby radio to identify your transmissions
on the same frequency while testing or use a Faraday box to test without
transmitting over the airwaves. I added the callsign field in an effort to lean
into the spirit of the Part 97 regulations, even if I relied on a phone
emission to identify the Frames.
As an aside, I asked the ARRL for input here, and their stance to me over email
was I d be OK according to the regs if I were to stick to UHF and put my
callsign into the BPSK stream using a widely understood encoding (even with no
knowledge of PACKRAT, the callsign is ASCII over BPSK and should be easily
demodulatable for followup with me). Even with all this, I opted to use FM
phone to transmit my callsign when I was active on the air (specifically, using
an SDR and a small bash script to automate transmission while I watched for
interference or other band users).
Right, back to the Frame:
sync
dest
source
callsign
type
payload
crc
With all that done, I put that layout into a struct, so that we can marshal
and unmarshal bytes to and from our Frame objects, and work with it
in software.
Time to pick some consts
I picked a unique and distinctive sync sequence, which the sender
will transmit before the Frame, while the receiver listens for that
sequence to know when it s in byte alignment with the symbol stream.
My sync sequence is [3]byte 'U', 'f', '~' which works out to be a
very pleasant bit sequence of 01010101 01100110 01111110. It s important
to have soothing preambles for your Frames. We need all the good energy
we can get at this point.
var (
FrameStart = [3]byte'U', 'f', '~'FrameMaxPayloadSize = 1500
)
Next, I defined some FrameType values for the type field,
which I can use to determine what is done with that data next,
something Ethernet was originally missing, but has since grown
to depend on (who needs Length anyway? Not me. See below!)
FrameType
Description
Bytes
Raw
Bytes in the Payload field are opaque and not to be parsed.
[2]byte 0x00, 0x01
IPv4
Bytes in the Payload field are an IPv4 packet.
[2]byte 0x00, 0x02
And finally, I decided on a maximum length of the Payload, and decided on
limiting it to 1500 bytes to align with the MTU of Ethernet.
var (
FrameTypeRaw = FrameType0, 1FrameTypeIPv4 = FrameType0, 2
)
Given we know how we re going to marshal and unmarshal binary data to and from
Frames, we can now move on to looking through the bit stream for our Frames.
Why is there no Length field?
I was initially a bit surprised that Ethernet Frames didn t have a Length field
in use, but the more I thought about it, the more it seemed like a big ole'
failure mode without a good implementation outcome. Either the Length is right
(resulting in no action and used bits on every packet) or the Length is not the
length of the Payload and the driver needs to determine what to do with the
packet does it try and trim the overlong payload and ignore the rest? What
if both the end of the read bytes and the end of the subset of the packet
denoted by Length have a valid CRC? Which is used? Will everyone agree? What
if Length is longer than the Payload but the CRC is good where we detected a
lost carrer?
I decided on simplicity. The end of a Frame is denoted by the loss of the BPSK
carrier when the signal is no longer being transmitted (or more correctly,
when the signal is no longer received), we know we ve hit the end of a packet.
Missing a single symbol will result in the Frame being finalized. This can
cause some degree of corruption, but it s also a lot easier than doing tricks
like bit stuffing to create an end of symbol stream delimiter.
Finding the Frame start in a Symbol Stream
First thing we need to do is find our sync bit pattern in the symbols we re
receiving from our BPSK demodulator. There s some smart ways to do this, but
given that I m not much of a smart man, I again decided to go for simple
instead. Given our incoming vector of symbols (which are still float values)
prepend one at a time to a vector of floats that is the same length as the
sync phrase, and compare against the sync phrase, to determine if we re in
sync with the byte boundary within the symbol stream.
The only trick here is that because we re using BPSK to modulate and demodulate
the data, post phaselock we can be 180 degrees out of alignment (such that a +1
is demodulated as -1, or vice versa). To deal with that, I check against both
the sync phrase as well as the inverse of the sync phrase (both [1, -1, 1] as well as [-1, 1, -1]) where if the inverse sync is matched, all symbols
to follow will be inverted as well. This effectively turns our symbols back
into bits, even if we re flipped out of phase. Other techniques like
NRZI will represent a 0 or
1 by a change in phase state which is great, but can often cascade into long
runs of bit errors, and is generally more complex to implement. That
representation isn t ambiguous, given you look for a phase change, not the
absolute phase value, which is incredibly compelling.
Here s a notional example of how I ve been thinking about the phrase sliding
window and how I ve been thinking of the checks. Each row is a new symbol
taken from the BPSK receiver, and pushed to the head of the sliding window,
moving all symbols back in the vector by one.
Given the pseudocode above, let s step through what the checks would be doing
at each step:
Buffer
Sync
Inverse Sync
[ ]float 0, ,0
[ ]float -1, ,-1
[ ]float 1, ,1
[ ]float 0, ,1
[ ]float -1, ,-1
[ ]float 1, ,1
[more bits in]
[ ]float -1, ,-1
[ ]float 1, ,1
[ ]float 1, ,1
[ ]float -1, ,-1
[ ]float 1, ,1
After this notional set of comparisons, we know that at the last step, we are
now aligned to the frame and byte boundary the next symbol / bit will be the
MSB of the 0th Frame byte. Additionally, we know we re also 180 degrees out of
phase, so we need to flip the symbol s sign to get the bit. From this point on
we can consume 8 bits at a time, and re-assemble the byte stream. I don t know
what this technique is called or even if this is used in real grown-up
implementations, but it s been working for my toy implementation.
Next Steps
Now that we can read/write Frames to and from PACKRAT, the next steps here are
going to be implementing code to encode and decode Ethernet traffic into
PACKRAT, coming next in Part 5!
This post is part of a series called "PACKRAT". If this is the first post
you've found, it'd be worth reading the
intro post first and then looking over
all posts in the series.
In the last post, we worked through how to generate
a BPSK signal, and hopefully transmit it using one of our SDRs.
Let s take that and move on to Receiving BPSK and turning that back
into symbols!
Demodulating BPSK data is a bit more tricky than transmitting BPSK data, mostly
due to tedious facts of life such as space, time, and hardware built with
compromises because not doing that makes the problem impossible. Unfortunately,
it s now our job to work within our imperfect world to recover perfect data. We
need to handle the addition of noise, differences in frequency, clock
synchronization and interference in order to recover our information. This
makes life a lot harder than when we transmit information, and as a result, a
lot more complex.
Coarse Sync
Our starting point for this section will be working from a capture of a number
of generated PACKRAT packets
as heard by a PlutoSDR at (xz compressed interleaved int16, 2,621,440 samples per second)
Every SDR has its own oscillator, which eventually controls a number of
different components of an SDR, such as the IF (if it s a superheterodyne
architecture) and the sampling rate. Drift in oscillators lead to drifts in
frequency such that what one SDR may think is 100MHz may be 100.01MHz for
another radio. Even if the radios were perfectly in sync, other artifacts such
as doppler time dilation due to motion
can cause the frequency to appear higher or lower in frequency than it was
transmitted.
All this is a long way of saying, we need to determine when we see a strong
signal that s close-ish to our tuned frequency, and take steps to roughly
correct it to our center frequency (in the order of 100s of Hz to kHz) in order
to acquire a phase lock on the signal to attempt to decode information contained
within.
The easiest way of detecting the loudest signal of interest is to use an
FFT. Getting into how
FFTs work is out of scope of this post, so if this is the first time you re
seeing mention of an FFT, it may be a good place to take a quick break to learn
a bit about the time domain (which is what the IQ data we ve been working with
so far is), frequency domain, and how the FFT and iFFT operations can convert
between them.
Lastly, because FFTs average power over the window, swapping phases such that
the transmitted wave has the same number of in-phase and inverted-phase symbols
the power would wind up averaging to zero. This is not helpful, so I took a tip
from Dr. Marc Lichtman s PySDR
project
and used complex squaring to drive
our BPSK signal into a single detectable carrier by squaring the IQ data.
Because points are on the unit circle and at tau/2 (specifically, tau/(2^1)
for BPSK, 2^2 for QPSK) angles, and given that squaring has the effect of
doubling the angle, and angles are all mod tau, this will drive our wave
comprised of two opposite phases back into a continuous wave effectively
removing our BPSK modulation, making it much easier to detect in the
frequency domain. Thanks to Tom Bereknyei
for helping me with that!
...variq []complexvarfreq []complexfori:=rangeiqiq[i] = iq[i] *iq[i]
// perform an fft, computing the frequency
// domain vector in freq given the iq data
// contained in iq .
fft(iq, freq)
// get the array index of the max value in the
// freq array given the magnitude value of the
// complex numbers.
varbinIdx = max(abs(freq))
...
Now, most FFT operations will lay the frequency domain data out a bit
differently than you may expect (as a human), which is that the 0th element of
the FFT is 0Hz, not the most negative number (like in a waterfall). Generally
speaking, zero first is the most common frequency domain layout (and
generally speaking the most safe assumption if there s no other documentation
on fft layout). Negative first is usually used when the FFT is being rendered
for human consumption such as a waterfall plot.
Given that we now know which FFT bin (which is to say, which index into the
FFT array) contains the strongest signal, we ll go ahead and figure out what
frequency that bin relates to.
In the time domain, each complex number is the next time instant. In the
frequency domain, each bin is a discrete frequency or more specifically a
frequency range. The bandwidth of the bin is a function of the sampling rate
and number of time domain samples used to do the FFT operation. As you increase
the amount of time used to preform the FFT, the more precise the FFT
measurement of frequency can be, but it will cover the same bandwidth, as
defined by the sampling rate.
...varsampleRate = 2,621,440// bandwidth is the range of frequencies
// contained inside a single FFT bin,
// measured in Hz.
varbandwidth = sampleRate/len(freq)
...
Now that we know we have a zero-first layout and the bin bandwidth,
we can compute what our frequency offset is in Hz.
...// binIdx is the index into the freq slice
// containing the frequency domain data.
varbinIdx = 0// binFreq is the frequency of the bin
// denoted by binIdx
varbinFreq = 0ifbinIdx > len(freq)/2// This branch covers the case where the bin
// is past the middle point - which is to say,
// if this is a negative frequency.
binFreq = bandwidth* (binIdx- len(freq))
else// This branch covers the case where the bin
// is in the first half of the frequency array,
// which is to say - if this frequency is
// a positive frequency.
binFreq = bandwidth*binIdx...
However, sice we squared the IQ data, we re off in frequency by twice the
actual frequency if we are reading 12kHz, the bin is actually 6kHz. We
need to adjust for that before continuing with processing.
...varbinFreq = 0...// [compute the binFreq as above]
...// Adjust for the squaring of our IQ data
binFreq = binFreq/2...
Finally, we need to shift the frequency by the inverse of the binFreq
by generating a carrier wave at a specific frequency and rotating every
sample by our carrier wave so that a wave at the same frequency will
slow down (or stand still!) as it approaches 0Hz relative to the carrier
wave.
vartau = pi*2// ts tracks where in time we are (basically: phase)
vartsfloat// inc is the amount we step forward in time (seconds)
// each sample.
varincfloat = (1/sampleRate)
// amount to shift frequencies, in Hz,
// in this case, shift +12 kHz to 0Hz
varshift = -12,000fori:=rangeiqts+=incifts > tau// not actually needed, but keeps ts within
// 0 to 2*pi (since it is modulus 2*pi anyway)
ts-=tau// Here, we're going to create a carrier wave
// at the provided frequency (in this case,
// -12kHz)
cwIq = complex(cos(tau*shift*ts), sin(tau*shift*ts))
iq[i] = iq[i] *cwIq
Now we ve got the strong signal we ve observed (which may or may not be
our BPSK modulated signal!) close enough to 0Hz that we ought to be able
to Phase Lock the signal in order to begin demodulating the signal.
Filter
After we re roughly in the neighborhood of a few kHz, we can now take some
steps to cut out any high frequency components (both positive high frequencies
and negative high frequencies). The normal way to do this would be to do an
FFT, apply the filter in the frequency domain, and then do an iFFT to turn it
back into time series data. This will work in loads of cases, but I ve
found it to be incredibly tricky to get right when doing PSK. As such, I ve
opted to do this the old fashioned way in the time domain.
I ve again opted to go simple rather than correct, and haven t used
nearly any of the advanced level trickery I ve come across for fear of using
it wrong. As a result, our process here is going to be generating a
sinc filter by computing a
number of taps, and applying that in the time domain directly on the IQ
stream.
...// Apply sinc taps to an IQ stream
variq []complex// taps as created in dst above
vartaps []floatvardelay = make([]complex, len(taps))
fori:=rangeiq// let's shift the next sample into
// the delay buffer
copy(delay[1:], delay)
delay[0] = iq[i]
varphasorcomplexforj:=rangedelay// for each sample in the buffer, let's
// weight them by the tap values, and
// create a new complex number based on
// filtering the real and imag values.
phasor+= complex(
taps[j] * real(delay[j]),
taps[j] * imag(delay[j]),
)
// now that we've run this sample
// through the filter, we can go ahead
// and scale it back (since we multiply
// above) and drop it back into the iq
// buffer.
iq[i] = complex(
real(phasor) / len(taps),
imag(phasor) / len(taps),
)
...
After running IQ samples through the taps and back out, we ll have a signal
that s been filtered to the shape of our designed Sinc filter which will
cut out captured high frequency components (both positive and negative).
Astute observers will note that we re using the real (float) valued taps
on both the real and imaginary values independently. I m sure there s a way
to apply taps using complex numbers, but it was a bit confusing to work
through without being positive of the outcome. I may revisit this in the
future!
Downsample
Now, post-filter, we ve got a lot of extra RF bandwidth being represented in
our IQ stream at our high sample rate All the high frequency values are now
filtered out, which means we can reduce our sampling rate without losing much
information at all. We can either do nothing about it and process at the fairly
high sample rate we re capturing at, or we can drop the sample rate down and
help reduce the volume of numbers coming our way.
There s two big ways of doing this; either you can take every Nth sample (e.g.,
take every other sample to half the sample rate, or take every 10th to decimate
the sample stream to a 10th of what it originally was) which is the easiest to
implement (and easy on the CPU too), or to average a number of samples to
create a new sample.
A nice bonus to averaging samples is that you can trade-off some CPU time for a
higher effective number of bits
(ENOB) in your IQ
stream, which helps reduce noise, among other things. Some hardware does
exactly this (called Oversampling ), and like many things, it has
some pros and some cons.
I ve opted to treat our IQ stream like an oversampled IQ stream
and average samples to get a marginal bump in ENOB.
Taking a group of 4 samples and averaging them results in a bit of added
precision. That means that a stream of IQ data at 8 ENOB can be bumped to 9
ENOB of precision after the process of oversampling and averaging. That
resulting stream will be at 1/4 of the sample rate, and this process can be
repeated 4 samples can again be taken for a bit of added precision; which is
going to be 1/4 of the sample rate (again), or 1/16 of the original sample
rate. If we again take a group of 4 samples, we ll wind up with another bit and
a sample rate that s 1/64 of the original sample rate.
Phase Lock
Our starting point for this section is the same capture as above, but
post-coarse sync, filtering downsampling (xz compressed interleaved float32, 163,840 samples per second)
The PLL in PACKRAT was one
of the parts I spent the most time stuck on. There s no shortage of discussions
of how hardware PLLs work, or even a few software PLLs, but very little by way
of how to apply them and/or troubleshoot them. After getting frustrated trying
to follow the well worn path, I decided to cut my own way through the bush
using what I had learned about the concept, and hope that it works well enough
to continue on.
PLLs, in concept are fairly simple you generate a carrier wave at a
frequency, compare the real-world SDR IQ sample to where your carrier wave is
in phase, and use the difference between the local wave and the observed wave
to adjust the frequency and phase of your carrier wave. Eventually, if all goes
well, that delta is driven as small as possible, and your carrier wave can be
used as a reference clock to determine if the observed signal changes in
frequency or phase.
In reality, tuning PLLs is a total pain, and basically no one outlines how to
apply them to BPSK signals in a descriptive way. I ve had to steal an approach
I ve seen in hardware to implement my software PLL, with any hope it s close
enough that this isn t a hazard to learners. The concept is to generate the
carrier wave (as above) and store some rolling averages to tune the carrier
wave over time. I use two constants, alpha and beta (which appear to be
traditional PLL variable names for this function) which control how quickly the
frequency and phase is changed according to observed mismatches. Alpha is set
fairly high, which means discrepancies between our carrier and observed data
are quickly applied to the phase, and a lower constant for Beta, which will
take long-term errors and attempt to use that to match frequency.
This is all well and good. Getting to this point isn t all that obscure, but
the trouble comes when processing a BPSK signal. Phase changes kick the PLL
out of alignment and it tends to require some time to get back into phase lock,
when we really shouldn t even be loosing it in the first place. My attempt is
to generate two predicted samples, one for each phase of our BPSK signal. The
delta is compared, and the lower error of the two is used to adjust the PLL,
but the carrier wave itself is used to rotate the sample.
varalpha = 0.1varbeta = (alpha*alpha) /2varphase = 0.0varfrequency = 0.0...fori:=rangeiqpredicted = complex(cos(phase), sin(phase))
sample = iq[i] *conj(predicted)
delta = phase(sample)
predicted2 = complex(cos(phase+pi), sin(phase+pi))
sample2 = iq[i] *conj(predicted2)
delta2 = phase(sample2)
ifabs(delta2) < abs(delta)
// note that we do not update 'sample'.
delta = delta2phase+=alpha*deltafrequency+=beta*delta// adjust the iq sample to the PLL rotated
// sample.
iq[i] = sample...
If all goes well, this loop has the effect of driving a BPSK signal s imaginary
values to 0, and the real value between +1 and -1.
Average Idle / Carrier Detect
Our starting point for this section is the same capture as above, but
post-PLL (xz compressed interleaved float32, 163,840 samples per second)
When we start out, we have IQ samples that have been mostly driven to an
imaginary component of 0 and real value range between +1 and -1 for each symbol
period. Our goal now is to determine if we re receiving a signal, and if so,
determine if it s +1 or -1. This is a deceptively hard problem given it spans a
lot of other similarly entertaining hard problems. I ve opted to not solve the
hard problems involved and hope that in practice my very haphazard
implementation works well enough. This turns out to be both good (not solving a
problem is a great way to not spend time on it) and bad (turns out it does
materially impact performance). This segment is the one I plan on revisiting,
first. Expect more here at some point!
Given that I want to be able to encapsulate three states in the output from
this section (our Symbols are no carrier detected ( 0 ), real value 1 ( 1 ) or
real value -1 ("-1")), which means spending cycles to determine what the
baseline noise is to try and identify when a signal breaks through the noise
becomes incredibly important.
varidleThresholdvarthresholdFactor = 10...// sigThreshold is used to determine if the symbol
// is -1, +1 or 0. It's 1.3 times the idle signal
// threshold.
varsigThreshold = (idleThreshold*0.3) +idleThreshold// iq contains a single symbol's worth of IQ samples.
// clock alignment isn't really considered; so we'll
// get a bad packet if we have a symbol transition
// in the middle of this buffer. No attempt is made
// to correct for this yet.
variq []complex// avg is used to average a chunk of samples in the
// symbol buffer.
varavgfloatvarmid = len(iq) /2// midNum is used to determine how many symbols to
// average at the middle of the symbol.
varmidNum = len(iq) /50forj:=mid; j < mid+midNum; j++avg+= real(iq[j])
avg/=midNumvarsymbolfloatswitchcaseavg > sigThreshold:
symbol = 1caseavg < -sigThreshold:
symbol = -1default:
symbol = 0// update the idleThreshold using the thresholdFactor
// to average the idleThreshold over more samples to
// get a better idea of average noise.
idleThreshold = (
(idleThreshold*(thresholdFactor-1) +symbol) \
/thresholdFactor
)
// write symbol to output somewhere
...
Next Steps
Now that we have a stream of values that are either +1, -1 or 0, we
can frame / unframe the data contained in the stream, and decode Packets
contained inside, coming next in Part 4!
A new version of the RcppAPT package interfacing from R to the C++ library behind the awesome apt, apt-get, apt-cache, commands and their cache powering Debian, Ubuntu and the like arrived on CRAN earlier today.
RcppAPT allows you to query the (Debian or Ubuntu) package dependency graph at will, with build-dependencies (if you have deb-src entries), reverse dependencies, and all other goodies. See the vignette and examples for illustrations.
This release updates some package metadata, adds a new package testing helper, and, just like digest three days ago, drat two days ago, and littler yesterday, we converted the vignettes from using the minidown package to the (fairly new) simplermarkdown package which is so much more appropriate for our use of the minimal water.css style.
Changes in version 0.0.8 (2021-12-04)
New test file version.R ensures NEWS file documents current package version
The sixteenth release of littler as a CRAN package just landed, following in the now fifteen year history (!!) as a package started by Jeff in 2006, and joined by me a few weeks later.
littler is the first command-line interface for R as it predates Rscript. It allows for piping as well for shebang scripting via #!, uses command-line arguments more consistently and still starts faster. It also always loaded the methods package which Rscript only started to do in recent years.
littler lives on Linux and Unix, has its difficulties on macOS due to yet-another-braindeadedness there (who ever thought case-insensitive filesystems as a default were a good idea?) and simply does not exist on Windows (yet the build system could be extended see RInside for an existence proof, and volunteers are welcome!). See the FAQ vignette on how to add it to your PATH. A few examples are highlighted at the Github repo, as well as in the examples vignette.
This release brings a more robust and featureful install2.r script (thanks to Gergely Dar czi), corrects some documentation typos (thanks to John Kerl), and now compacts pdf vignette better when using the build.r helper. It also one more updates the URLs for the two RStudio downloaders, and adds a simplermarkdown wrapper. Next, we removed the YAML file (and badge) for the disgraced former continuous integration service we shall not name (yet that we all used to use). And, following digest two days ago and drat yesterday, we converted the vignettes from using the minidown package to the (fairly new) simplermarkdown package which is so much more appropriate for our use of the minimal water.css style.
The full change description follows.
Changes in littler version 0.3.15 (2021-12-03)
Changes in examples
The install2 script can select download methods, and cope with errors from parallel download (thanks to Gergely Daroczi)
The build.r now uses both as argument to --compact-vignettes
The RStudio download helper were once again updated for changed URLs
New caller for simplermarkdown::mdweave_to_html
Changes in package
Several typos were correct (thanks to John Kerl)
Travis artifacts and badges have been pruned
Vignettes now use simplermarkdown
My CRANberries service provides a comparison to the previous release. Full details for the littler release are provided as usual at the ChangeLog page, and also on the package docs website. The code is available via the GitHub repo, from tarballs and now of course also from its CRAN page and via install.packages("littler"). Binary packages are available directly in Debian as well as soon via Ubuntu binaries at CRAN thanks to the tireless Michael Rutter.
Comments and suggestions are welcome at the GitHub repo.
If you like this or other open-source work I do, you can now sponsor me at GitHub.
A few days ago, a productive translator started working on a new
translation of the Made with Creative
Commons book for Brazilian Portuguese. The translation take place on
the
Weblate web based translation system. Once the translation is
complete and proof read, we can publish it on paper as well as in PDF,
ePub and HTML format. The translation is already 16% complete, and if
more people get involved I am conviced it can very quickly reach 100%.
If you are interested in helping out with this or other translations
of the Made with Creative Commons book, start translating on
Weblate. There are partial translations available in Azerbaijani,
Bengali, Brazilian Portuguese, Dutch, French, German, Greek, Polish,
Simplified Chinese, Swedish, Thai and Ukrainian.
The git
repository for the book contain all source files needed to build
the book for yourself.
HTML editions
to help with proof reading is also available.
As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
I hope you enjoyed my last post about Ansible Galaxy Namespaces. In there I noted that I originally looked for something completely different and the namespace takeover was rather accidental.
Well, originally I was looking at how the different Ansible content hosting services and their client (ansible-galaxy) behave in regard to clashes in naming of the hosted content.
"Ansible content hosting services"?! There are currently three main ways for users to obtain Ansible content:
Ansible Galaxy - the original, community oriented, free hosting platform
Automation Hub - the place for Red Hat certified and supported content, available only with a Red Hat subscription, hosted by Red Hat
Ansible Automation Platform - the on-premise version of Automation Hub, syncs content from there and allows customers to upload own content
Now the question I was curious about was: how would the tooling behave if different sources would offer identically named content?
This was inspired by Alex Birsan: Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies and zofrex: Bundler is Still Vulnerable to Dependency Confusion Attacks (CVE - 2020 - 36327), who showed that the tooling for Python, Node.js and Ruby can be tricked into fetching content from "the wrong source", thus allowing an attacker to inject malicious code into a deployment.
For the rest of this article, it's not important that there are different implementations of the hosting services, only that users can configure and use multiple sources at the same time.
The problem is that, if the user configures their server_list to contain multiple Galaxy-compatible servers, like Ansible Galaxy and Automation Hub, and then asks to install a collection, the Ansible Galaxy CLI will ask every server in the list, until one returns a successful result. The exact order seems to differ between versions, but this doesn't really matter for the issue at hand.
Imagine someone wants to install the redhat.satellite collection from Automation Hub (using ansible-galaxy collection install redhat.satellite). Now if their configuration defines Galaxy as the first, and Automation Hub as the second server, Galaxy is always asked whether it has redhat.satellite and only if the answer is negative, Automation Hub is asked. Today there is no redhat namespace on Galaxy, but there is a redhat user on GitHub, so
The canonical answer to this issue is to use a requirements.yml file and setting the source parameter. This parameter allows you to express "regardless which sources are configured, please fetch this collection from here". That's is nice, but I think this not being the default syntax (contrary to what e.g. Bundler does) is a bad approach. Users might overlook the security implications, as the shorter syntax without the source just "magically" works.
However, I think this is not even the main problem here. The documentation says: Once a collection is found, any of its requirements are only searched within the same Galaxy instance as the parent collection. The install process will not search for a collection requirement in a different Galaxy instance. But as it turns out, the source behavior waschanged and now only applies to the exact collection it is set for, not for any dependencies this collection might have.
For the sake of the example, imagine two collections: evgeni.test1 and evgeni.test2, where test2 declares a dependency on test1 in its galaxy.yml. Actually, no need to imagine, both collections are available in version 1.0.0 from galaxy.ansible.com and test1 version 2.0.0 is available from galaxy-dev.ansible.com.
Now, given our recent reading of the docs, we craft the following requirements.yml:
In a perfect world, following the documentation, this would mean that both collections are fetched from galaxy.ansible.com, right? However, this is not what ansible-galaxy does. It will fetch evgeni.test2 from the specified source, determine it has a dependency on evgeni.test1 and fetch that from the "first" available source from the configuration.
Take for example the following ansible.cfg:
And try to install collections, using the above requirements.yml:
% ansible-galaxy collection install -r requirements.yml -vvv
ansible-galaxy 2.9.27 config file = /home/evgeni/Devel/ansible-wtf/collections/ansible.cfg configured module search path = ['/home/evgeni/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python3.10/site-packages/ansible executable location = /usr/bin/ansible-galaxy python version = 3.10.0 (default, Oct 4 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]Using /home/evgeni/Devel/ansible-wtf/collections/ansible.cfg as config fileReading requirement file at '/home/evgeni/Devel/ansible-wtf/collections/requirements.yml'Found installed collection theforeman.foreman:3.0.0 at '/home/evgeni/.ansible/collections/ansible_collections/theforeman/foreman'Process install dependency mapProcessing requirement collection 'evgeni.test2'Collection 'evgeni.test2' obtained from server explicit_requirement_evgeni.test2 https://galaxy.ansible.com/api/Opened /home/evgeni/.ansible/galaxy_tokenProcessing requirement collection 'evgeni.test1' - as dependency of evgeni.test2Collection 'evgeni.test1' obtained from server test_galaxy https://galaxy-dev.ansible.com/apiStarting collection install processInstalling 'evgeni.test2:1.0.0' to '/home/evgeni/.ansible/collections/ansible_collections/evgeni/test2'Downloading https://galaxy.ansible.com/download/evgeni-test2-1.0.0.tar.gz to /home/evgeni/.ansible/tmp/ansible-local-133/tmp9uqyjgkiInstalling 'evgeni.test1:2.0.0' to '/home/evgeni/.ansible/collections/ansible_collections/evgeni/test1'Downloading https://galaxy-dev.ansible.com/download/evgeni-test1-2.0.0.tar.gz to /home/evgeni/.ansible/tmp/ansible-local-133/tmp9uqyjgki
As you can see, evgeni.test1 is fetched from galaxy-dev.ansible.com, instead of galaxy.ansible.com. Now, if those servers instead would be Galaxy and Automation Hub, and somebody managed to snag the redhat namespace on Galaxy, I would be now getting the wrong stuff Another problematic setup would be with Galaxy and on-prem Ansible Automation Platform, as you can have any namespace on the later and these most certainly can clash with namespaces on public Galaxy.
I have reported this behavior to Ansible Security on 2021-08-26, giving a 90 days disclosure deadline, which expired on 2021-11-24.
So far, the response was that this is working as designed, to allow cross-source dependencies (e.g. a private collection referring to one on Galaxy) and there is an issue to update the docs to match the code. If users want to explicitly pin sources, they are supposed to name all dependencies and their sources in requirements.yml. Alternatively they obviously can configure only one source in the configuration and always mirror all dependencies.
I am not happy with this and I think this is terrible UX, explicitly inviting people to make mistakes.
It s been over 20 months since the first COVID lockdown kicked in here in Northern Ireland and I started working from home. Even when the strict lockdown was lifted the advice here has continued to be If you can work from home you should work from home . I ve been into the office here and there (for new starts given you need to hand over a laptop and sort out some login details it s generally easier to do so in person, and I ve had a couple of whiteboard sessions that needed the high bandwidth face to face communication), but day to day is all from home.
Early on I commented that work had taken over my study. This has largely continued to be true. I set my work laptop on the stand on a Monday morning and it sits there until Friday evening, when it gets switched for the personal laptop. I have a lovely LG 34UM88 21:9 Ultrawide monitor, and my laptops are small and light so I much prefer to use them docked. Also my general working pattern is to have a lot of external connections up and running (build machine, test devices, log host) which means a suspend/resume cycle disrupts things. So I like to minimise moving things about.
I spent a little bit of time trying to find a dual laptop stand so I could have both machines setup and switch between them easily, but I didn t find anything that didn t seem to be geared up for DJs with a mixer + laptop combo taking up quite a bit of desk space rather than stacking laptops vertically. Eventually I realised that the right move was probably a desktop machine.
Now, I haven t had a desktop machine since before I moved to the US, realising at the time that having everything on my laptop was much more convenient. I decided I didn t want something too big and noisy. Cheap GPUs seem hard to get hold of these days - I m not a gamer so all I need is something that can drive a ~ 4K monitor reliably enough. Looking around the AMD Ryzen 7 5700G seemed to be a decent CPU with one of the better integrated GPUs. I spent some time looking for a reasonable Mini-ITX case + motherboard and then I happened upon the ASRock DeskMini X300. This turns out to be perfect; I ve no need for a PCIe slot or anything more than an m.2 SSD. I paired it with a Noctua NH-L9a-AM4 heatsink + fan (same as I use in the house server), 32GB DDR4 and a 1TB WD SN550 NVMe SSD. Total cost just under 650 inc VAT + delivery (and that s a story for another post).
A desktop solves the problem of fitting both machines on the desk at once, but there s still the question of smoothly switching between them. I read Evgeni Golov s article on a simple KVM switch for 30. My monitor has multiple inputs, so that s sorted. I did have a cheap USB2 switch (all I need for the keyboard/trackball) but it turned out to be pretty unreliable at the host detecting the USB change. I bought a UGREEN USB 3.0 Sharing Switch Box instead and it s turned out to be pretty reliable. The problem is that the LG 32UM88 turns out to have a poor DDC implementation, so while I can flip the keyboard easily with the UGREEN box I also have to manually select the monitor input. Which is a bit annoying, but not terrible.
The important question is whether this has helped. I built all this at the end of October, so I ve had a month to play with it. Turns out I should have done it at some point last year. At the end of the day instead of either sitting at work for a bit longer, or completely avoiding the study, I m able to lock the work machine and flick to my personal setup. Even sitting in the same seat that disconnect , and the knowledge I won t see work Slack messages or emails come in and feeling I should respond, really helps. It also means I have access to my personal setup during the week without incurring a hit at the start of the working day when I have to set things up again. So it s much easier to just dip in to some personal tech stuff in the evening than it was previously. Also from the point of view I don t need to setup the personal config, I can pick up where I left off. All of which is really nice.
It s also got me thinking about other minor improvements I should make to my home working environment to try and improve things. One obvious thing now the winter is here again is to improve my lighting; I have a good overhead LED panel but it s terribly positioned for video calls, being just behind me. So I think I m looking some sort of strip light I can have behind the large monitor to give a decent degree of backlight (possibly bouncing off the white wall). Lots of cheap options I m not convinced about, and I ve had a few ridiculously priced options from photographer friends; suggestions welcome.