Search Results: "ed"

24 February 2023

Scarlett Gately Moore: Snowstorms, Kittens and Shattered dreams

Icy morning Witch Wells AzIcy morning Witch Wells Az
Long ago I applied for my dream job at a company I have wanted to wok for since its beginning and I wasn t ready technically. Fast forward to now, I am ready! A big thank you goes out to Blue Systems for that. So I go out and find the perfect role and start the application process. The process was months long, but was going very well, the interviews and I passed the technical with flying colors. I got to the end where the hiring lead told me he was submitting my offer I was so excited, so much so, I told my husband and parents I got the job! I know, I jinxed myself there. Soon I receive the There was a problem .. One obscure assessment called GIA came back not so good. I remember that day, we were in the middle of a long series of winter storms and I when I took the test, my kitten decided right then it was me time. I couldn t very well throw her out into the snowstorm, so I continued on the best I could. It is my fault, it clearly states to be distraction free. So I speak again to the hiring lead and we both feel with my experience and technical knowledge and abilities we can still move forward. I still had hope. After some time passes, I asked for an update and got the dreaded rejection. I am told it wasn t just the GIA, but that I am not a good overall fit for the company. In one fell swoop my dreams are dashed and final, for this and all roles within that company. I wasn t given a reason either. I am devastated, heart broken, and shocked. I get along with everyone, I exceed the technical requirements, and I work well in the community. Dream door closed. I will not let this get me down. I am moving on. I will find my place where I fit in . With that said, I no longer have the will, passion, or drive to work on snaps anymore. I will leave instructions with Jonathon as to what needs to be done to move forward. The good news is my core22 kde-neon extension was merged into upstream snapcraft, so whomever takes over will have a much easier time knocking them out. @kubuntu-council I will do whatever it takes to pay back the money for the hardware you provided me to do snaps, I am truly sorry about this. What does my future hold? I will still continue with my Debian efforts. In fact, I have ventured out from the KDE umbrella and joined the go-team. I am finalizing my packaging for https://github.com/charmbracelet/gum and it s dependencies: roff, mango, mango-kong. I had my first golang patch for a failing test and have submitted it upstream. I will upload these to experimental while the freeze is on. I will be moving all the libraries in the mycroft team to the python umbrella as they are useful for other things and mycroft is no more. During the holidays I was tinkering around with selenium UI testing and stumbled on some accessibility issues within KDE, so I think this is a good place for me to dive into for my KDE contributions. I have been approached to collaborate with OpenOS on a few things, time permitting I will see what I can do there. I have a possible gig to do some websites, while I move forward in my job hunt. I will not give up! I will find my place where I fit in . Meanwhile, I must ask for donations to get us by. Anything helps, thank you for your consideration. https://gofund.me/a9c36b87

23 February 2023

Steve Kemp: A quick hack for Emacs

As I've mentioned in the past I keep a work-log, or work-diary, recording my activities every day. I have a bunch of standard things that I record, but one thing that often ends up happening is that I make references to external bug trackers, be they Jira, Bugzilla, or something else. Today I hacked up a simple emacs minor-mode for converting these references to hyperlinks, automatically, via the use of regular expressions. Given this configuration: 
(setq linkifier-patterns '(
          ("\\\<XXX-[0-9]+\\\>" "https://jira.example.com/browse/%s")
          ("\\\<BUG-[0-9]+\\\>" "https://bugzilla.example.com/show?id=%s")))
When the minor-mode is active the any literal text that matches the pattern, for example "XXX-1234", will suddenly become a clickable button that will open Jira, and BUG-1234 will become a clickable button that opens the appropriate bug in Bugzilla. There's no rewriting of the content, this is just a bit of magic that changes the display of the text (i.e. I'm using a button/text-property). Since I mostly write in org-mode I could have written my text like so: 
[[jira:XXX-1234][XXX-1234]]
But that feels like an ugly thing to do, and that style of links wouldn't work outside org-files anyway. That said it's a useful approach if you're only using org-mode, and the setup is simple: 
(add-to-list 'org-link-abbrev-alist
    '("jira" . "http://jira.example.com/browse/%s"))
Anyway, cute hack. Useful too.

Paul Tagliamonte: Announcing hz.tools

Interested in future updates? Follow me on mastodon at @paul@soylent.green. Posts about hz.tools will be tagged #hztools.

If you're on the Fediverse, I'd very much appreciate boosts on my announcement toot!
Ever since 2019, I ve been learning about how radios work, and trying to learn about using them the hard way by writing as much of the stack as is practical (for some value of practical) myself. I wrote my first Hello World in 2018, which was a simple FM radio player, which used librtlsdr to read in an IQ stream, did some filtering, and played the real valued audio stream via pulseaudio. Over 4 years this has slowly grown through persistence, lots of questions to too many friends to thank (although I will try), and the eternal patience of my wife hearing about radios nonstop for years into a number of Go repos that can do quite a bit, and support a handful of radios. I ve resisted making the repos public not out of embarrassment or a desire to keep secrets, but rather, an attempt to keep myself free of any maintenance obligations to users so that I could freely break my own API, add and remove API surface as I saw fit. The worst case was to have this project feel like work, and I can t imagine that will happen if I feel frustrated by PRs that are getting ahead of me solving problems I didn t yet know about, or bugs I didn t understand the fix for. As my rate of changes to the most central dependencies has slowed, i ve begun to entertain the idea of publishing them. After a bit of back and forth, I ve decided it s time to make a number of them public, and to start working on them in the open, as I ve built up a bit of knowledge in the space, and I and feel confident that the repo doesn t contain overt lies. That s not to say it doesn t contain lies, but those lies are likely hidden and lurking in the dark. Beware. That being said, it shouldn t be a surprise to say I ve not published everything yet for the same reasons as above. I plan to open repos as the rate of changes slows and I understand the problems the library solves well enough or if the project dead ends and I ve stopped learning.

Intention behind hz.tools It s my sincere hope that my repos help to make Software Defined Radio (SDR) code a bit easier to understand, and serves as an understandable framework to learn with. It s a large codebase, but one that is possible to sit down and understand because, well, it was written by a single person. Frankly, I m also not productive enough in my free time in the middle of the night and on weekends and holidays to create a codebase that s too large to understand, I hope! I remain wary of this project turning into work, so my goal is to be very upfront about my boundaries, and the limits of what classes of contributions i m interested in seeing. Here s some goals of open sourcing these repos:
  • I do want this library to be used to learn with. Please go through it all and use it to learn about radios and how software can control them!
  • I am interested in bugs if there s a problem you discover. Such bugs are likely a great chance for me to fix something I ve misunderstood or typoed.
  • I am interested in PRs fixing bugs you find. I may need a bit of a back and forth to fully understand the problem if I do not understand the bug and fix yet. I hope you may have some grace if it s taking a long time.
Here s a list of some anti-goals of open sourcing these repos.
  • I do not want this library to become a critical dependency of an important project, since I do not have the time to deal with the maintenance burden. Putting me in that position is going to make me very uncomfortable.
  • I am not interested in feature requests, the features have grown as I ve hit problems, I m not interested in building or maintaining features for features sake. The API surface should be exposed enough to allow others to experiment with such things out-of-tree.
  • I m not interested in clever code replacing clear code without a very compelling reason.
  • I use GNU/Linux (specifically Debian ), and from time-to-time I ve made sure that my code runs on OpenBSD too. Platforms beyond that will likely not be supported at the expense of either of those two. I ll take fixes for bugs that fix a problem on another platform, but not damage the code to work around issues / lack of features on other platforms (like Windows).
I m not saying all this to be a jerk, I do it to make sure I can continue on my journey to learn about how radios work without my full time job becoming maintaining a radio framework single-handedly for other people to use even if it means I need to close PRs or bugs without merging it or fixing the issue. With all that out of the way, I m very happy to announce that the repos are now public under github.com/hztools.

Should you use this? Probably not. The intent here is not to provide a general purpose Go SDR framework for everyone to build on, although I am keenly aware it looks and feels like it, since that what it is to me. This is a learning project, so for any use beyond joining me in learning should use something like GNU Radio or a similar framework that has a community behind it. In fact, I suspect most contributors ought to be contributing to GNU Radio, and not this project. If I can encourage people to do so, contribute to GNU Radio! Nothing makes me happier than seeing GNU Radio continue to be the go-to, and well supported. Consider donating to GNU Radio!

hz.tools/rf - Frequency types The hz.tools/rf library contains the abstract concept of frequency, and some very basic helpers to interact with frequency ranges (such as helpers to deal with frequency ranges, or frequency range math) as well as frequencies and some very basic conversions (to meters, etc) and parsers (to parse values like 10MHz). This ensures that all the hz.tools libraries have a shared understanding of Frequencies, a standard way of representing ranges of Frequencies, and the ability to handle the IO boundary with things like CLI arguments, JSON or YAML. The git repo can be found at github.com/hztools/go-rf, and is importable as hz.tools/rf. 
 // Parse a frequency using hz.tools/rf.ParseHz, and print it to stdout.
 freq := rf.MustParseHz("-10kHz")
fmt.Printf("Frequency: %s\n", freq+rf.MHz)
// Prints: 'Frequency: 990kHz'

// Return the Intersection between two RF ranges, and print
 // it to stdout.
 r1 := rf.Range rf.KHz, rf.MHz 
r2 := rf.Range rf.Hz(10), rf.KHz * 100 
fmt.Printf("Range: %s\n", r1.Intersection(r2))
// Prints: Range: 1000Hz->100kHz
These can be used to represent tons of things - ranges can be used for things like the tunable range of an SDR, the bandpass of a filter or the frequencies that correspond to a bin of an FFT, while frequencies can be used for things such as frequency offsets or the tuned center frequency.

hz.tools/sdr - SDR I/O and IQ Types This is the big one. This library represents the majority of the shared types and bindings, and is likely the most useful place to look at when learning about the IO boundary between a program and an SDR. The git repo can be found at github.com/hztools/go-sdr, and is importable as hz.tools/sdr. This library is designed to look (and in some cases, mirror) the Go io idioms so that this library feels as idiomatic as it can, so that Go builtins interact with IQ in a way that s possible to reason about, and to avoid reinventing the wheel by designing new API surface. While some of the API looks (and is even called) the same thing as a similar function in io, the implementation is usually a lot more naive, and may have unexpected sharp edges such as concurrency issues or performance problems. The following IQ types are implemented using the sdr.Samples interface. The hz.tools/sdr package contains helpers for conversion between types, and some basic manipulation of IQ streams.
IQ Format hz.tools Name Underlying Go Type
Interleaved uint8 (rtl-sdr) sdr.SamplesU8 [][2]uint8
Interleaved int8 (hackrf, uhd) sdr.SamplesI8 [][2]int8
Interleaved int16 (pluto, uhd) sdr.SamplesI16 [][2]int16
Interleaved float32 (airspy, uhd) sdr.SamplesC64 []complex64
The following SDRs have implemented drivers in-tree.
SDR Format RX/TX State
rtl u8 RX Good
HackRF i8 RX/TX Good
PlutoSDR i16 RX/TX Good
rtl kerberos u8 RX Old
uhd i16/c64/i8 RX/TX Good
airspyhf c64 RX Exp
The following major packages and subpackages exist at the time of writing:
Import What is it?
hz.tools/sdr Core IQ types, supporting types and implementations that interact with the byte boundary
hz.tools/sdr/rtl sdr.Receiver implementation using librtlsdr.
hz.tools/sdr/rtl/kerberos Helpers to enable coherent RX using the Kerberos SDR.
hz.tools/sdr/rtl/e4k Helpers to interact with the E4000 RTL-SDR dongle.
hz.tools/sdr/fft Interfaces for performing an FFT, which are implemented by other packages.
hz.tools/sdr/rtltcp sdr.Receiver implementation for rtl_tcp servers.
hz.tools/sdr/pluto sdr.Transceiver implementation for the PlutoSDR using libiio.
hz.tools/sdr/uhd sdr.Transceiver implementation for UHD radios, specifically the B210 and B200mini
hz.tools/sdr/hackrf sdr.Transceiver implementation for the HackRF using libhackrf.
hz.tools/sdr/mock Mock SDR for testing purposes.
hz.tools/sdr/airspyhf sdr.Receiver implementation for the AirspyHF+ Discovery with libairspyhf.
hz.tools/sdr/internal/simd SIMD helpers for IQ operations, written in Go ASM. This isn t the best to learn from, and it contains pure go implemtnations alongside.
hz.tools/sdr/stream Common Reader/Writer helpers that operate on IQ streams.

hz.tools/fftw - hz.tools/sdr/fft implementation The hz.tools/fftw package contains bindings to libfftw3 to implement the hz.tools/sdr/fft.Planner type to transform between the time and frequency domain. The git repo can be found at github.com/hztools/go-fftw, and is importable as hz.tools/fftw. This is the default throughout most of my codebase, although that default is only expressed at the leaf package libraries should not be hardcoding the use of this library in favor of taking an fft.Planner, unless it s used as part of testing. There are a bunch of ways to do an FFT out there, things like clFFT or a pure-go FFT implementation could be plugged in depending on what s being solved for.

hz.tools/ fm,am - analog audio demodulation and modulation The hz.tools/fm and hz.tools/am packages contain demodulators for AM analog radio, and FM analog radio. This code is a bit old, so it has a lot of room for cleanup, but it ll do a very basic demodulation of IQ to audio. The git repos can be found at github.com/hztools/go-fm and github.com/hztools/go-am, and are importable as hz.tools/fm and hz.tools/am. As a bonus, the hz.tools/fm package also contains a modulator, which has been tested on the air and with some of my handheld radios. This code is a bit old, since the hz.tools/fm code is effectively the first IQ processing code I d ever written, but it still runs and I run it from time to time. 
 // Basic sketch for playing FM radio using a reader stream from
 // an SDR or other IQ stream.

bandwidth := 150*rf.KHz
reader, err = stream.ConvertReader(reader, sdr.SampleFormatC64)
if err != nil  
...
 
demod, err := fm.Demodulate(reader, fm.DemodulatorConfig 
Deviation: bandwidth / 2,
Downsample: 8, // some value here depending on sample rate
 Planner: fftw.Plan,
 )
if err != nil  
...
 
speaker, err := pulseaudio.NewWriter(pulseaudio.Config 
Format: pulseaudio.SampleFormatFloat32NE,
Rate: demod.SampleRate(),
AppName: "rf",
StreamName: "fm",
Channels: 1,
SinkName: "",
 )
if err != nil  
...
 
buf := make([]float32, 1024*64)
for  
i, err := demod.Read(buf)
if err != nil  
...
 
if i == 0  
panic("...")
 
if err := speaker.Write(buf[:i]); err != nil  
...

hz.tools/rfcap - byte serialization for IQ data The hz.tools/rfcap package is the reference implementation of the rfcap spec , and is how I store IQ captures locally, and how I send them across a byte boundary. The git repo can be found at github.com/hztools/go-rfcap, and is importable as hz.tools/rfcap. If you re interested in storing IQ in a way others can use, the better approach is to use SigMF rfcap exists for cases like using UNIX pipes to move IQ around, through APIs, or when I send IQ data through an OS socket, to ensure the sample format (and other metadata) is communicated with it. rfcap has a number of limitations, for instance, it can not express a change in frequency or sample rate during the capture, since the header is fixed at the beginning of the file.

22 February 2023

Dirk Eddelbuettel: RcppArmadillo 0.12.0.1.0 on CRAN: New Upstream, New Features

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra and scientific computing. It aims towards a good balance between speed and ease of use, has a syntax deliberately close to Matlab, and is useful for algorithm development directly in C++, or quick conversion of research code into production environments. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 1042 other packages on CRAN, downloaded 28.1 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 513 times according to Google Scholar. This release brings a new upstream release 12.0.1. We found a small regression with the 12.0.0 release when we tested prior to a CRAN upload. Conrad very promptly fixed this with a literal one liner and made it 12.0.1 which we wrapped up as 0.12.0.1.0. Subsequent testing revealed no issues for us, and CRAN autoprocessed it as I tweeted earlier. This is actually quite impressive given the over 1000 CRAN packages using it all of which got tested again by CRAN. All this is testament to the rigour, as well as the well-oiled process at the repository. Our thanks go to the tireless maintainers! The releases actually has a rather nice set of changes (detailed below) to which we added one robustification thanks to Kevin. The full set of changes follows. We include the previous changeset as we may have skipped the usual blog post here.

Changes in RcppArmadillo version 0.12.0.1.0 (2023-02-20)
  • Upgraded to Armadillo release 12.0.1 (Cortisol Profusion)
    • faster fft() and ifft() via optional use of FFTW3
    • faster min() and max()
    • faster index_min() and index_max()
    • added .col_as_mat() and .row_as_mat() which return matrix representation of cube column and cube row
    • added csv_opts::strict option to loading CSV files to interpret missing values as NaN
    • added check_for_zeros option to form 4 of sparse matrix batch constructors
    • inv() and inv_sympd() with options inv_opts::no_ugly or inv_opts::allow_approx now use a scaled threshold similar to pinv()
    • set_cout_stream() and set_cerr_stream() are now no-ops; instead use the options ARMA_WARN_LEVEL, or ARMA_COUT_STREAM, or ARMA_CERR_STREAM
    • fix regression (mis-compilation) in shift() function (reported by us in #409)
  • The include directory order is now more robust (Kevin Ushey in #407 addressing #406)

Changes in RcppArmadillo version 0.11.4.4.0 (2023-02-09)
  • Upgraded to Armadillo release 11.4.4 (Ship of Theseus)
    • extended pow() with various forms of element-wise power operations
    • added find_nan() to find indices of NaN elements
    • faster handling of compound expressions by sum()
  • The package no longer sets a compilation standard, or progagates on in the generated packages as R ensures C++11 on all non-ancient versions
  • The CITATION file was updated to the current format

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

21 February 2023

Billy Warren: A straight Guide to Salsa CI - A Debian Continuous Integration tool

I won t waste your time with introductions. The title says it all so let s jump right in. I ll give you as many links as possible so that this article stays as short as possible. So first, what is Salsa? Salsa is a name of a GitLab instance that is used by Debian teams to manage Debian packages and also collaborate on Development. If you have used GitLab before, the Salsa platform is not any different. To have a feel of it, it is available at https://salsa.debian.org. Still, want to know more? Find more information in the wiki. Intrigued to a point of getting started? Setup up your account by following this information Secondly, what is Salsa CI? Like many large projects with different contributors and strict maintenance, Debian is no different. This Linux distribution is made up of many packages which need to follow a certain standard and structure or purpose of compatibility, scalability and maintainability. The Salsa CI is a continuous integration tool that does just that. I hope that is precise and satisfying . I would have ended here but since our focus is Salsa CI tool, let me get a little deeper and wider. You could also make great use of your time when I provide more information. The Salsa CI was developed to continuously check for the health of Debian packages before they can be uploaded to the archive by running a series of CI/CD jobs. The jobs are run against setup images that are already uploaded and updated regularly to reduce build time. A screenshot showing the Salsa Ci jobs. The use of Salsa CI is becoming prominent ever since its inception. The Salsa CI pipeline has become popular (used by ~8k projects, from MariaDB to the Linux kernel packaging), and it is even the base for more complex CI pipelines used by other Linux flavours. The issue is the more popular it becomes, the more efficient it has to get and the more need to make the build time as shorter as possible. This happens by iterating and testing out different tools during different stages of the pipeline to find the best industrial tool. This is one of the priorities for anyone who develops for or maintains Salsa CI. So that is how deep I can go for now. But wait, what if you what to contribute?
If you have working knowledge in bash, git, CI, python and knowledge in building Debian packages it could be easy for you to figure out where components are and how they interact with each other. What if you don t have the knowledge? Then that is where the fun comes in. Getting started on making a meaningful contribution to Salsa CI will need more passion and discipline, the expertise comes later and slowly. I have contributed to Salsa CI even without high-level expertise and knowledge in some of the tools. When I started contributing to Salsa CI what a Debian package is, I even didn t know that the tool that I am trying to navigate is being used by prominent software teams. But it is the challenge that I set for myself that as of now, enabled me to be able to work on a crucial part of the whole Continous integration. Wanna know what it is? I am, as at the time of writing this article, integrating sbuild into Salsa CI to replace it with dpkg-buildpackage. This in turn will help to reduce the build time by getting rid of some jobs hence making the CI work faster. Cool, right? Contributing to such a significant project can be a little challenging at the start but when you realize how important the piece you are working on is, you suddenly fall in love with it and want to follow through so that you can also be part of the large community that helps to make this world a better place in obscure ways. So why don t you check out some of the Salsa CI open issues and see if you d be interested in improving it?

Freexian Collaborators: Monthly report about Debian Long Term Support, January 2023 (by Anton Gladky)

Like each month, have a look at the work funded by Freexian s Debian LTS offering. This is the first monthly report in 2023.

Debian LTS contributors In January, 17 contributors have been paid to work on Debian LTS. which is possibly the highest number of active contributors per month! Their reports are available:
  • Abhijith PA did 0.0h (out of 3.0h assigned and 11.0h from previous period), thus carrying over 14.0h to the next month.
  • Adrian Bunk did 26.25h (out of 26.25h assigned).
  • Anton Gladky did 11.5h (out of 8.0h assigned and 7.0h from previous period), thus carrying over 3.5h to the next month.
  • Ben Hutchings did 8.0h (out of 24.0h assigned), thus carrying over 16.0h to the next month.
  • Chris Lamb did 18.0h (out of 18.0h assigned).
  • Emilio Pozuelo Monfort did 8.0h (out of 0h assigned and 43.0h from previous period), thus carrying over 35.0h to the next month.
  • Guilhem Moulin did 20.0h (out of 17.5h assigned and 2.5h from previous period).
  • Helmut Grohne did 10.0h (out of 15.0h assigned), thus carrying over 5.0h to the next month.
  • Lee Garrett did 7.5h (out of 20.0h assigned), thus carrying over 12.5h to the next month.
  • Markus Koschany did 26.25h (out of 26.25h assigned).
  • Ola Lundqvist did 4.5h (out of 10.0h assigned and 6.0h from previous period), thus carrying over 11.5h to the next month.
  • Roberto C. S nchez did 3.75h (out of 18.75h assigned and 7.5h from previous period), thus carrying over 22.5h to the next month.
  • Stefano Rivera did 4.5h (out of 0h assigned and 32.5h from previous period), thus carrying over 28.0h to the next month.
  • Sylvain Beucler did 23.5h (out of 0h assigned and 38.5h from previous period), thus carrying over 15.0h to the next month.
  • Thorsten Alteholz did 14.0h (out of 10.0h assigned and 4.0h from previous period).
  • Tobias Frost did 19.0h (out of 19.0h assigned).
  • Utkarsh Gupta did 43.25h (out of 26.25h assigned and 17.0h from previous period).

Evolution of the situation Furthermore, we released 46 DLAs in January, which resolved 146 CVEs. We are working diligently to reduce the number of packages listed in dla-needed.txt, and currently, we have 55 packages listed. We are constantly growing and seeking new contributors. If you are a Debian Developer and want to join the LTS team, please contact us.

Thanks to our sponsors Sponsors that joined recently are in bold.

20 February 2023

Jonathan McDowell: Fixing mobile viewing

It was brought to my attention recently that the mobile viewing experience of this blog was not exactly what I d hope for. In my poor defence I proof read on my desktop and the only time I see my posts on mobile is via FreshRSS. Also my UX ability sucks. Anyway. I ve updated the theme to a more recent version of minima and tried to make sure I haven t broken it all in the process (I did break tagging, but then I fixed it again). I double checked the generated feed to confirm it was the same (other than some re-tagging I did), so hopefully I haven t flooded anyone s feed. Hopefully I can go back to ignoring the underlying blog engine for another 5+ years. If not I ll have to take a closer look at Enrico s staticsite.

19 February 2023

Russell Coker: New 18 Core CPU and NVMe

I just got a E5-2696 v3 CPU for my ML110 Gen9 home workstation, this has a Passmark score of 23326 which is almost 3 times faster than the E5-2620 v4 which rated 9224. Previously it took over 40 minutes real time to compile a 6.10 kernel that was based on the Debian kernel configuration, now it takes 14 minutes of real time, 202 minutes of user time, and 37 minutes of system CPU time. That s a definite benefit of having a faster CPU, I don t often compile kernels but when I do I don t want to wait 40+ minutes for a result. I also expanded the system from 96G of RAM to 128G, most of the time I don t need so much RAM but it s better to have too much than too little, particularly as my friend got me a good deal on RAM. The extra RAM might have helped improve performance too, going from 6/8 DIMM slots full to 8/8 might help the CPU balance access. That series of HP machines has a plastic mounting bracket for the CPU, see this video about the HP Proliant Smart Socket for details [1]. I was working on this with a friend who has the same model of HP server as I do, after buying myself a system I was so happy with it that I bought another the same when I saw it going for a good price and then sold it to my friend when I realised that I had too many tower servers at home. It turns out that getting the same model of computer as a friend is a really good strategy so then you can work together to solve problems with it. My friend s first idea was to try and buy new clips for the new CPUs (which would have delayed things and cost more money), but Reddit and some blog posts suggested that you can just skip the smart-socket guide clip and when the chip was resting in the socket it felt secure as the protrusions on the sides of the socket fit firmly enough into the notches in the CPU to prevent it moving far enough to short a connection. Testing on 2 systems showed that you don t need the clip. As an aside it would be nice if Intel made every CPU that fits a particular socket have the same physical dimensions so clips and heatsinks can work well on all CPUs. The TDP of the new CPU is 145W and the old one was 85W. One would hope that in a server class system that wouldn t make a lot of difference but unfortunately the difference was significant. Previously I could have the system running 7/8 cores with BOINC 24*7 and I wouldn t notice the fans being louder. It is possible that 100% CPU use on a hot day might make the fans sound louder if I didn t have an air-conditioner on that was loud enough to drown them out, but the noteworthy fact is that with the previous CPU the system fans were a minor annoyance. Now if I have 16 cores running BOINC it s quite loud, the sort of noise that makes most people avoid using tower servers as workstations! I ve found that if I limit it to 4 or 5 cores then the system is about as quiet as it was before. As a rough approximation I can use as much CPU power as before without making the fans louder but if I use more CPU power than was previously available it gets noisy. I also got some new NVMe devices, I was previously using 2*Crucial 1TB P1 NVMes in a BTRFS RAID-1 and now I have 2*Crucial 1TB P3 NVMes (where P1 is the slowest Crucial offering, P3 is better and more expensive, P5 is even better, etc). When doing the BTRFS migrations to move my workstation to new NVMe devices and my server to the old NVMe devices I found that the P3 series seem to have a limit of about 70MB/s for sustained random writes and the P1 series is about 35MB/s. Apparently with the cheaper NVMe devices they slow down if you do lots of random writes, pity that all the review articles talking about GB/s speeds don t mention this. To see how bad reviews are Google some reviews of these SSDs, you will find a couple of comment threads on places like Reddit of them slowing down with lots of writes and lots of review articles on well known sites that don t mention it. Generally I d recommend not upgrading from P1 to P3 NVMe devices, the benefit isn t enough to cover the effort. For every capacity of NVMe devices the most expensive devices cost more than twice as much as the cheapest devices, and sometimes it will be worth the money. Getting the most expensive device won t guarantee great performance but getting cheap devices will guarantee that it s slow. It seems that CPU development isn t progressing as well as it used to, the CPU I just bought was released in 2015 and scored 23,343 according to Passmark [2]. The most expensive Intel CPU on offer at my local computer store is the i9-13900K which was released this year and scores 62,914 [3]. One might say that CPUs designed for servers are different from ones designed for desktop PCs, but the i9 in question has a TDP Up of 253W which is too big for the PSU I have! According to the HP web site the new ML110 Gen10 servers aren t sold with a CPU as fast as the E5-2696 v3! In the period from 1988 to about 2015 every year there were new CPUs with new capabilities that were worth an upgrade. Now for the last 8 years or so there hasn t been much improvement at all. Buy a new PC for better USB ports or something not for a faster CPU!
Related posts:
  1. SSD Endurance I previously wrote about the issue of swap potentially breaking...
  2. core files The issue of core file management has come up for...
  3. A Good Time to Upgrade PCs PC hardware just keeps getting cheaper and faster. Now that...

17 February 2023

Enrico Zini: Monitoring a heart rate monitor

I bought myself a cheap wearable Bluetooth LE heart rate monitor in order to play with it, and this is a simple Python script to monitor it and plot data. Bluetooth LE I was surprised that these things seem decently interoperable. You can use hcitool to scan for devices: 
hcitool lescan
You can then use gatttool to connect to device and poke at them interactively from a command line. Bluetooth LE from Python There is a nice library called Bleak which is also packaged in Debian. It's modern Python with asyncio and works beautifully! Heart rate monitors Things I learnt: How about a proper fitness tracker? I found OpenTracks, also on F-Droid, which seems nice Why script it from a desktop computer? The question is: why not? A fitness tracker on a phone is useful, but there are lots of silly things one can do from one's computer that one can't do from a phone. A heart rate monitor is, after all, one more input device, and there are never enough input devices! There are so many extremely important use cases that seem entirely unexplored:

Jonathan McDowell: First impressions of the VisionFive 2

VisionFive 2 packaging Back in September last year I chose to back the StarFive VisionFive 2 on Kickstarter. I don t have a particular use in mind for it, but I felt it was one of the first RISC-V systems that were relatively capable (mentally I have it as somewhere between a Raspberry Pi 3 + a Pi 4). In particular it s a quad 1.5GHz 64-bit RISC-V core with 8G RAM, USB3, GigE ethernet and a single M.2 PCIe slot. More than ample as a personal machine for playing around with RISC-V and doing local builds. I ended up paying 67 for the Early Bird variant (dual GigE ethernet rather than 1 x 100Mb and 1 x GigE). A couple of weeks ago I got an email with a tracking number and last week it finally turned up. Being impatient the first thing I did was plug it into a monitor, connect up a keyboard, and power it on. Nothing except some flashing lights. Looking at the boot selector DIP switches suggested it was configured to boot from UART, so I flipped them to (what I thought was) the flash setting. It wasn t - turns out the ON marking on the switches represents logic 0 and it was correctly setup when I got it. I went to read the documentation which talked about writing an image to a MicroSD card, but also had details of the UART connection. Wanting to make sure the device was at least doing something before I actually tried an OS on it I hooked up a USB/serial dongle and powered the board up again. Success! U-Boot appeared and I could interact with it. I went to the VisionFive2 Debian page and proceeded to torrent the Image-69 image, writing it to a MicroSD card and inserting it in the slot on the bottom of the board. It booted fine. I can t even tell you what graphical environment it booted up because I don t remember; it worked fine though (at 1080p, I ve seen reports that 4K screens will make it croak). Poking around the image revealed that it s built off a snapshot.debian.org snapshot from 20220616T194833Z, which is a little dated at this point but I understand the rationale behind picking something that works and sticking with it. The kernel is of course a vendor special, based on 5.15.0. Further investigation revealed that the entire X/graphics stack is living in /usr/local, which isn t overly surprising; it s Imagination based. I was pleasantly surprised to discover there is work to upstream the Imagination support, but I m not planning to run the board with a monitor attached so it s not a high priority for me. Having discovered all that I decided to see how well a clean Debian unstable install from Debian Ports would go. I had a spare Intel Optane lying around (it s a stupid 22110 M.2 which is too long for any machine I own), so I put it in the slot on the bottom of the board. To my surprise it Just Worked and was detected ok: 
# lspci
0000:00:00.0 PCI bridge: PLDA XpressRich-AXI Ref Design (rev 02)
0000:01:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller (rev 01)
0001:00:00.0 PCI bridge: PLDA XpressRich-AXI Ref Design (rev 02)
0001:01:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]
I created a single partition with an ext4 filesystem (initially tried btrfs, but the StarFive kernel doesn t support it), and kicked off a debootstrap with: 
# mkfs -t ext4 /dev/nvme0n1p1
# mount /dev/nvme0n1p1 /mnt
# debootstrap --keyring=/etc/apt/trusted.gpg.d/debian-ports-archive-2023.gpg \
	unstable /mnt https://deb.debian.org/debian-ports
The u-boot setup has a convoluted set of vendor scripts that eventually ends up reading a /boot/extlinux/extlinux.conf config from /dev/mmcblk1p2, so I added an additional entry there using the StarFive kernel but pointing to the NVMe device for /. Made sure to set a root password (not that I ve been bitten by that before, too many times), and rebooted. Success! Well. Sort of. I hit a bunch of problems with having a getty running on ttyS0 as well as one running on hvc0. The second turns out to be a console device from the RISC-V SBI. I did a systemctl mask serial-getty@hvc0.service which made things a bit happier, but I was still seeing odd behaviour and output. Turned out I needed to reboot the initramfs as well; the StarFive one was using Plymouth and doing some other stuff that seemed to be confusing matters. An update-initramfs -k 5.15.0-starfive -c built me a new one and everything was happy. Next problem; the StarFive kernel doesn t have IPv6 support. StarFive are good citizens and make their 5.15 kernel tree available, so I grabbed it, fed it the existing config, and tweaked some options (including adding IPV6 and SECCOMP, which chrony wanted). Slight hiccup when it turned out trying to do things like make sound modular caused it to fail to compile, and having to backport the fix that allowed the use of GCC 12 (as present in sid), but it got there. So I got cocky and tried to update it to the latest 5.15.94. A few manual merge fixups (which I may or may not have got right, but it compiles and boots for me), and success. Timings: 
$ time make -j 4 bindeb-pkg
  [linux-image-5.15.94-00787-g1fbe8ac32aa8]
real	37m0.134s
user	117m27.392s
sys	6m49.804s
On the subject of kernels I am pleased to note that there are efforts to upstream the VisionFive 2 support, with what appears to be multiple members of StarFive engaging in multiple patch submission rounds. It s really great to see this and I look forward to being able to run an unmodified mainline kernel on my board. Niggles? I have a few. The provided u-boot doesn t have NVMe support enabled, so at present I need to keep a MicroSD card to boot off, even though root is on an SSD. I m also seeing some errors in dmesg from the SSD: 
[155933.434038] nvme nvme0: I/O 436 QID 4 timeout, completion polled
[156173.351166] nvme nvme0: I/O 48 QID 3 timeout, completion polled
[156346.228993] nvme nvme0: I/O 108 QID 3 timeout, completion polled
It doesn t seem to cause any actual issues, and it could be the SSD, the 5.15 kernel or an actual hardware thing - I ll keep an eye on it (I will probably end up with a different SSD that actually fits, so that ll provide another data point). More annoying is the temperature the CPU seems to run at. There s no heatsink or fan, just the metal heatspreader on top of the CPU, and in normal idle operation it sits at around 60 C. Compiling a kernel it hit 90 C before I stopped the job and sorted out some additional cooling in the form of a desk fan, which kept it as just over 30 C. Bare VisionFive 2 SBC board with a small desk fan pointed at it I haven t seen any actual stability problems, but I wouldn t want to run for any length of time like that. I ve ordered a heatsink and also realised that the board supports a Raspberry Pi style PoE Hat , so I ve got one of those that includes a fan ordered (I am a complete convert to PoE especially for small systems like this). With the desk fan setup I ve been able to run the board for extended periods under load (I did a full recompile of the Debian 6.1.12-1 kernel package and it took about 10 hours). The M.2 slot is unfortunately only a single PCIe v2 lane, and my testing topped out at about 180MB/s. IIRC that is about half what the slot should be capable of, and less than a 10th of what the SSD can do. Ethernet testing with iPerf3 sustained about 941Mb/s, so basically maxing out the port. The board as a whole isn t going to set any speed records, but it s perfectly usable, and pretty impressive for the price point. On the Debian side I ve not hit any surprises. There s work going on to move RISC-V to a proper release architecture, and I m hoping to be able to help out with that, but the version of unstable I installed from the ports infrastructure has looked just like any other Debian install. Which is what you want. And that pretty much sums up my overall experience of the VisionFive 2; it s not noticeably different than any other single board computer. That s a good thing, FWIW, and once the kernel support lands properly upstream (it ll be post 6.3 at least it seems) it ll be a boring mainline supported platform that just happens to be RISC-V.

Valhalla's Things: git status Side Effects

Posted on February 17, 2023
TIL, from a conversation with friends1, that git status can indeed have side effects, of some sort. By default, running git status causes a background refresh of the index to happen, which holds the write lock on the repository. In theory, if somebody is really unlucky, this could break some script / process that is also trying to work on the repo at the same time, especially on a huge repository where git status takes a significant time, rather than the usual fraction of a second2. There is a way to prevent this, by running git status --no-optional-locks (https://git-scm.com/docs/git#Documentation/git.txt---no-optional-locks) or by setting GIT_OPTIONAL_LOCKS to 0, as writing the updated index is just an optimization and git knows it can be avoided. I don t think there are many chances to actually stumble on this in the real life, but I m writing this down so that if I ever do I have an easy way to remember what happened and find the solution.
  1. I won t name name or provide details to protect the innocents (and the guilty), but thanks to all of the people involved in the conversations who helped find the answer.
  2. Related, but unrelated TIL: there is a place called Secondo (second), near Venice, but it s already a frazione (fraction / subdivision of municipality).

Reproducible Builds (diffoscope): diffoscope 236 released

The diffoscope maintainers are pleased to announce the release of diffoscope version 236. This version includes the following changes: 
[ FC Stegerman ]
* Update code to match latest version of Black. (Closes: #1031433)
[ Chris Lamb ]
* Require at least Black version 23.1.0 to run the internal Black tests.
* Update copyright years.
You find out more by visiting the project homepage.

16 February 2023

Scarlett Gately Moore: KDE Snaps, Security updates, Debian Freeze

Icy morning Witch Wells AzIcy morning Witch Wells Az
Much like our trees, Debian is now in freeze stage for Bookworm. I am still working on packages locally until development opens up again. My main focus is getting mycroft packages updated to the new fork at https://github.com/orgs/OpenVoiceOS/repositories. On the KDE Snaps side of things: My PPA is not going well. There is a problem in Focal here qhelpgenerator-qt5 is a missing dependency, HOWEVER it is there as shown here: https://launchpad.net/ubuntu/focal/+package/qhelpgenerator-qt5 a fun circular dependency for qtbase. It also builds fine in a focal chroot. I have tried copying packages, source recipe builds, adding another PPA with successful builds, to no avail. My PPA does not seem to use its own packages, or universe for that matter. I have tried all of the dependency settings available and nothing changes. If any of you launchpad experts out there want to help me out, or point me in the right direction, it would be appreciated ! This PPA is mandatory for our core20 snaps moving forward, as they currently have no security updates, and I refuse to have my name on security riddled snaps. As for the kde-neon extension for core22, I have fixed most of the tests and Sergio is wrapping it up, thank you!!!! I am still looking for work!!! Please reach out if you, or anyone you know is looking for my skill set. Once again, I ask for you to please consider a donation. We managed to get the bills paid this month ( Big thank you! ) but, March is quickly approaching. The biggest thing is Phone/Internet so I can keep working on things and Job hunt. Thank you so much for your support! https://gofund.me/a9c36b87

Gunnar Wolf: We are GREAT at handling multimedia!

I have mentioned several times in this blog, as well as by other communication means, that I am very happy with the laptop I bought (used) about a year and a half ago: an ARM-based Lenovo Yoga C630. Yes, I knew from the very beginning that using this laptop would pose a challenge to me in many ways, as full hardware support for ARM laptops are nowhere as easy as for plain boring x86 systems. But the advantages far outweigh the inconvenience (i.e. the hoops I had to jump through to handle video-out when I started teaching presentially, which are fortunately a thing of the past now). Anyway This post is not about my laptop. Back in 2018, I was honored to be appointed as a member of the Debian Technical Committee. Of course, that meant (due to the very clear and clever point 6.2.7.1 of the Debian Constitution that my tenure in the Committee (as well as Niko Tyni s) finished in January 1, 2023. We were invited to take part of a Jitsi call as a last meeting, as well as to welcome Matthew Garrett to the Committee. Of course, I arranged so I would be calling from my desktop system at work (for which I have an old, terrible webcam but as long as I don t need to control screen sharing too finely, mostly works). Out of eight people in the call, two had complete or quite crippling failures with their multimedia setup, and one had a frozen image (at least as far as I could tell). So Yes, Debian is indeed good and easy and simple and reliable for most nontechnical users using standard tools. But I guess that we power users enjoy tweaking our setup to our precise particular liking. Or that we just don t care about frivolities such as having a working multimedia setup. Or I don t know what happens. But the fact that close to half of the Technical Committee, which should consist of Debian Developers who know their way around technical obstacles, cannot get a working multimedia setup for a simple, easy WebRTC call (even after a pandemic that made us all work via teleconferencing solutions on a daily basis!) is just Beautiful

15 February 2023

Marco d'Itri: I replaced grub with systemd-boot

To be able to investigate and work on the the measured boot features I have switched from grub to systemd-boot (sd-boot). This initial step is optional, but it is useful because this way /etc/kernel/cmdline will become the new place where the kernel command line can be configured: 
. /etc/default/grub
echo "root=/dev/mapper/root $GRUB_CMDLINE_LINUX $GRUB_CMDLINE_LINUX_DEFAULT" > /etc/kernel/cmdline
Do not forget to set the correct root file system there, because initramfs-tools does not support discovering it at boot time using the Discoverable Partitions Specification. The installation has been automated since systemd version 252.6-1, so installing the package has the effect of installing sd-boot in the ESP, enabling it in the UEFI boot sequence and then creating boot loader entries for the kernels already installed on the system: 
apt install systemd-boot
If needed, it could be manually installed again just by running bootctl install. I like to show the boot menu by default, at least until I will be more familiar with sd-boot: 
bootctl set-timeout 4
Since other UEFI binaries can be easily chainloaded, I am also going to keep around grub for a while, just to be sure: 
cat <<END > /boot/efi/loader/entries/grub.conf
title Grub
linux /EFI/debian/grubx64.efi
END
At this point sd-boot works, but I still had to enable secure boot. So far sd-boot has not been signed with a Debian key known to the shim bootloader, so I needed to create a Machine Owner Key (MOK), enroll it in UEFI and then use it to sign everything. I dislike the complexity of mokutil and the other related programs, so after removing it and the boot shim I have decided to use sbctl instead. With it I easily created new keys, enrolled them in the EFI key store and then signed everything: 
sbctl create-keys
sbctl enroll-keys
for file in /boot/efi/*/*/linux /boot/efi/EFI/*/*.efi; do
  sbctl sign -s $file
done
Since there is no sbctl package yet I need to make sure that also the kernels installed in the future will be automatically signed, so I have created a trivial script in /etc/kernel/install.d/ which automatically runs sbctl sign -s or sbctl remove-file. The Debian wiki SecureBoot page documents how do do this with mokutil and sbsigntool, but I think that sbctl is much friendlier. Since I am not using the boot shim, I also had to set DisableShimForSecureBoot=true in /etc/fwupd/uefi_capsule.conf to make firmware updates work automatically. As a bonus, I have also added to the boot menu the excellent Debian-based GRML live distribution. Since sd-boot is not capable of loopback-mounting CD-ROM images like grub, I first had to extract the kernel and initramfs and copy them to the ESP: 
mount -o loop /boot/grml/grml64-full_2022.11.iso /mnt/
mkdir /boot/efi/grml/
cp /mnt/boot/grml64full/* /boot/efi/grml/
umount /mnt/
cat <<END > /boot/efi/loader/entries/grml.conf
title GRML
linux /grml/vmlinuz
initrd /grml/initrd.img
options boot=live bootid=grml64full202211 findiso=/grml/grml64-full_2022.11.iso live-media-path=/live/grml64-full net.ifnames=0 
END
As expected, after a reboot bootctl reports the new security features: 
System:
      Firmware: UEFI 2.70 (Lenovo 0.4496)
 Firmware Arch: x64
   Secure Boot: enabled (user)
  TPM2 Support: yes
  Boot into FW: supported
Current Boot Loader:
      Product: systemd-boot 252.5-2
     Features:   Boot counting
                 Menu timeout control
                 One-shot menu timeout control
                 Default entry control
                 One-shot entry control
                 Support for XBOOTLDR partition
                 Support for passing random seed to OS
                 Load drop-in drivers
                 Support Type #1 sort-key field
                 Support @saved pseudo-entry
                 Support Type #1 devicetree field
                 Boot loader sets ESP information
          ESP: /dev/disk/by-partuuid/1b767f8e-70fa-5a48-b444-cfe5c272d66e
         File:  /EFI/systemd/systemd-bootx64.efi
...
Relevant documentation:

Lukas M rdian: Netplan v0.106 is now available

I m happy to announce that Netplan version 0.106 is now available on GitHub and is soon to be deployed into an Ubuntu/Debian/Fedora installation near you! Six months and 65 commits after the previous version, this release is brought to you by 4 free software contributors from around the globe. Highlights Highlights of this release include the new netplan status command, which queries your system for IP addresses, routes, DNS information, etc in addition to the Netplan backend renderer (NetworkManager/networkd) in use and the relevant Netplan YAML configuration ID. It displays all this in a nicely formatted way (or alternatively in machine readable YAML/JSON format).
Furthermore, we implemented a clean libnetplan API which can be used by external tools to parse Netplan configuration, migrated away from non-inclusive language (PR#303) and improved the overall Netplan documentation. Another change that should be noted, is that the match.macaddress stanza now only matches on PermanentMACAddress= on the systemd-networkd backend, as has been the case on the NetworkManager backend ever since (see PR#278 for background information on this slight change in behavior). Changelog Bug fixes:

Valhalla's Things: My experience with a PinePhone

Posted on February 15, 2023
I ve had and used1 a PinePhone for quite some time now, and a shiny new blog sounds like a good time to do a review of my experience. TL;DL: I love it, but my use cases may not be very typical. While I ve had a mobile phone since an early time (my parents made me carry one for emergencies before it was usual for my peers) I ve never used a typical smartphone (android / iPhone / those other proprietary things) because I can t trust them not to be designed to work against me (data collection, ads, tricking users into micro payments, and other antipatterns of proprietary software design), and bringing them to sanity as most of the people I know do is too much effort for my tastes. Instead, as a phone I keep using an old nokia featurephone2 which is reliable, only requires charging once a week, and can easily survive falling from hand height to the ground, even if thrown 3. And then I ve been carrying a variety of other devices to do other computer-like tasks; earlier it was just a laptop, or a netbook, a Pandora (all of which used a dongle to connect to mobile network internet) then I tried a phone with FirefoxOS (it could have been better) and now the PinePhone has taken their place, at least when I m not carrying a laptop anyway. So, the tasks I use the PinePhone for are mostly:
  1. sending and receiving xmpp messages, with no need for notifications (when somebody needs to tell me something where urgency is required, they know to use the other phone, either with a call or an sms);
  2. tethering an internet connection to the laptop;
  3. reading djvu scans of old books while standing in a queue or something;
  4. checking something on the internet when I m not close to a real computer (i.e. one with a keyboard and a big screen);
  5. running a timer while heat-setting fabric paint with an iron (and reading a djvu book at the same time yes, this is a very specific task, but it has happened multiple times already :D );
  6. running a calculator with unit conversions;
  7. running the odd command line program;
  8. taking pictures, especially those that I want to send soon (I often also carry a DSLR camera, but I tend to wait a few days before I download them from the card);
  9. map related things.
3 and 5 work perfectly well, no issues there. 1, 2 and 4 usually work just fine, except for the fact that sometimes while the phone is suspended it forgets about being a phone, and needs to be restarted to turn the modem back on. It s not a big deal while using the phone, I just need to check before I try to use it after a few hours. For 1, I also had to take care to install dino-im from experimental, as up to now the features required to fit the interface in a mobile screen aren t available in the official release, but I believe that this has just been fixed. Somewhat related to 4, I ve also installed kiwix and the dumps of wiktionary and wikivoyage, but I haven t had a chance to travel much, so I m not really using it. For 6 I m quite happy with qalculate, the GUI version of qalc (which is what I use on my laptop), even if it has a few minor interface issues, and 7 of course works as well as it can, given the limitations of a small screen and virtual keyboard. 8 is, let us say, problematic. The camera on the PinePhone is peculiar, only works with a specific software, and even there the quality of the pictures is, well, low-fi, vintagey pictures are a look and that s a specific artistic choice, right? Thanks to the hard work of the megapixels maintainer the quality has improved a lot, and these days it is usable, but there are still limits (no webcam in the browser, no recording of videos). 9 is really bad. A few times I can remember getting a GPS fix. A few times in many months, and now and then I keep trying, to see if a miracle has happened, but usually I only get a vague position from wifi data (which isn t great, when walking through less densely populated areas). I ve seen another PinePhone running gpsd and getting data from an external GPS receiver via bluetooth, and if I really needed it I may seriously consider that solution. Also, the apps available in mobian aren t great either, even when compared to running tangogps on an OpenMoko with pre-downloaded maps (I mean! I don t think my expectations are too high!). I ve heard that PureMaps is quite good as a software, but a bit of a PITA to package for Debian, and I really hope that one day there will be a linux-first mobile device with good GPS hardware, so that people will be encouraged to fix the software side. Thankfully, I don t usually need GPS and navigation software; when I m driving into places I don t know I usually have a human navigator, and when walking into places I can do with just a static map (either printed on paper or on the PinePhone), maybe some pre-calculated route from the OSM website and looking at street names to find out where I am. Overall, for my use cases the PinePhone works just fine and is an useful addition to the things I always carry with me, and I don t feel the pressing need to get an android phone. I don t think it s ready as a daily driver for everybody, but I think that depending on one s needs it s worth asking around (I d recommend doing so on the fediverse with a #PinePhone and #mobian hashtag), as there is a non-zero chance that it may be a good fit for you.
  1. which, if I m not mistaken, is often not implied by the fact of owning it :D
  2. for very low values of feature: it doesn t have any kind of internet access, and there are only 3 games, one of which is snake.
  3. don t ask.

14 February 2023

Holger Levsen: 20230214-i-love-osuosl

I love free software and I OSUOSL So in December 2018 I was approached somewhat out of the blue by someone from OSUOSL who offered eight servers to the Reproducible Builds project and as these machines had 32 cores and 144 GB Ram each (plus 3 TB on a single HDD) and they also offered free hosting, I very happyly said yes. And since them I'm a very happy Oregon State University Open Source Labs user, and these days we're switching the setup to different machines, which is another story to tell some other time...! ;-) My point now is: since 2018 I got to know OSUOSL and every year I like them more. They are super friendly, reliable (a working ticket system and a great IRC channel), offer help in various ways, be it with DNS names (and renamings...) or finding new hardware suited to our needs or whatever else we come up with. They are really dedicated to help free software projects and I'm grateful having the privilege to enjoy this since more than four years now. To quote https://osuosl.org/: "The Open Source Lab is a nonprofit organization working for the advancement of open source technologies. The lab, in partnership with the School of Electrical Engineering and Computer Science at Oregon State University, provides hosting for more than 160 projects, including those of worldwide leaders like the Apache Software Foundation, the Linux Foundation and Drupal. Together, the OSL s hosted sites deliver nearly 430 terabytes of information to people around the world every month. The most active organization of its kind, the OSL offers world-class hosting services, professional software development and on-the-ground training for promising students interested in open source management and programming." So, IOW, I'm just seeing a very tiny tip of the iceberg of their awesome work. Check out https://osuosl.org/communities/ to see what I mean with very tiny tip. Search for "Debian" and "Reproducible" on that page :) Thank you, everybody at OSUOSL! You rock and make a big difference for many projects!

13 February 2023

Jonathan Dowland: A visit to Prusa Labs

.
.
In September I was in Czechia for a Red Hat event. I ended up travelling via Prague, and had an unexpected extra day due to an airline strike causing my flight home to be cancelled. I took the opportunity to visit Prusa's offices/factory/Lab, and it was amazing! The Prusa team were all busy getting ready for the Prague Maker Faire that was happening the day afterwards.1 On arriving at the street which houses Prusa's Lab and Office buildings, the first thing that hit me was the smell. I find the melted-plastic smell of FDM printing (with PLA, at least) quite pleasant, and this was a super-condensed version of that, pumping out of their ground-floor windows. I started at the reception area on the ground floor. Outside reception there's a lovely sculpture representing the history of the development of the MK3S+.
The Reception The Reception
SLS Farm in the former Hack lab SLS Farm in the former Hack lab
History of the MK3S+ History of the MK3S+
At the reception you have a small waiting area with shelves of demonstration prints and some spools of Prusament. From here, our kind guide first took us to a region on the ground floor that used to be (prior to COVID times) the public maker/hack lab. The lab contained two modest farms of printers: one of their flagship FDM printer, the MK3S+, and another of their SLS resin printers. A close up of some example resin models is pictured above. The bicycle was tiny: about the size of a thumbnail.
A Historic display A Historic display
Bespoke QE equipment Bespoke QE equipment
The MK3S+ Farm The MK3S+ Farm
. Moi in the Farm
2KG orange PLA spools for the Farm 2KG orange PLA spools for the Farm
The rest of the ground floor area was full of heavy machinery and prototyping equipment. Onwards we went to the upstairs floors. Upstairs, past a nice graphic of Prusa's historic products, we visited the assembly and QE testing areas. They have a very organised system of parts buckets and some thorough QE processes, including some bespoke equipment that produces the "receipt" of tests and calibration that they provide for you in the box when you buy a 3D printer. After that, we visited the production Farm: a large room full of MK3S+ printers churning out parts for other printers. The noise was remarkable. The printers were running custom firmware to continually print the part they had been set for. Some of the printers were designated for printing with ASA: they were colour-coded (yellow controller surround) and within boxed regions to prevent the fumes causing problems. (picture at the top) Outside the room sat palettes of orange PETG plastic for the printer farm on 2KG spools (not a size they sell to the public just yet) The final part of the trip was outside, to the real farm: Prusa have a smallholding with Alpacas at the rear of their estate. Whilst we visited it, Josef Prusa himself turned up (in a snazzy looking custom colour Tesla) to feed the animals, say hello and pose for a picture. Overall, it was a fantastic visit. I'm very grateful to Air France for cancelling my flight home, and to Prusa Labs (in particular Luk ) for allowing me to come and say hello!
  1. I managed to squeeze that in too on my way to my rescheduled flight, although it was a rush visit and I don't have much to say or show from it. Suffice to say that it was lovely and bittersweet since the UK Maker Faire used to be hosted in my fair home city of Newcastle before they stopped.

Vincent Bernat: Building a SQL-like language to filter flows

Akvorado collects network flows using IPFIX or sFlow. It stores them in a ClickHouse database. A web console allows a user to query the data and plot some graphs. A nice aspect of this console is how we can filter flows with a SQL-like language:
Filter editor in Akvorado console
Often, web interfaces expose a query builder to build such filters. I think combining a SQL-like language with an editor supporting completion, syntax highlighting, and linting is a better approach.1 The language parser is built with pigeon (Go) from a parsing expression grammar or PEG. The editor component is CodeMirror (TypeScript).

Language parser PEG grammars are relatively recent2 and are an alternative to context-free grammars. They are easier to write and they can generate better error messages. Python switched from an LL(1)-based parser to a PEG-based parser in Python 3.9. pigeon generates a parser for Go. A grammar is a set of rules. Each rule is an identifier, with an optional user-friendly label for error messages, an expression, and an action in Go to be executed on match. You can find the complete grammar in parser.peg. Here is a simplified rule: 
ConditionIPExpr "condition on IP"  
  column:("ExporterAddress"i   return "ExporterAddress", nil  
        / "SrcAddr"i   return "SrcAddr", nil  
        / "DstAddr"i   return "DstAddr", nil  ) _ 
  operator:("=" / "!=") _ 
  ip:IP  
    return fmt.Sprintf("%s %s IPv6StringToNum(%s)",
      toString(column), toString(operator), quote(ip)), nil
The rule identifier is ConditionIPExpr. It case-insensitively matches ExporterAddress, SrcAddr, or DstAddr. The action for each case returns the proper case for the column name. That s what is stored in the column variable. Then, it matches one of the possible operators. As there is no code block, it stores the matched string directly in the operator variable. Then, it tries to match the IP rule, which is defined elsewhere in the grammar. If it succeeds, it stores the result of the match in the ip variable and executes the final action. The action turns the column, operator, and IP into a proper expression for ClickHouse. For example, if we have ExporterAddress = 203.0.113.15, we get ExporterAddress = IPv6StringToNum('203.0.113.15'). The IP rule uses a rudimentary regular expression but checks if the matched address is correct in the action block, thanks to netip.ParseAddr(): 
IP "IP address"   [0-9A-Fa-f:.]+  
  ip, err := netip.ParseAddr(string(c.text))
  if err != nil  
    return "", errors.New("expecting an IP address")
   
  return ip.String(), nil
Our parser safely turns the filter into a WHERE clause accepted by ClickHouse:3 
WHERE InIfBoundary = 'external' 
AND ExporterRegion = 'france' 
AND InIfConnectivity = 'transit' 
AND SrcAS = 15169 
AND DstAddr BETWEEN toIPv6('2a01:e0f:ffff::') 
                AND toIPv6('2a01:e0f:ffff:ffff:ffff:ffff:ffff:ffff')

Integration in CodeMirror CodeMirror is a versatile code editor that can be easily integrated into JavaScript projects. In Akvorado, the Vue.js component, InputFilter, uses CodeMirror as its foundation and leverages features such as syntax highlighting, linting, and completion. The source code for these capabilities can be found in the codemirror/lang-filter/ directory.

Syntax highlighting The PEG grammar for Go cannot be utilized directly4 and the requirements for parsers for editors are distinct: they should be error-tolerant and operate incrementally, as code is typically updated character by character. CodeMirror offers a solution through its own parser generator, Lezer. We don t need this additional parser to fully understand the filter language. Only the basic structure is needed: column names, comparison and logic operators, quoted and unquoted values. The grammar is therefore quite short and does not need to be updated often: 
@top Filter  
  expression
 
expression  
 Not expression  
 "(" expression ")"  
 "(" expression ")" And expression  
 "(" expression ")" Or expression  
 comparisonExpression And expression  
 comparisonExpression Or expression  
 comparisonExpression
 
comparisonExpression  
 Column Operator Value
 
Value  
  String   Literal   ValueLParen ListOfValues ValueRParen
 
ListOfValues  
  ListOfValues ValueComma (String   Literal)  
  String   Literal
 
// [ ]
@tokens  
  // [ ]
  Column   std.asciiLetter (std.asciiLetter std.digit)*  
  Operator   $[a-zA-Z!=><]+  
  String  
    '"' (![\\\n"]   "\\" _)* '"'?  
    "'" (![\\\n']   "\\" _)* "'"?
   
  Literal   (std.digit   std.asciiLetter   $[.:/])+  
  // [ ]
The expression SrcAS = 12322 AND (DstAS = 1299 OR SrcAS = 29447) is parsed to: 
Filter(Column, Operator, Value(Literal),
  And, Column, Operator, Value(Literal),
  Or, Column, Operator, Value(Literal))
The last step is to teach CodeMirror how to map each token to a highlighting tag: 
export const FilterLanguage = LRLanguage.define( 
  parser: parser.configure( 
    props: [
      styleTags( 
        Column: t.propertyName,
        String: t.string,
        Literal: t.literal,
        LineComment: t.lineComment,
        BlockComment: t.blockComment,
        Or: t.logicOperator,
        And: t.logicOperator,
        Not: t.logicOperator,
        Operator: t.compareOperator,
        "( )": t.paren,
       ),
    ],
   ),
 );

Linting We offload linting to the original parser in Go. The /api/v0/console/filter/validate endpoint accepts a filter and returns a JSON structure with the errors that were found: 
 
  "message": "at line 1, position 12: string literal not terminated",
  "errors": [ 
    "line":    1,
    "column":  12,
    "offset":  11,
    "message": "string literal not terminated",
   ]
The linter source for CodeMirror queries the API and turns each error into a diagnostic.

Completion The completion system takes a hybrid approach. It splits the work between the frontend and the backend to offer useful suggestions for completing filters. The frontend uses the parser built with Lezer to determine the context of the completion: do we complete a column name, an operator, or a value? It also extracts the column name if we are completing something else. It forwards the result to the backend through the /api/v0/console/filter/complete endpoint. Walking the syntax tree was not as easy as I thought, but unit tests helped a lot. The backend uses the parser generated by pigeon to complete a column name or a comparison operator. For values, the completions are either static or extracted from the ClickHouse database. A user can complete an AS number from an organization name thanks to the following snippet: 
results := []struct  
  Label  string  ch:"label" 
  Detail string  ch:"detail" 
 
columnName := "DstAS"
sqlQuery := fmt.Sprintf( 
 SELECT concat('AS', toString(%s)) AS label, dictGet('asns', 'name', %s) AS detail
 FROM flows
 WHERE TimeReceived > date_sub(minute, 1, now())
 AND detail != ''
 AND positionCaseInsensitive(detail, $1) >= 1
 GROUP BY label, detail
 ORDER BY COUNT(*) DESC
 LIMIT 20
 , columnName, columnName)
if err := conn.Select(ctx, &results, sqlQuery, input.Prefix); err != nil  
  c.r.Err(err).Msg("unable to query database")
  break
 
for _, result := range results  
  completions = append(completions, filterCompletion 
    Label:  result.Label,
    Detail: result.Detail,
    Quoted: false,
   )
In my opinion, the completion system is a major factor in making the field editor an efficient way to select flows. While a query builder may have been more beginner-friendly, the completion system s ease of use and functionality make it more enjoyable to use once you become familiar.
  1. Moreover, building a query builder did not seem like a fun task for me.
  2. They were introduced in 2004 in Parsing Expression Grammars: A Recognition-Based Syntactic Foundation. LR parsers were introduced in 1965, LALR parsers in 1969, and LL parsers in the 1970s. Yacc, a popular parser generator, was written in 1975.
  3. The parser returns a string. It does not generate an intermediate AST. This makes it simpler and it currently fits our needs.
  4. It could be manually translated to JavaScript with PEG.js.

Next.

Previous.