Russ Allbery: Review: A Master of Djinn
| Series: | Dead Djinn Universe #1 |
| Publisher: | Tordotcom |
| Copyright: | 2021 |
| ISBN: | 1-250-26767-6 |
| Format: | Kindle |
| Pages: | 391 |
| Series: | Dead Djinn Universe #1 |
| Publisher: | Tordotcom |
| Copyright: | 2021 |
| ISBN: | 1-250-26767-6 |
| Format: | Kindle |
| Pages: | 391 |
Welcome to the 36th post of the really randomly reverberating R, or R4 for short, write-ups. Today s post is about using Redis, and especially RcppRedis, for live or (near) real-time monitoring with R.
There is an saying that you can take the boy out of the valley, but you cannot the valley out of the boy so for those of us who spent a decade or two in finance and on trading floors, having some market price information available becomes second nature. And/or sometimes it is just good fun to program this.
A good while back Josh posted a gist on a simple-yet-robust while loop. It (very cleverly) uses his quantmod package to access the SP500 in real-time . (I use quotes here because at the end of retail broadband one is not at the same market action as someone co-located in a New Jersey data center. It is however not delayed: as an index, it is not immediately tradeable as a stock, etf, or derivative may be all of which are only disseminated as delayed price information, usually by ten minutes.) I quite enjoyed the gist and used it and started tinkering with it. For example, it collects data but only saves (i.e. persists ) it after market close. If for whatever reason one needs to restart recent history is gone. In any event, I used his code and generalized it a little and published this about a year ago as function intradayMarketMonitor() in my dang package. (See this blog post announcing it.) The chart of the left shows this in action, the chart is a snapshot from a couple of days ago when the vignettes (more on them below) were written.
As lovely as intradayMarketMonitor() is, it also limits itself to market hours. And sometimes you want to see, say, how the market opens on Sunday (futures usually restart at 17h Chicago time), or how news dissipates during the night, or where markets are pre-open, or . So I both wanted to complement this with futures, and also cache it locally so that, say, one machine might collect data and one (or several others) can visualize. For such tasks, Redis is unparalleled.
(Yet I also always felt Redis could do with another, simple, short and sweet introduction stressing the key features of i) being multi-lingual: write in one language, consume in another and ii) loose coupling: no linking as one talks to Redis via standard tcp/ip networking. So I wrote a new intro vignette that is now in RcppRedis. I hope this comes in handy. Comments welcome!)
Our RcppRedis package had long been used for such tasks, and it was easy to set it up. Standard use is to loop, fetch some data, push it to Redis, sleep, and start over. Clients do the same: fetch most recent data, plot or report it, sleep, start over. That works, but it has a dual delay as the client sleeping may miss the data update!
The standard answer to this is called publish/pubscribe, or pub/sub. Libraries such as 0mq or zeromq specialise in this. But it turns out Redis already has it. I had some initial difficulty adding it to RcppRedis so for a trial I tested the marvellous rredis package by Bryan and simply instantiated two Redis clients. Now the data getter simply publishes a new data point in a given channel, by convention named after the security it tracks. Clients register with the Redis server which does all the actual work of keeping track of who listens to what. The clients now simply listen (which is a blocking operation) and as soon as data comes in receive it.
This is quite mesmerizing when you just run two command-line clients (in a byobu session, say). As sone as the data is written (as shown on console log) it is consumed. No measruable overhead. Just lovely.
Bryan and I then talked a litte as he may or may not retire rredis. Having implemented the pub/sub logic for both sides once, he took a good hard look at RcppRedis and just like that added it there. With some really clever wrinkles for (optional) per-symbol callback as closure attached to the instance. Truly amazeballs And once we had it in there, generalizing from publishing or subscribing to just one symbol easily generalizes to having one listener collect and publish for multiple symbols, and having one or more clients subscribe and listen one, more or even all symbol. All with ease thanks tp Redis. The second chart, also from a few days ago, shows four symbols for four (front-contract) futures for Bitcoin, Crude Oil, SP500, and Gold.
As all this can get a little technical, I wrote a second vignette for RcppRedis on just this: market monitoring. Give this a read, if interested, feedback on this one is most welcome too! But all the code you need is included in the package just run a local Redis instance.
Before closing, one sour note. I uploaded all this in a new and much improved updated RcppRedis 0.2.0 to CRAN on March 13 ten days ago. Not only is it still not there , but CRAN in their most delightful way also refuses to answer any emails of mine. Just lovely. The package exhibited just one compiler warning: a C++ compiler objected to the (embedded) C library hiredis (included as a fallback) for using a C language construct. Yes. A C++ compiler complaining about C. It s a non-issue. Yet it s been ten days and we still have nothing. So irritating and demotivating. Anyway, you can get the package off its GitHub repo.
If you like this or other open-source work I do, you can sponsor me at GitHub.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
nov 15 16:12:19 angela systemd[2305]: Starting pull emails with syncmaildir...
nov 15 16:12:22 angela systemd[2305]: smd-pull.service: Succeeded.
nov 15 16:12:22 angela systemd[2305]: Finished pull emails with syncmaildir.
nov 15 16:14:08 angela systemd[2305]: Starting pull emails with syncmaildir...
nov 15 16:14:11 angela systemd[2305]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
nov 15 16:14:11 angela systemd[2305]: smd-pull.service: Failed with result 'exit-code'.
nov 15 16:14:11 angela systemd[2305]: Failed to start pull emails with syncmaildir.
nov 15 16:16:14 angela systemd[2305]: Starting pull emails with syncmaildir...
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: Network error.
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: Unable to get any data from the other endpoint.
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: This problem may be transient, please retry.
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: Hint: did you correctly setup the SERVERNAME variable
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: on your client? Did you add an entry for it in your ssh
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: configuration file?
nov 15 16:16:17 angela smd-pull[27178]: smd-client: ERROR: Network error
nov 15 16:16:17 angela smd-pull[27188]: register: smd-client@localhost: TAGS: error::context(handshake) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
nov 15 16:16:17 angela systemd[2305]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
nov 15 16:16:17 angela systemd[2305]: smd-pull.service: Failed with result 'exit-code'.
nov 15 16:16:17 angela systemd[2305]: Failed to start pull emails with syncmaildir.
What is frustrating is that there's actually no network error
here. Running the command by hand I did see a different message, but
now I have lost it in my backlog. It had something to do with a
filename being too long, and I gave up debugging after a while. This
happened suddenly too, which added to the confusion.
In a fit of rage I started this blog post and experimenting with
alternatives, which led me down a lot of rabbit holes.
Reviewing my previous mail crash
documentation, it seems most
solutions involve talking to an IMAP server, so I figured I would just
do that. Wanting to try something new, i gave isync (AKA
mbsync) a try. Oh dear, I did not expect how much trouble just
talking to my IMAP server would be, which wasn't not isync's fault,
for what that's worth. It was the primary tool I used to debug things,
and served me well in that regard.
mbsync would stop on a FETCH command and Dovecot
would give me those errors on the server side.
nov 16 15:31:27 marcos dovecot[3621800]: imap(anarcat)<3630489><wAmSzO3QZtfAqAB1>: Error: Mailbox junk: Maildir filename has wrong W value, renamed the file from /home/anarcat/Maildir/.junk/cur/1454623938.M101164P22216.marcos,S=2495,W=2578:2,S to /home/anarcat/Maildir/.junk/cur/1454623938.M101164P22216.marcos,S=2495:2,S
nov 16 15:31:27 marcos dovecot[3621800]: imap(anarcat)<3630489><wAmSzO3QZtfAqAB1>: Error: Mailbox junk: Deleting corrupted cache record uid=1582: UID 1582: Broken virtual size in mailbox junk: read(/home/anarcat/Maildir/.junk/cur/1454623938.M101164P22216.marcos,S=2495,W=2578:2,S): FETCH BODY[] got too little data: 2540 vs 2578
At least this first error was automatically healed by Dovecot (by
renaming the file without the W= flag). The problem is that the
FETCH command fails and mbsync exits noisily. So you need to
constantly restart mbsync with a silly command like:
while ! mbsync -a; do sleep 1; done
nov 16 13:53:08 marcos dovecot[3520770]: imap(anarcat)<3594402><M5JHb+zQ3NLAqAB1>: Error: Mailbox Sent: UID=19288: read(/home/anarcat/Maildir/.Sent/cur/1224790447.M898726P9811V000000000000FE06I00794FB1_0.marvin,S=2588:2,S) failed: Cached message size larger than expected (2588 > 2482, box=Sent, UID=19288) (read reason=mail stream)
nov 16 13:53:08 marcos dovecot[3520770]: imap(anarcat)<3594402><M5JHb+zQ3NLAqAB1>: Error: Mailbox Sent: Deleting corrupted cache record uid=19288: UID 19288: Broken physical size in mailbox Sent: read(/home/anarcat/Maildir/.Sent/cur/1224790447.M898726P9811V000000000000FE06I00794FB1_0.marvin,S=2588:2,S) failed: Cached message size larger than expected (2588 > 2482, box=Sent, UID=19288)
nov 16 13:53:08 marcos dovecot[3520770]: imap(anarcat)<3594402><M5JHb+zQ3NLAqAB1>: Error: Mailbox Sent: UID=19288: read(/home/anarcat/Maildir/.Sent/cur/1224790447.M898726P9811V000000000000FE06I00794FB1_0.marvin,S=2588:2,S) failed: Cached message size larger than expected (2588 > 2482, box=Sent, UID=19288) (read reason=)
nov 16 13:53:08 marcos dovecot[3520770]: imap-login: Panic: epoll_ctl(del, 7) failed: Bad file descriptor
This second problem is much harder to fix, because dovecot does not
recover automatically. This is Dovecot complaining that the cached
size (the S= field, but also present in Dovecot's metadata files)
doesn't match the file size.
I wonder if at least some of those messages were corrupted in the
OfflineIMAP to syncmaildir migration
because part of that procedure is to run the strip_header script
to remove content from the emails. That could easily have broken
things since the files do not also get renamed.
mbsync was able to sync the entire mail
spool.
And no, rebuilding the index files didn't work. Also tried
doveadm force-resync -u anarcat which didn't do anything.
In the end I also had to do this, because the wrong cache values were
also stored elsewhere.
service dovecot stop ; find -name 'dovecot*' -delete; service dovecot start
This would have totally broken any existing clients, but thankfully
I'm starting from scratch (except maybe webmail, but I'm hoping it
will self-heal as well, assuming it only has a cache and not a full
replica of the mail spool).
mbsync was incomplete as it was missing about
15,000 mails:
anarcat@angela:~(main)$ find Maildir -type f -type f -a \! -name '.*' wc -l
384836
anarcat@angela:~(main)$ find Maildir-mbsync/ -type f -a \! -name '.*' wc -l
369221
As it turns out, mbsync was not at fault here either: this was yet
more mail spool corruption.
It's actually 26 folders (out of 205) with inconsistent sizes, which can
be found with:
for folder in * .[^.]* ; do
printf "%s\t%d\n" $folder $(find "$folder" -type f -a \! -name '.*' wc -l );
done
The special \! -name '.*' bit is to ignore the mbsync metadata,
which creates .uidvalidity and .mbsyncstate in every folder. That
ignores about 200 files but since they are spread around all folders,
which was making it impossible to review where the problem was.
Here is what the diff looks like:
--- Maildir-list 2021-11-17 20:42:36.504246752 -0500
+++ Maildir-mbsync-list 2021-11-17 20:18:07.731806601 -0500
@@ -6,16 +6,15 @@
[...]
.Archives 1
.Archives.2010 3553
-.Archives.2011 3583
-.Archives.2012 12593
+.Archives.2011 3582
+.Archives.2012 620
.Archives.2013 8576
.Archives.2014 11057
-.Archives.2015 8173
+.Archives.2015 8165
.Archives.2016 54
.band 34
.bitbuck 1
@@ -38,13 +37,12 @@
.couchsurfers 2
-cur 11285
+cur 11280
.current 130
.cv 2
.debbug 262
-.debian 37544
-drafts 1
-.Drafts 4
+.debian 37533
+.Drafts 2
.drone 241
.drupal 188
.drupal-devel 303
[...]
Archives
folders. As it turns out, at least 12,000 of those missing mails were
actually misfiled: instead of being in the
Maildir/.Archives.2012/cur/ folder, they were directly in
Maildir/.Archives.2012/. This is something that doesn't matter for
SMD (mbsync...
After moving those files around, we still have 4,000 message missing:
anarcat@angela:~(main)$ find Maildir-mbsync/ -type f -a \! -name '.*' wc -l
381196
anarcat@angela:~(main)$ find Maildir/ -type f -a \! -name '.*' wc -l
385053
The problem is that those 4,000 missing mails are harder to
track. Take, for example, .Archives.2011, which has a single message
missing, out of 3,582. And the files are not identical: the checksums
don't match after going through the IMAP transport, so we can't use a
tool like hashdeep to compare the trees and find why any single
file is missing.
register in my spool, which I am syncing separately (see Securing
registration email for details on that setup). That actually
covers 3,700 of those messages, so I actually have a more modest 300
messages to figure out, after (easily!) configuring mbsync to sync
that folder separately:
@@ -30,9 +33,29 @@ Slave :anarcat-local:
# Exclude everything under the internal [Gmail] folder, except the interesting folders
#Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred" "[Gmail]/All Mail"
# Or include everything
-Patterns *
+#Patterns *
+Patterns * !register !.register
# Automatically create missing mailboxes, both locally and on the server
#Create Both
Create slave
# Sync the movement of messages between folders and deletions, add after making sure the sync works
#Expunge Both
+
+IMAPAccount anarcat-register
+Host imap.anarc.at
+User register
+PassCmd "pass imap.anarc.at-register"
+SSLType IMAPS
+CertificateFile /etc/ssl/certs/ca-certificates.crt
+
+IMAPStore anarcat-register-remote
+Account anarcat-register
+
+MaildirStore anarcat-register-local
+SubFolders Maildir++
+Inbox ~/Maildir-mbsync/.register/
+
+Channel anarcat-register
+Master :anarcat-register-remote:
+Slave :anarcat-register-local:
+Create slave
anarcat@angela:~(main)$ find Maildir-mbsync/ -type f -a \! -name '.*' wc -l
384900
anarcat@angela:~(main)$ find Maildir/ -type f -a \! -name '.*' wc -l
385059
Argh. After more digging, I have found 131 mails in the tmp/
directories of the client's mail spool. Mysterious! On the server
side, it's even more files, and not the same ones. Possible that
those were mails that were left there during a failed delivery of some
sort, during a power failure or some sort of crash? Who knows. It
could be another race condition in SMD if it runs while mail is being
delivered in tmp/...
The first thing to do with those is to cleanup a bunch of empty files
(21 on angela):
find .[^.]*/tmp -type f -empty -delete
As it turns out, they are all duplicates, in the sense that
notmuch can easily find a copy of files with the same message ID in
its database. In other words, this hairy command returns nothing
find .[^.]*/tmp -type f while read path; do
msgid=$(grep -m 1 -i ^message-id "$path" sed 's/Message-ID: //i;s/[<>]//g');
if notmuch count --exclude=false "id:$msgid" grep -q 0; then
echo "$path <$msgid> not in notmuch" ;
fi;
done
... which is good. Or, to put it another way, this is safe:
find .[^.]*/tmp -type f -delete
Poof! 314 mails cleaned on the server side. Interestingly, SMD doesn't
pick up on those changes at all and still sees files in tmp/
directories on the client side, so we need to operate the same twisted
logic there.
anarcat@angela:~(main)$ find Maildir/ -type f -a \! -name '.*' wc -l
384928
anarcat@angela:~(main)$ find Maildir-mbsync/ -type f -a \! -name '.*' wc -l
384901
Ha! 27 mails difference. Those are the really sticky, unclear ones. I
was hoping a full sync might clear that up, but after deleting the
entire directory and starting from scratch, I end up with:
anarcat@angela:~(main)$ find Maildir -type f -type f -a \! -name '.*' wc -l
385034
anarcat@angela:~(main)$ find Maildir-mbsync -type f -type f -a \! -name '.*' wc -l
384993
That is: even more messages missing (now 37). Sigh.
Thankfully, this is something notmuch can help with:
it can index all files by Message-ID (which I learned is
case-insensitive, yay) and tell us which messages don't make it
through.
Considering the corruption I found in the mail spool, I wouldn't be
the least surprised those messages are just skipped by the IMAP
server. Unfortunately, there's nothing on the Dovecot server logs that
would explain the discrepancy.
Here again, notmuch comes to the rescue. We can list all message IDs
to figure out that discrepancy:
notmuch search --exclude=false --output=messages '*' pv -s 18M sort > Maildir-msgids
notmuch --config=.notmuch-config-mbsync search --exclude=false --output=messages '*' pv -s 18M sort > Maildir-mbsync-msgids
And then we can see how many messages notmuch thinks are missing:
$ wc -l *msgids
372723 Maildir-mbsync-msgids
372752 Maildir-msgids
That's 29 messages. Oddly, it doesn't exactly match the find output:
anarcat@angela:~(main)$ find Maildir-mbsync -type f -type f -a \! -name '.*' wc -l
385204
anarcat@angela:~(main)$ find Maildir -type f -type f -a \! -name '.*' wc -l
385241
That is 10 more messages. Ugh. But actually, I know what those are:
more misfiled messages (in a .folder/draft/ directory, bizarrely, so
the totals actually match.
In the notmuch output, there's a lot of stuff like this:
id:notmuch-sha1-fb880d673e24f5dae71b6b4d825d4a0d5d01cde4
Those are messages without a valid Message-ID. Notmuch (presumably)
constructs one based on the file's checksum. Because the files differ
between the IMAP server and the local mail spool (which is
unfortunate, but possibly inevitable), those do not match. There are
exactly the same number of those on both sides, so I'll go ahead and
assume those are all accounted for.
What remains is:
anarcat@angela:~(main)$ diff -u Maildir-mbsync-msgids Maildir-msgids grep '^\-[^-]' grep -v sha1 wc -l
2
anarcat@angela:~(main)$ diff -u Maildir-mbsync-msgids Maildir-msgids grep '^\+[^+]' grep -v sha1 wc -l
21
anarcat@angela:~(main)$
ie. 21 missing from mbsync, and, surprisingly, 2 missing from the
original mail spool.
Further inspection also showed they were all messages with some sort
of "corruption": no body and only headers. I am not sure that is a
legal email format in the first place. Since they were mostly spam or
administrative emails ("You have been unsubscribed from mailing
list..."), it seems fairly harmless to ignore those.
mbsync provide fast and reliable transport,
including over SSH. See the next
article for further discussion of
the alternatives.
| Illustrator: | Pauline Baynes |
| Series: | Chronicles of Narnia #6 |
| Publisher: | Collier Books |
| Copyright: | 1955 |
| Printing: | 1978 |
| ISBN: | 0-02-044230-0 |
| Format: | Mass market |
| Pages: | 186 |
All of the faces they could see were certainly nice. Both the men and women looked kind and wise, and they seemed to come of a handsome race. But after the children had gone a few steps down the room they came to faces that looked a little different. These were very solemn faces. You felt you would have to mind your P's and Q's, if you ever met living people who looked like that. When they had gone a little farther, they found themselves among faces they didn't like: this was about the middle of the room. The faces here looked very strong and proud and happy, but they looked cruel. A little further on, they looked crueller. Further on again, they were still cruel but they no longer looked happy. They were even despairing faces: as if the people they belonged to had done dreadful things and also suffered dreadful things.The last statue is of a fierce, proud woman that Digory finds strikingly beautiful. (Lewis notes in an aside that Polly always said she never found anything specially beautiful about her. Here, as in The Silver Chair, the girl is the sensible one and things would have gone better if the boy had listened to her, a theme that I find immensely frustrating because Susan was the sensible one in the first two books of the series but then Lewis threw that away.) There is a bell in the middle of this hall, and the pillar that holds that bell has an inscription on it that I think every kid who grew up on Narnia knows by heart.
Make your choice, adventurous Stranger;Polly has no intention of striking the bell, but Digory fights her and does it anyway, waking Jadis from where she sat as the final statue in the hall and setting off one of the greatest reimaginings of a villain in children's literature. Jadis will, of course, become the White Witch who holds Narnia in endless winter some thousand Narnian years later. But the White Witch was a mediocre villain at best, the sort of obvious and cruel villain common in short fairy tales where the author isn't interested in doing much characterization. She exists to be evil, do bad things, and be defeated. She has a few good moments in conflict with Aslan, but that's about it. Jadis in this book is another matter entirely: proud, brilliant, dangerous, and creative. The death of everything on Charn was Jadis's doing: an intentional spell, used to claim a victory of sorts from the jaws of defeat by her sister in a civil war. (I find it fascinating that Lewis puts aside his normally sexist roles here.) Despite the best attempts of the kids to lose her both in Charn and in the Wood (which is inimical to her, in another nice bit of world-building), she manages to get back to England with them. The result is a remarkably good bit of villain characterization. Jadis is totally out of her element, used to a world-spanning empire run with magic and (from what hints we get) vaguely medieval technology. Her plan to take over their local country and eventually the world should be absurd and is played somewhat for laughs. Her magic, which is her great weapon, doesn't even work in England. But Jadis learns at a speed that the reader can watch. She's observant, she pays attention to things that don't fit her expectations, she changes plans, and she moves with predatory speed. Within a few hours in London she's stolen jewels and a horse and carriage, and the local police seem entirely overmatched. There's no way that one person without magic should be a real danger to England around the turn of the 20th century, but by the time the kids manage to pull her back into the Wood, you're not entirely sure England would have been safe. A chaotic confrontation, plus the ability of the rings to work their magic through transitive human contact, ends up with the kids, Uncle Andrew, Jadis, a taxicab driver and his horse all transported through the Wood to a new world. In this case, literally a new world: Narnia at the point of its creation. Here again, Lewis translates Christian myth, in this case the Genesis creation story, into a more vivid and in many ways more beautiful story than the original. Aslan singing the world into existence is an incredible image, as is the newly-created world so bursting with life that even things that normally could not grow will do so. (Which, of course, is why there is a lamp post burning in the middle of the western forest of Narnia for the Pevensie kids to find later.) I think my favorite part is the creation of the stars, but the whole sequence is great. There's also an insightful bit of human psychology. Uncle Andrew can't believe that a lion is singing, so he convinces himself that Aslan is not singing, and thus prevents himself from making any sense of the talking animals later.
Strike the bell and bide the danger,
Or wonder, till it drives you mad,
What would have followed if you had.
Now the trouble about trying to make yourself stupider than you really are is that you very often succeed.As with a lot in Lewis, he probably meant this as a statement about faith, but it generalizes well beyond the religious context. What disappointed me about the creation story, though, is the animals. I didn't notice this as a kid, but this re-read has sensitized me to how Lewis consistently treats the talking animals as less than humans even though he celebrates them. That happens here too: the newly-created, newly-awakened animals are curious and excited but kind of dim. Some of this is an attempt to show that they're young and are just starting to learn, but it also seems to be an excuse for Aslan to set up a human king and queen over them instead of teaching them directly how to deal with the threat of Jadis who the children inadvertently introduced into the world. The other thing I dislike about The Magician's Nephew is that the climax is unnecessarily cruel. Once Digory realizes the properties of the newly-created world, he hopes to find a way to use that to heal his mother. Aslan points out that he is responsible for Jadis entering the world and instead sends him on a mission to obtain a fruit that, when planted, will ward Narnia against her for many years. The same fruit would heal his mother, and he has to choose Narnia over her. (It's a fairly explicit parallel to the Garden of Eden, except in this case Digory passes.) Aslan, in the end, gives Digory the fruit of the tree that grows, which is still sufficient to heal his mother, but this sequence made me angry when re-reading it. Aslan knew all along that what Digory is doing will let him heal his mother as well, but hides this from him to make it more of a test. It's cruel and mean; Aslan could have promised to heal Digory's mother and then seen if he would help Narnia without getting anything in return other than atoning for his error, but I suppose that was too transactional for Lewis's theology or something. Meh. But, despite that, the only reason why this is not the best Narnia book is because The Voyage of the Dawn Treader is the only Narnia book that also nails the ending. The Magician's Nephew, up through Charn, Jadis's rampage through London, and the initial creation of Narnia, is fully as good, perhaps better. It sags a bit at the end, partly because it tries to hard to make the Narnian animals humorous and partly because of the unnecessary emotional torture of Digory. But this still holds up as the second-best Narnia book, and one I thoroughly enjoyed re-reading. If anything, Jadis and Charn are even better than I remembered. Followed by the last book of the series, the somewhat notorious The Last Battle. Rating: 9 out of 10
Despite having worked on a number of ARM platforms I ve never actually had an ARM based development box at home. I have a Raspberry Pi B Classic (the original 256MB rev 0002 variant) a coworker gave me some years ago, but it s not what you d choose for a build machine and generally gets used as a self contained TFTP/console server for hooking up to devices under test. Mostly I ve been able to do kernel development with the cross compilers already built as part of Debian, and either use pre-built images or Debian directly when I need userland pieces. At a previous job I had a Marvell MACCHIATObin available to me, which works out as a nice platform - quad core A72 @ 2GHz with 16GB RAM, proper SATA and a PCIe slot. However they re still a bit pricey for a casual home machine. I really like the look of the HoneyComb LX2 - 16 A72 cores, up to 64GB RAM - but it s even more expensive.
So when I saw the existence of the 8GB Raspberry Pi 4 I was interested. Firstly, the Pi 4 is a proper 64 bit device (my existing Pi B is ARMv6 which means it needs to run Raspbian instead of native Debian armhf), capable of running an upstream kernel and unmodified Debian userspace. Secondly the Pi 4 has a USB 3 controller sitting on a PCIe bus rather than just the limited SoC USB 2 controller. It s not SATA, but it s still a fairly decent method of attaching some storage that s faster/more reliable than an SD card. Finally 8GB RAM is starting to get to a decent amount - for a headless build box 4GB is probably generally enough, but I wanted some headroom.
The Pi comes as a bare board, so I needed a case. Ideally I wanted something self contained that could take the Pi, provide a USB/SATA adaptor and take the drive too. I came across the pre-order for the DeskPi Pro, decided it was the sort of thing I was after, and ordered one towards the end of September. It finally arrived at the start of December, at which point I got round to ordering a Pi 4 from CPC.
Total cost ~ 120 for the case + Pi.
I managed to break a USB port on the Desk Pi. It has a pair of forward facing ports, I plugged my wireless keyboard dongle into it and when trying to remove it the solid spacer bit in the socket broke off. I ve never had this happen to me before and I ve been using USB devices for 20 years, so I m putting the blame on a shoddy socket.
The first drive I tried was an old Crucial M500 mSATA device. I have an adaptor that makes it look like a normal 2.5 drive so I used that. Unfortunately it resulted in a boot loop; the Pi would boot its initial firmware, try to talk to the drive and then reboot before even loading Linux. The DeskPi Pro comes with an m2 adaptor and I had a spare m2 drive, so I tried that and it all worked fine. This might just be power issues, but it was an unfortunate experience especially after the USB port had broken off.
(Given I ended up using an M.2 drive another case option would have been the Argon ONE M.2, which is a bit more compact.)
The case is a little snug; I was worried I was going to damage things as I slid it in. Additionally the construction process is a little involved. There s a good set of instructions, but there are a lot of pieces and screws involved. This includes a couple of FFC cables to join things up. I think this is because they ve attempted to make a compact case rather than allowing a little extra room, and it does have the advantage that once assembled it feels robust without anything loose in it.
I hate the need for an external USB3 dongle to bridge from the Pi to the USB/SATA adaptor. All the cases I ve seen with an internal drive bay have to do this, because the USB3 isn t brought out internally by the Pi, but it just looks ugly to me. It s hidden at the back, but meh.
Fan control is via a USB/serial device, which is fine, but it attaches to the USB C power port which defaults to being a USB peripheral. Raspbian based kernels support device tree overlays which allows easy reconfiguration to host mode, but for a Debian based system I ended up rolling my own dtb file. I changed
#include "bcm283x-rpi-usb-peripheral.dtsi"
#include "bcm283x-rpi-usb-host.dtsi"
arch/arm/boot/dts/bcm2711-rpi-4-b.dts and then I did:
cpp -nostdinc -I include -I arch -undef -x assembler-with-cpp \
arch/arm/boot/dts/bcm2711-rpi-4-b.dts > rpi4.preprocessed
dtc -I dts -O dtb rpi4.preprocessed -o bcm2711-rpi-4-b.dtb
bcm2711-rpi-4-b.dtb file replaced the one in /boot/firmware. This isn t a necessary step if you don t want to use the cooling fan in the case, or the front USB ports, and it s not really anyone s fault, but it was an annoying extra step to have to figure out.
The DeskPi came with a microSD card that was supposed to have RaspiOS already on it. It didn t, it was blank. In my case that was fine, because I wanted to use Debian, but it was a minor niggle.
apt upgrade and got updated to the Buster 10.7 release, as well as the latest 5.9 backport kernel, and everything came back without effort after a reboot. It s lovely to be able to run Debian on this device without having to futz around with self-compiled kernels.
The DeskPi makes a lot of effort to route things externally. The SD slot is brought out to the front, making it easy to fiddle with the card contents without having to open the case to replace it. All the important ports are brought out to the back either through orientation of the Pi, or extenders in the case. That means the built in Pi USB ports, the HDMI sockets (conveniently converted to full size internally), an audio jack and a USB-C power port. The aforementioned USB3 dongle for the bridge to the drive is the only external thing that s annoying.
Thermally things seem good too. I haven t done a full torture test yet, but with the fan off the system is sitting at about 40 C while fairly idle. Some loops in bash that push load up to above 2 get the temperature up to 46 C or so, and turning the fan on brings it down to 40 C again. It s audible, but quieter than my laptop and not annoying.
I liked the way the case came with everything I needed other than the Pi 4 and a suitable disk drive. There was an included PSU (a proper USB-C PD device, UK plug), the heatsink/fan is there, the USB/SATA converter is there and even an SD card is provided (though that s just because I had a pre-order).
Speaking of the SD, I only needed it for initial setup. Recent Pi 4 bootloaders are capable of booting directly from USB mass storage devices. So I upgraded using the RPi EEPROM Recovery image (which just needs extracted to the SD FAT partition, no need for anything complicated - boot with it and the screen goes all green and you know it s ok), then created a FAT partition at the start of the drive for the kernel / bootloader config and a regular EXT4 partition for root. Copies everything over, updated paths, took out the SD and it all just works happily.
Although I still read a lot, during my college sophomore years my reading
habits shifted from novels to more academic works. Indeed, reading dry
textbooks and economic papers for classes often kept me from reading anything
else substantial. Nowadays, I tend to binge read novels: I won't touch a book
for months on end, and suddenly, I'll read 10 novels back to back1.
At the start of a novel binge, I always follow the same ritual: I take out my
e-reader from its storage box, marvel at the fact the battery is still pretty
full, turn on the WiFi and check if there are OS updates. And I have to admit,
Kobo Inc. (now Rakuten Kobo) has done a stellar job of keeping my e-reader up
to date. I've owned this model (a Kobo Aura 1st generation) for 7
years now and I'm still running the latest version of Kobo's Linux-based OS.
Having recently had trouble updating my Nexus 5 (also manufactured 7 years ago)
to Android 102, I asked myself:
Why is my e-reader still getting regular OS updates, while Google stopped issuing security patches for my smartphone four years ago?To try to answer this, let us turn to economic incentives theory. Although not the be-all and end-all some think it is3, incentives theory is not a bad tool to analyse this particular problem. Executives at Google most likely followed a very business-centric logic when they decided to drop support for the Nexus 5. Likewise, Rakuten Kobo's decision to continue updating older devices certainly had very little to do with ethics or loyalty to their user base. So, what are the incentives that keep Kobo updating devices and why are they different than smartphone manufacturers'? A portrait of the current long-term software support offerings for smartphones and e-readers Before delving deeper in economic theory, let's talk data. I'll be focusing on 2 brands of e-readers, Amazon's Kindle and Rakuten's Kobo. Although the e-reader market is highly segmented and differs a lot based on geography, Amazon was in 2015 the clear worldwide leader with 53% of the worldwide e-reader sales, followed by Rakuten Kobo at 13%4. On the smartphone side, I'll be differentiating between Apple's iPhones and Android devices, taking Google as the barometer for that ecosystem. As mentioned below, Google is sadly the leader in long-term Android software support. Rakuten Kobo According to their website and to this Wikipedia table, the only e-readers Kobo has deprecated are the original Kobo eReader and the Kobo WiFi N289, both released in 2010. This makes their oldest still supported device the Kobo Touch, released in 2011. In my book, that's a pretty good track record. Long-term software support does not seem to be advertised or to be a clear selling point in their marketing. Amazon According to their website, Amazon has dropped support for all 8 devices produced before the Kindle Paperwhite 2nd generation, first sold in 2013. To put things in perspective, the first Kindle came out in 2007, 3 years before Kobo started selling devices. Like Rakuten Kobo, Amazon does not make promises of long-term software support as part of their marketing. Apple Apple has a very clear software support policy for all their devices:
Owners of iPhone, iPad, iPod or Mac products may obtain a service and parts from Apple or Apple service providers for five years after the product is no longer sold or longer, where required by law.This means in the worst-case scenario of buying an iPhone model just as it is discontinued, one would get a minimum of 5 years of software support. Android Google's policy for their Android devices is to provide software support for 3 years after the launch date. If you buy a Pixel device just before the new one launches, you could theoretically only get 2 years of support. In 2018, Google decided OEMs would have to provide security updates for at least 2 years after launch, threatening not to license Google Apps and the Play Store if they didn't comply. A question of cost structure From the previous section, we can conclude that in general, e-readers seem to be supported longer than smartphones, and that Apple does a better job than Android OEMs, providing support for about twice as long. Even Fairphone, who's entire business is to build phones designed to last and to be repaired was not able to keep the Fairphone 1 (2013) updated for more than a couple years and seems to be struggling to keep the Fairphone 2 (2015) running an up to date version of Android. Anyone who has ever worked in IT will tell you: maintaining software over time is hard work and hard work by specialised workers is expensive. Most commercial electronic devices are sold and developed by for-profit enterprises and software support all comes down to a question of cost structure. If companies like Google or Fairphone are to be expected to provide long-term support for the devices they manufacture, they have to be able to fund their work somehow. In a perfect world, people would be paying for the cost of said long-term support, as it would likely be cheaper then buying new devices every few years and would certainly be better for the planet. Problem is, manufacturers aren't making them pay for it. Economists call this type of problem externalities: things that should be part of the cost of a good, but aren't for one a reason or another. A classic example of an externality is pollution. Clearly pollution is bad and leads to horrendous consequences, like climate change. Sane people agree we should drastically cut our greenhouse gas emissions, and yet, we aren't. Neo-classical economic theory argues the way to fix externalities like pollution is to internalise these costs, in other words, to make people pay for the "real price" of the goods they buy. In the case of climate change and pollution, neo-classical economic theory is plain wrong (spoiler alert: it often is), but this is where band-aids like the carbon tax comes from. Still, coming back to long-term software support, let's see what would happen if we were to try to internalise software maintenance costs. We can do this multiple ways. 1 - Include the price of software maintenance in the cost of the device This is the choice Fairphone makes. This might somewhat work out for them since they are a very small company, but it cannot scale for the following reasons:
After we gave our talk at DebConf 20, Doing things together,
there were 5 minutes left for the live Q&A. Pollo asked a question that I
think is interesting and deserves a longer answer: How can we still have a
good code review process without making it a "you need to be perfect"
scenario? I often find picky code reviews help me write better code.
I find it useful to first disentangle
what code reviews are good for, how we do them, why we do them that way, and
how we can potentially improve processes.
What are code reviews good for?
Code review and peer review are great methods for cooperation aiming at:
the program code would be doing this:int socket = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);
According to the ;login: article, converting a C program to use TLS would normally modify only 5-10 lines in the code, which is amazing when compared to using for example the OpenSSL API. The project has set up the https://securesocketapi.org/ web site to spread the idea, and the code for a kernel module and the associated system daemon is available from two github repositories: ssa and ssa-daemon. Unfortunately there is no explicit license information with the code, so its copyright status is unclear. A request to solve this about it has been unsolved since 2018-08-17. I love the idea of extending socket() to gain TLS support, and understand why it is an advantage to implement this as a kernel module and system wide service daemon, but can not help to think that it would be a lot easier to get projects to move to this way of setting up TLS if it was done with a user space approach where programs wanting to use this API approach could just link with a wrapper library. I recommend you check out this simple and powerful approach to more secure network connections. :) As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.int socket = socket(PF_INET, SOCK_STREAM, IPPROTO_TLS);
| Publisher: | Redhook |
| Copyright: | September 2019 |
| ISBN: | 0-316-42198-7 |
| Format: | Kindle |
| Pages: | 373 |
Once there was a brave and temeraryous (sp?) girl who found a Door. It was a magic Door that's why it has a capital D. She opened the Door.The Door led to a bluff over the sea and above a city, a place very far from Kentucky, and she almost stayed, but she came back through the Door when her guardian, Mr. Locke, called. The adventure cost her a diary, several lectures, days of being locked in her room, and the remnants of her strained relationship with her father. When she went back, the frame of the Door was burned to the ground. That was the end of Doors for January for some time, and the continuation of a difficult childhood. She was cared for by her father's employer as a sort of exotic pet, dutifully attempting to obey, grateful for Mr. Locke's protection, and convinced that he was occasionally sneaking her presents through a box in the Pharaoh Room out of some hidden kindness. Her father appeared rarely, said little, and refused to take her with him. Three things helped: the grocery boy who smuggled her stories, an intimidating black woman sent by her father to replace her nurse, and her dog.
Once upon a time there was a good girl who met a bad dog, and they became the very best of friends. She and her dog were inseparable from that day forward.I will give you a minor spoiler that I would have preferred to have had, since it would have saved me some unwarranted worry and some mental yelling at the author: The above story strains but holds. January's adventure truly starts the day before her seventeenth birthday, when she finds a book titled The Ten Thousand Doors in the box in the Pharaoh Room. As you may have guessed from the title, The Ten Thousand Doors of January is a portal fantasy, but it's the sort of portal fantasy that is more concerned with the portal itself than the world on the other side of it. (Hello to all of you out there who, like me, have vivid memories of the Wood between the Worlds.) It's a book about traveling and restlessness and the possibility of escape, about the ability to return home again, and about the sort of people who want to close those doors because the possibility of change created by people moving around freely threatens the world they have carefully constructed. Structurally, the central part of the book is told by interleaving chapters of January's tale with chapters from The Ten Thousand Doors. That book within a book starts with the framing of a scholarly treatment but quickly becomes a biography of a woman: Adelaide Lee Larson, a half-wild farm girl who met her true love at the threshold of a Door and then spent much of her life looking for him. I am not a very observant reader for plot details, particularly for books that I'm enjoying. I read books primarily for the emotional beats and the story structure, and often miss rather obvious story clues. (I'm hopeless at guessing the outcomes of mysteries.) Therefore, when I say that there are many things January is unaware of that are obvious to the reader, that's saying a lot. Even more clues were apparent when I skimmed the first chapter again, and a more observant reader would probably have seen them on the first read. Right down to Mr. Locke's name, Harrow is not very subtle about the moral shape of this world. That can make the early chapters of the book frustrating. January is being emotionally (and later physically) abused by the people who have power in her life, but she's very deeply trapped by false loyalty and lack of external context. Winning free of that is much of the story of the book, and at times it has the unpleasantness of watching someone make excuses for her abuser. At other times it has the unpleasantness of watching someone be abused. But this is the place where I thought the nested story structure worked marvelously. January escapes into the story of The Ten Thousand Doors at the worst moments of her life, and the reader escapes with her. Harrow uses the desire to switch scenes back to the more adventurous and positive story to construct and reinforce the emotional structure of the book. For me, it worked extremely well. It helps that the ending is glorious. The payoff is worth all the discomfort and tension-building in the first half of the book. Both The Ten Thousand Doors and the surrounding narrative reach deeply satisfying conclusions, ones that are entangled but separate in just the ways that they need to be. January's abilities, actions, and decisions at the end of the book were just the outcome that I needed but didn't entirely guess in advance. I could barely put down the last quarter of this story and loved every moment of the conclusion. This is the sort of book that can be hard to describe in a review because its merits don't rest on an original twist or easily-summarized idea. The elements here are all elements found in other books: portal fantasy, the importance of story-telling, coming of age, found family, irrepressible and indomitable characters, and the battle of the primal freedom of travel and discovery and belief against the structural forces that keep rulers in place. The merits of this book are in the small details: the way that January's stories are sparse and rare and sometimes breathtaking, the handling of tattoos, the construction of other worlds with a few deft strokes, and the way Harrow embraces the emotional divergence between January's life and Adelaide's to help the reader synchronize the emotional structure of their reading experience with January's.
She writes a door of blood and silver. The door opens just for her.The Ten Thousand Doors of January is up against a very strong slate for both the Nebula and the Hugo this year, and I suspect it may be edged out by other books, although I wouldn't be unhappy if it won. (It probably has a better shot at the Nebula than the Hugo.) But I will be stunned if Harrow doesn't walk away with the Mythopoeic Award. This seems like exactly the type of book that award was created for. This is an excellent book, one of the best I've read so far this year. Highly recommended. Rating: 9 out of 10
Sealioning (also spelled sea-lioning and sea lioning) is a type of trolling or harassment which consists of pursuing people with persistent requests for evidence or repeated questions, while maintaining a pretense of civility and sincerity.[1][2][3][4] It may take the form of "incessant, bad-faith invitations to engage in debate".[5]
Tone policing (also tone trolling, tone argument, and tone fallacy) is an ad hominem (personal attack) and antidebate tactic based on criticizing a person for expressing emotion. Tone policing detracts from the validity of a statement by attacking the tone in which it was presented rather than the message itself.
"A brawler who tattoos a message onto his knuckles does not throw every punch with the weight of First Amendment protection behind him," the brief stated. "Conduct like this does not constitute speech, nor should it. A deliberate attempt to cause physical injury to someone does not come close to the expression which the First Amendment is designed to protect."
It s no secret that times are changing. It used to be that men were men, jokes were jokes, and all facts came from one white guy in a suit who you trusted because he looked like your dad. Now I know I could get in a lot of trouble for just saying this, but I don t care because someone has to tell the truth: These days, you can t say anything racist at all without being called a racist.
Russia's neighbor has developed a plan for countering misinformation. Can it be exported to the rest of the world?
So, I wrote my BIFF Response but Marvin wrote me another angry email. Actually, he wrote 6 more this week, so what s up with that? Why didn t he stop after my first email?
The BIFF Response Method will teach you how to respond to angry emails, texts, or social media posts while maintaining your dignity and personal power.
My monthly report covers a large part of what I have been doing in the free software world. I write it for my donors (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me.
Debian LTS
This month I was allocated 12h but I only spent 10.5h. During this time, I continued my work on exiv2. I finished reproducing all the issues and then went on doing code reviews to confirm that vulnerabilities were not present when the issue was not reproducible. I found two CVE where the vulnerability was present in the wheezy version and I posted patches in the upstream bug tracker: #57 and #55.
Then another batch of 10 CVE appeared and I started the process over I m currently trying to reproduce the issues.
While doing all this work on exiv2, I also uncovered a failure to build on the package in experimental (reported here).
Misc Debian/Kali work
Debian Live. I merged 3 live-build patches prepared by Matthijs Kooijman and added an armel fix to cope with the the rename of the orion5x image into the marvell one. I also uploaded a new live-config to fix a bug with the keyboard configuration. Finally, I also released a new live-installer udeb to cope with a recent live-build change that broke the locale selection during the installation process.
Debian Installer. I prepared a few patches on pkgsel to merge a few features that had been added to Ubuntu, most notably the possibility to enable unattended-upgrades by default.
More bug reports. I investigated much further my problem with non-booting qemu images when they are built by vmdebootstrap in a chroot managed by schroot (cf #872999) and while we have much more data, it s not yet clear why it doesn t work. But we have a working work-around
While investigating issues seen in Kali, I opened a bunch of reports on the Debian side:
No comment Liked this article? Click here. My blog is Flattr-enabled.
| Publisher: | 47North |
| Copyright: | 2015 |
| ISBN: | 1-5039-4975-3 |
| Format: | Kindle |
| Pages: | 225 |
The problem with crying in the woods, by the side of a white road that leads somewhere terrible, is that the reason for crying isn't inside your head. You have a perfectly legitimate and pressing reason for crying, and it will still be there in five minutes, except that your throat will be raw and your eyes will itch and absolutely nothing else will have changed.Lord Crevan, when Rhea finally reaches him, toys with her by giving her progressively more horrible puzzle tasks, threatening her with the promised marriage if she fails at any of them. The way this part of the book finally resolves is one of the best moments I've read in any book. Kingfisher captures an aspect of moral decisions, and a way in which evil doesn't work the way that evil people expect it to work, that I can't remember seeing an author capture this well. There are a lot of things here for Rhea to untangle: the nature of Crevan's power, her unexpected allies in his manor, why he proposed marriage to her, and of course how to escape his power. The plot works, but I don't think it was the best part of the book, and it tends to happen to Rhea rather than being driven by her. But I have rarely read a book quite this confident of its moral center, or quite as justified in that confidence. I am definitely reading everything Vernon has published under the T. Kingfisher name, and quite possibly most of her children's books as well. Recommended, particularly if you liked the excerpt above. There's an entire book full of paragraphs like that waiting for you. Rating: 8 out of 10
iproute2,
ethtool, and iptables are generally consistent across distributions,
even though each distribution has its own way of using them.
Yet Linux is not dominant, why? Mukherjee identified the problem as
"packaging issues" and listed a set of features he would like Linux to
improve:
ethtool interface. The idea is to make
ethtool a de-facto standard to manage switches and ports.
Mukherjee gave the example that data centers spend more money on
cables than any other hardware and explained that making it easier
to identify cables is therefore a key functionality. Getting
consistent interface naming was also a key problem identified by
numerous participants at the conference. While systemd tried to fix
this with the predictable network interface
names
specification, the interface names are not necessarily predictable
across virtual machines or on special hardware; in fact, this was
the topic of the first talk of the
conference.
ethtool also needs to support interfaces that run faster than
1 Gbps, something that still has limited support in Linux at
the moment.switchdev_ops, which we previously
covered, ethtool_ops, and netdev_ops. Certain
switches, however, also need distributed switch architecture
(DSA)
features to be properly handled. DSA is a more obscure part of the
network stack that allows Linux to represent hardware switches or chains
of switches using regular Linux tools like bridge, ifconfig, and so
on. While switchdev is a new layer, DSA has been in the kernel since
2.6.28 in 2008. Originally developed to support
Marvell switches, DSA is now a generic layer deployed in WiFi access
points, set-top boxes, on-board flight entertainment systems, trains,
and other industrial equipment. Switches that have an Ethernet
controller need DSA, whereas the kernel can support switches without
Ethernet controllers directly with switchdev drivers.
The first years of DSA's development consisted only of basic maintenance
but, in the last three years, DSA has seen a resurgence of
contributions, as part of Linux networking push to support hardware
offloading and network switches. Between 2014 and 2015, DSA added
support for Broadcom hardware, wake on LAN, and hardware port bridging,
among other features.
DSA's development was parallel to
swconfig, written by
the OpenWrt project to support the small office
and home office (SOHO) routers that the project is targeting. The main
difference between swconfig and DSA is that DSA-supported switches show
one network interface per port, whereas swconfig-configured switches
show up as a single port, which limits the amount of information that
can be extracted from the switch. For example, you cannot have per-port
traffic statistics with swconfig. That limitation is what led to the
creation of the switchdev framework, when swconfig was
proposed (then refused) for inclusion in mainline.
Another goal of switchdev was to support bridge hardware offloading and
network interface card (NIC) virtualization.
Also, whereas swconfig uses virtual LAN
(VLAN) tagging to address
ports, DSA enables the use of device-specific tagging headers to address
different ports, which enables DSA to have better control over the
switches. This allows, for example, DSA to do internet group management
protocol
(IGMP)
snooping or implement the spanning tree
protocol, whereas
swconfig doesn't have those features. Some switches are actually
connected to the host CPU through an Ethernet interface instead of
regular PCI-Express interface, and DSA supports this as well.
One advantage that remains in the swconfig approach is that it treats
the internal switch as a simple external switch, and addresses ports
with standard VLAN tags. This is something DSA could do, as well, but no
one has bothered implementing this just yet. For now, DSA drivers use
device-specific tagging mechanisms that limit the number of supported
devices. Other areas of future improvement for DSA are better multi-chip
support, IGMP snooping, and bonding, as well as firewall, NAT, and TC
offloading.
The author would like to thank the Netdev organizers for travel assistance. Also, thanks to Andrew Lunn for a technical review of this article. Note: this article first appeared in the Linux Weekly News.
Today marks the 13th anniversary since the last passenger flight from New York arrived in the UK. Every seat was filled, a feat that had become increasingly rare for a plane that was a technological marvel but a commercial flop .
Back in August, I was looking for a
Homeserver replacement. During FrOSCon I was then reminded
of the Turris Omnia project by NIC.cz. The basic SoC (Marvel
Armada 38x) seemed to be nice hand have decent mainline support (and,
with the turris, users interested in keeping it working). Only I don't
want any WIFI and I wasn't sure the standard case would be all that
usefully. Fortunately, there's also a simple board available with the
same SoC called ClearFog and so I got one of these (the Base
version). With shipping and the SSD (the only 2242 M.2 SSD with 250
GiB I could find, a ADATA SP600) it slightly exceeds the budget but
well.
When installing the machine, the obvious goal was to use mainline FOSS
components only if possible. Fortunately there's mainline kernel
support for the device as well as mainline U-Boot. First attempts
to boot from a micro SD card did not work out at all, both with
mainline U-Boot and the vendor version though. Turns out the eMMC
version of the board does not support any micro SD cards at all, a
fact that is documented but others failed to notice as well.
U-Boot
As the board does not come with any loader on eMMC and booting
directly from M.2 requires removing some resistors from the board, the
easiest way is using UART for booting. The vendor wiki has some shell
script wrapping an included C fragment to feed U-Boot to the device
but all that is really needed is U-Boot's kwboot utility. For some
reason the SPL didn't properly detect UART booting on my device (wrong
magic number) but patching the if (in arch-mvebu's spl.c) and always
assume UART boot is an easy way around.
The plan then was to boot a Debian armhf rootfs with a
defconfig kernel from USB stick. and install U-Boot and the rootfs
to eMMC from within that system. Unfortunately U-Boot seems to be
unable to talk to the USB3 port so no kernel loading from there. One
could probably make UART loading work but switching between screen for
serial console and xmodem seemed somewhat fragile and I never got it
working. However ethernet can be made to work, though you need to set
eth1addr to eth3addr (or just the right one of these) in U-Boot,
saveenv and reboot. After that TFTP works (but is somewhat slow).
eMMC
There's one last step required to allow U-Boot and Linux to access the
eMMC. eMMC is wired to the same PINs as the SD card would be. However
the SD card has an additional indicator pin showing whether a card is
present. You might be lucky inserting a dummy card into the slot or go
the clean route and remove the pin specification from the device tree.
--- a/arch/arm/dts/armada-388-clearfog.dts
+++ b/arch/arm/dts/armada-388-clearfog.dts
@@ -306,7 +307,6 @@
sdhci@d8000
bus-width = <4>;
- cd-gpios = <&gpio0 20 GPIO_ACTIVE_LOW>;
no-1-8-v;
pinctrl-0 = <&clearfog_sdhci_pins
&clearfog_sdhci_cd_pins>;
Next Up is flashing the U-Boot to eMMC. This seems to
work with the vendor U-Boot but proves to be tricky with
mainline. The fun part boils down to the fact that the boot firmware
reads the first block from eMMC, but the second from SD card. If
you write the mainline U-Boot, which was written and tested for SD
card, to eMMC the SPL will try to load the main U-Boot starting from
it's second sector from flash -- obviously resulting in garbage. This
one took me several tries to figure out and made me read most of the
SPL code for the device. The fix however is trivial (apart from the
question on how to support all different variants from one codebase,
which I'll leave to the U-Boot developers):
--- a/include/configs/clearfog.h
+++ b/include/configs/clearfog.h
@@ -143,8 +143,7 @@
#define CONFIG_SPL_LIBDISK_SUPPORT
#define CONFIG_SYS_MMC_U_BOOT_OFFS (160 << 10)
#define CONFIG_SYS_U_BOOT_OFFS CONFIG_SYS_MMC_U_BOOT_OFFS
-#define CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR ((CONFIG_SYS_U_BOOT_OFFS / 512)\
- + 1)
+#define CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR (CONFIG_SYS_U_BOOT_OFFS / 512)
#define CONFIG_SYS_U_BOOT_MAX_SIZE_SECTORS ((512 << 10) / 512) /* 512KiB */
#ifdef CONFIG_SPL_BUILD
#define CONFIG_FIXED_SDHCI_ALIGNED_BUFFER 0x00180000 /* in SDRAM */
Linux
Now we have a System booting from eMMC with mainline U-Boot (which is
a most welcome speedup compared to the UART and TFTP combination from
the beginning). Getting to fine-tune linux on the device -- we want to
install the armmp Debian kernel and have it work. As all the drivers
are build as modules for that kernel this also means initrd
support. Funnily U-Boots bootz allows booting a plain vmlinux
kernel but I couldn't get it to boot a plain initrd. Passing a uImage
initrd and a normal kernel however works pretty well. Back when I
first tried there were some modules missing and ethernet didn't work
with the PHY driver built as a module. In the meantime the PHY problem
was fixed in the Debian kernel and almost all modules already
added. Ben then only added the USB3 module on my suggestion and as a
result, unstable's armhf armmp kernel should work perfectly well on
the device (you still need to patch the device tree similar to the
patch above). Still missing is an updated flash-kernel to
automatically generate the initrd uImage which is
work in progress but got stalled until I fixed the U-Boot on eMMC
problem and everything should be fine -- maybe get debian u-boot
builds for that board.
Pro versus Base
The main difference so far between the Pro and the Base version of the
ClearFog is the switch chip which is included on the Pro. The Base
instead "just" has two gigabit ethernet ports and a SFP. Both, linux'
and U-Boot's device tree are intended for the Pro version which makes
on of the ethernet ports unusable (it tries to find the switch behind
the ethernet port which isn't there). To get both ports working (or
the one you settled on earlier) there's a second patch to the device
tree (my version might be sub-optimal but works), U-Boot -- the
linux-kernel version is a trivial adaption:
--- a/arch/arm/dts/armada-388-clearfog.dts
+++ b/arch/arm/dts/armada-388-clearfog.dts
@@ -89,13 +89,10 @@
internal-regs
ethernet@30000
mac-address = [00 50 43 02 02 02];
+ managed = "in-band-status";
+ phy = <&phy1>;
phy-mode = "sgmii";
status = "okay";
-
- fixed-link
- speed = <1000>;
- full-duplex;
- ;
;
ethernet@34000
@@ -227,6 +224,10 @@
pinctrl-0 = <&mdio_pins>;
pinctrl-names = "default";
+ phy1: ethernet-phy@1 /* Marvell 88E1512 */
+ reg = <1>;
+ ;
+
phy_dedicated: ethernet-phy@0
/*
* Annoyingly, the marvell phy driver
@@ -386,62 +386,6 @@
tx-fault-gpio = <&expander0 13 GPIO_ACTIVE_HIGH>;
;
- dsa@0
- compatible = "marvell,dsa";
- dsa,ethernet = <ð1>;
- dsa,mii-bus = <&mdio>;
- pinctrl-0 = <&clearfog_dsa0_clk_pins &clearfog_dsa0_pins>;
- pinctrl-names = "default";
- #address-cells = <2>;
- #size-cells = <0>;
-
- switch@0
- #address-cells = <1>;
- #size-cells = <0>;
- reg = <4 0>;
-
- port@0
- reg = <0>;
- label = "lan1";
- ;
-
- port@1
- reg = <1>;
- label = "lan2";
- ;
-
- port@2
- reg = <2>;
- label = "lan3";
- ;
-
- port@3
- reg = <3>;
- label = "lan4";
- ;
-
- port@4
- reg = <4>;
- label = "lan5";
- ;
-
- port@5
- reg = <5>;
- label = "cpu";
- ;
-
- port@6
- /* 88E1512 external phy */
- reg = <6>;
- label = "lan6";
- fixed-link
- speed = <1000>;
- full-duplex;
- ;
- ;
- ;
- ;
-
gpio-keys
compatible = "gpio-keys";
pinctrl-0 = <&rear_button_pins>;
Conclusion
Apart from the mess with eMMC this seems to be a pretty nice
device. It's now happily running with a M.2 SSD providing enough
storage for now and still has a mSATA/mPCIe plug left for future
journeys. It seems to be drawing around 5.5 Watts with SSD and one
Ethernet connected while mostly idle and can feed around 500 Mb/s from
disk over an encrypted ethernet connection which is, I guess, not too
bad. My plans now include helping to finish flash-kernel support,
creating a nice case and probably get it deployed. I might bring
it to FOSDEM first though.
Working on it was really quite some fun (apart from the frustrating
parts finding the one-block-offset ..) and people were really
helpful. Big thanks here to Debian's arm folks, Ben Hutchings the
kernel maintainer and U-Boot upstream (especially Tom Rini and Stefan
Roese)
The majority of NAS devices supported in Debian are based on Marvell's Kirkwood platform. This platform is quite dated now and can only run Debian's armel port.
Debian now supports the Seagate Personal Cloud and Seagate NAS devices. They are based on Marvell's Armada 370, a platform which can run Debian's armhf port. Unfortunately, even the Armada 370 is a bit dated now, so I would not recommend these devices for new purchases. If you have one already, however, you now have the option to run native Debian.
There are some features I like about the Seagate NAS devices:
Adults worry a lot these days. Especially, they worry about how to make other people learn more about computers. They want to make us all computer-literate. Literacy means both reading and writing, but most books and courses about computers only tell you about writing programs. Worse, they only tell about commands and instructions and programming-language grammar rules. They hardly ever give examples. But real languages are more than words and grammar rules. There s also literature what people use the language for. No one ever learns a language from being told its grammar rules. We always start with stories about things that interest us.In a new paper titled Remixing as a pathway to Computational Thinking that was recently published at the ACM Conference on Computer Supported Collaborative Work and Social Computing (CSCW) conference, we used a series of quantitative measures of online behavior to try to uncover evidence that might support the theory that remixing in Scratch is positively associated with learning.
Of course, because Scratch is an informal environment with no set path for users, no lesson plan, and no quizzes, measuring learning is an open problem. In our study, we built on two different approaches to measure learning in Scratch. The first approach considers the number of distinct types of programming blocks available in Scratch that a user has used over her lifetime in Scratch (there are 120 in total) something that can be thought of as a block repertoire or vocabulary. This measure has been used to model informal learning in Scratch in an earlier study. Using this approach, we hypothesized that users who remix more will have a faster rate of growth for their code vocabulary.
Controlling for a number of factors (e.g. age of user, the general level of activity) we found evidence of a small, but positive relationship between the number of remixes a user has shared and her block vocabulary as measured by the unique blocks she used in her non-remix projects. Intriguingly, we also found a strong association between the number of downloads by a user and her vocabulary growth. One interpretation is that this learning might also be associated with less active forms of appropriation, like the process of reading source code described by Minksy.
The second approach we used considered specific concepts in programming, such as loops, or event-handling. To measure this, we utilized a mapping of Scratch blocks to key programming concepts found in this paper by Karen Brennan and Mitchel Resnick. For example, in the image below are all the Scratch blocks mapped to the concept of loop .
We looked at six concepts in total (conditionals, data, events, loops, operators, and parallelism). In each case, we hypothesized that if someone has had never used a given concept before, they would be more likely to use that concept after encountering it while remixing an existing project.
Using this second approach, we found that users who had never used a concept were more likely to do so if they had been exposed to the concept through remixing. Although some concepts were more widely used than others, we found a positive relationship between concept use and exposure through remixing for each of the six concepts. We found that this relationship was true even if we ignored obvious examples of cutting and pasting of blocks of code. In all of these models, we found what we believe is evidence of learning through remixing.
Of course, there are many limitations in this work. What we found are all positive correlations we do not know if these relationships are causal. Moreover, our measures do not really tell us whether someone has understood the usage of a given block or programming concept.However, even with these limitations, we are excited by the results of our work, and we plan to build on what we have. Our next steps include developing and utilizing better measures of learning, as well as looking at other methods of appropriation like viewing the source code of a project.
This blog post and the paper it describes are collaborative work with Sayamindu Dasgupta, Andr s Monroy-Hern ndez, and William Hale. The paper is released as open access so anyone can read the entire paper here. This blog post was also posted on Sayamindu Dasgupta s blog and on Medium by the MIT Media Lab.
| Series: | Steerswomen #3 |
| Publisher: | Rosemary Kirstein |
| Copyright: | 2003, 2014 |
| Printing: | 2014 |
| ISBN: | 0-9913546-2-1 |
| Format: | Kindle |
| Pages: | 432 |
Next.