|Series:||Dead Djinn Universe #1|
|Series:||Dead Djinn Universe #1|
whileloop. It (very cleverly) uses his quantmod package to access the SP500 in real-time . (I use quotes here because at the end of retail broadband one is not at the same market action as someone co-located in a New Jersey data center. It is however not delayed: as an index, it is not immediately tradeable as a stock, etf, or derivative may be all of which are only disseminated as delayed price information, usually by ten minutes.) I quite enjoyed the gist and used it and started tinkering with it. For example, it collects data but only saves (i.e. persists ) it after market close. If for whatever reason one needs to restart recent history is gone. In any event, I used his code and generalized it a little and published this about a year ago as function
intradayMarketMonitor()in my dang package. (See this blog post announcing it.) The chart of the left shows this in action, the chart is a snapshot from a couple of days ago when the vignettes (more on them below) were written. As lovely as
intradayMarketMonitor()is, it also limits itself to market hours. And sometimes you want to see, say, how the market opens on Sunday (futures usually restart at 17h Chicago time), or how news dissipates during the night, or where markets are pre-open, or . So I both wanted to complement this with futures, and also cache it locally so that, say, one machine might collect data and one (or several others) can visualize. For such tasks, Redis is unparalleled. (Yet I also always felt Redis could do with another, simple, short and sweet introduction stressing the key features of i) being multi-lingual: write in one language, consume in another and ii) loose coupling: no linking as one talks to Redis via standard tcp/ip networking. So I wrote a new intro vignette that is now in RcppRedis. I hope this comes in handy. Comments welcome!) Our RcppRedis package had long been used for such tasks, and it was easy to set it up. Standard use is to loop, fetch some data, push it to Redis, sleep, and start over. Clients do the same: fetch most recent data, plot or report it, sleep, start over. That works, but it has a dual delay as the client sleeping may miss the data update! The standard answer to this is called publish/pubscribe, or pub/sub. Libraries such as 0mq or zeromq specialise in this. But it turns out Redis already has it. I had some initial difficulty adding it to RcppRedis so for a trial I tested the marvellous rredis package by Bryan and simply instantiated two Redis clients. Now the data getter simply publishes a new data point in a given channel, by convention named after the security it tracks. Clients register with the Redis server which does all the actual work of keeping track of who listens to what. The clients now simply listen (which is a blocking operation) and as soon as data comes in receive it. This is quite mesmerizing when you just run two command-line clients (in a byobu session, say). As sone as the data is written (as shown on console log) it is consumed. No measruable overhead. Just lovely. Bryan and I then talked a litte as he may or may not retire rredis. Having implemented the pub/sub logic for both sides once, he took a good hard look at RcppRedis and just like that added it there. With some really clever wrinkles for (optional) per-symbol callback as closure attached to the instance. Truly amazeballs And once we had it in there, generalizing from publishing or subscribing to just one symbol easily generalizes to having one listener collect and publish for multiple symbols, and having one or more clients subscribe and listen one, more or even all symbol. All with ease thanks tp Redis. The second chart, also from a few days ago, shows four symbols for four (front-contract) futures for Bitcoin, Crude Oil, SP500, and Gold. As all this can get a little technical, I wrote a second vignette for RcppRedis on just this: market monitoring. Give this a read, if interested, feedback on this one is most welcome too! But all the code you need is included in the package just run a local Redis instance. Before closing, one sour note. I uploaded all this in a new and much improved updated RcppRedis 0.2.0 to CRAN on March 13 ten days ago. Not only is it still not there , but CRAN in their most delightful way also refuses to answer any emails of mine. Just lovely. The package exhibited just one compiler warning: a C++ compiler objected to the (embedded) C library hiredis (included as a fallback) for using a C language construct. Yes. A C++ compiler complaining about C. It s a non-issue. Yet it s been ten days and we still have nothing. So irritating and demotivating. Anyway, you can get the package off its GitHub repo. If you like this or other open-source work I do, you can sponsor me at GitHub.
What is frustrating is that there's actually no network error here. Running the command by hand I did see a different message, but now I have lost it in my backlog. It had something to do with a filename being too long, and I gave up debugging after a while. This happened suddenly too, which added to the confusion. In a fit of rage I started this blog post and experimenting with alternatives, which led me down a lot of rabbit holes. Reviewing my previous mail crash documentation, it seems most solutions involve talking to an IMAP server, so I figured I would just do that. Wanting to try something new, i gave isync (AKA
nov 15 16:12:19 angela systemd: Starting pull emails with syncmaildir... nov 15 16:12:22 angela systemd: smd-pull.service: Succeeded. nov 15 16:12:22 angela systemd: Finished pull emails with syncmaildir. nov 15 16:14:08 angela systemd: Starting pull emails with syncmaildir... nov 15 16:14:11 angela systemd: smd-pull.service: Main process exited, code=exited, status=1/FAILURE nov 15 16:14:11 angela systemd: smd-pull.service: Failed with result 'exit-code'. nov 15 16:14:11 angela systemd: Failed to start pull emails with syncmaildir. nov 15 16:16:14 angela systemd: Starting pull emails with syncmaildir... nov 15 16:16:17 angela smd-pull: smd-client: ERROR: Network error. nov 15 16:16:17 angela smd-pull: smd-client: ERROR: Unable to get any data from the other endpoint. nov 15 16:16:17 angela smd-pull: smd-client: ERROR: This problem may be transient, please retry. nov 15 16:16:17 angela smd-pull: smd-client: ERROR: Hint: did you correctly setup the SERVERNAME variable nov 15 16:16:17 angela smd-pull: smd-client: ERROR: on your client? Did you add an entry for it in your ssh nov 15 16:16:17 angela smd-pull: smd-client: ERROR: configuration file? nov 15 16:16:17 angela smd-pull: smd-client: ERROR: Network error nov 15 16:16:17 angela smd-pull: register: smd-client@localhost: TAGS: error::context(handshake) probable-cause(network) human-intervention(avoidable) suggested-actions(retry) nov 15 16:16:17 angela systemd: smd-pull.service: Main process exited, code=exited, status=1/FAILURE nov 15 16:16:17 angela systemd: smd-pull.service: Failed with result 'exit-code'. nov 15 16:16:17 angela systemd: Failed to start pull emails with syncmaildir.
mbsync) a try. Oh dear, I did not expect how much trouble just talking to my IMAP server would be, which wasn't not isync's fault, for what that's worth. It was the primary tool I used to debug things, and served me well in that regard.
mbsyncwould stop on a
FETCHcommand and Dovecot would give me those errors on the server side.
At least this first error was automatically healed by Dovecot (by renaming the file without the
nov 16 15:31:27 marcos dovecot: imap(anarcat)<3630489><wAmSzO3QZtfAqAB1>: Error: Mailbox junk: Maildir filename has wrong W value, renamed the file from /home/anarcat/Maildir/.junk/cur/1454623938.M101164P22216.marcos,S=2495,W=2578:2,S to /home/anarcat/Maildir/.junk/cur/1454623938.M101164P22216.marcos,S=2495:2,S nov 16 15:31:27 marcos dovecot: imap(anarcat)<3630489><wAmSzO3QZtfAqAB1>: Error: Mailbox junk: Deleting corrupted cache record uid=1582: UID 1582: Broken virtual size in mailbox junk: read(/home/anarcat/Maildir/.junk/cur/1454623938.M101164P22216.marcos,S=2495,W=2578:2,S): FETCH BODY got too little data: 2540 vs 2578
W=flag). The problem is that the
FETCHcommand fails and
mbsyncexits noisily. So you need to constantly restart
mbsyncwith a silly command like:
while ! mbsync -a; do sleep 1; done
This second problem is much harder to fix, because dovecot does not recover automatically. This is Dovecot complaining that the cached size (the
nov 16 13:53:08 marcos dovecot: imap(anarcat)<3594402><M5JHb+zQ3NLAqAB1>: Error: Mailbox Sent: UID=19288: read(/home/anarcat/Maildir/.Sent/cur/1224790447.M898726P9811V000000000000FE06I00794FB1_0.marvin,S=2588:2,S) failed: Cached message size larger than expected (2588 > 2482, box=Sent, UID=19288) (read reason=mail stream) nov 16 13:53:08 marcos dovecot: imap(anarcat)<3594402><M5JHb+zQ3NLAqAB1>: Error: Mailbox Sent: Deleting corrupted cache record uid=19288: UID 19288: Broken physical size in mailbox Sent: read(/home/anarcat/Maildir/.Sent/cur/1224790447.M898726P9811V000000000000FE06I00794FB1_0.marvin,S=2588:2,S) failed: Cached message size larger than expected (2588 > 2482, box=Sent, UID=19288) nov 16 13:53:08 marcos dovecot: imap(anarcat)<3594402><M5JHb+zQ3NLAqAB1>: Error: Mailbox Sent: UID=19288: read(/home/anarcat/Maildir/.Sent/cur/1224790447.M898726P9811V000000000000FE06I00794FB1_0.marvin,S=2588:2,S) failed: Cached message size larger than expected (2588 > 2482, box=Sent, UID=19288) (read reason=) nov 16 13:53:08 marcos dovecot: imap-login: Panic: epoll_ctl(del, 7) failed: Bad file descriptor
S=field, but also present in Dovecot's metadata files) doesn't match the file size. I wonder if at least some of those messages were corrupted in the OfflineIMAP to syncmaildir migration because part of that procedure is to run the strip_header script to remove content from the emails. That could easily have broken things since the files do not also get renamed.
mbsyncwas able to sync the entire mail spool. And no, rebuilding the index files didn't work. Also tried
doveadm force-resync -u anarcatwhich didn't do anything. In the end I also had to do this, because the wrong cache values were also stored elsewhere.
This would have totally broken any existing clients, but thankfully I'm starting from scratch (except maybe webmail, but I'm hoping it will self-heal as well, assuming it only has a cache and not a full replica of the mail spool).
service dovecot stop ; find -name 'dovecot*' -delete; service dovecot start
mbsyncwas incomplete as it was missing about 15,000 mails:
As it turns out,
anarcat@angela:~(main)$ find Maildir -type f -type f -a \! -name '.*' wc -l 384836 anarcat@angela:~(main)$ find Maildir-mbsync/ -type f -a \! -name '.*' wc -l 369221
mbsyncwas not at fault here either: this was yet more mail spool corruption. It's actually 26 folders (out of 205) with inconsistent sizes, which can be found with:
for folder in * .[^.]* ; do printf "%s\t%d\n" $folder $(find "$folder" -type f -a \! -name '.*' wc -l ); done
\! -name '.*'bit is to ignore the
mbsyncmetadata, which creates
.mbsyncstatein every folder. That ignores about 200 files but since they are spread around all folders, which was making it impossible to review where the problem was. Here is what the diff looks like:
--- Maildir-list 2021-11-17 20:42:36.504246752 -0500 +++ Maildir-mbsync-list 2021-11-17 20:18:07.731806601 -0500 @@ -6,16 +6,15 @@ [...] .Archives 1 .Archives.2010 3553 -.Archives.2011 3583 -.Archives.2012 12593 +.Archives.2011 3582 +.Archives.2012 620 .Archives.2013 8576 .Archives.2014 11057 -.Archives.2015 8173 +.Archives.2015 8165 .Archives.2016 54 .band 34 .bitbuck 1 @@ -38,13 +37,12 @@ .couchsurfers 2 -cur 11285 +cur 11280 .current 130 .cv 2 .debbug 262 -.debian 37544 -drafts 1 -.Drafts 4 +.debian 37533 +.Drafts 2 .drone 241 .drupal 188 .drupal-devel 303 [...]
Archivesfolders. As it turns out, at least 12,000 of those missing mails were actually misfiled: instead of being in the
Maildir/.Archives.2012/cur/folder, they were directly in
Maildir/.Archives.2012/. This is something that doesn't matter for SMD (
mbsync... After moving those files around, we still have 4,000 message missing:
The problem is that those 4,000 missing mails are harder to track. Take, for example,
anarcat@angela:~(main)$ find Maildir-mbsync/ -type f -a \! -name '.*' wc -l 381196 anarcat@angela:~(main)$ find Maildir/ -type f -a \! -name '.*' wc -l 385053
.Archives.2011, which has a single message missing, out of 3,582. And the files are not identical: the checksums don't match after going through the IMAP transport, so we can't use a tool like hashdeep to compare the trees and find why any single file is missing.
registerin my spool, which I am syncing separately (see Securing registration email for details on that setup). That actually covers 3,700 of those messages, so I actually have a more modest 300 messages to figure out, after (easily!) configuring
mbsyncto sync that folder separately:
@@ -30,9 +33,29 @@ Slave :anarcat-local: # Exclude everything under the internal [Gmail] folder, except the interesting folders #Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred" "[Gmail]/All Mail" # Or include everything -Patterns * +#Patterns * +Patterns * !register !.register # Automatically create missing mailboxes, both locally and on the server #Create Both Create slave # Sync the movement of messages between folders and deletions, add after making sure the sync works #Expunge Both + +IMAPAccount anarcat-register +Host imap.anarc.at +User register +PassCmd "pass imap.anarc.at-register" +SSLType IMAPS +CertificateFile /etc/ssl/certs/ca-certificates.crt + +IMAPStore anarcat-register-remote +Account anarcat-register + +MaildirStore anarcat-register-local +SubFolders Maildir++ +Inbox ~/Maildir-mbsync/.register/ + +Channel anarcat-register +Master :anarcat-register-remote: +Slave :anarcat-register-local: +Create slave
Argh. After more digging, I have found 131 mails in the
anarcat@angela:~(main)$ find Maildir-mbsync/ -type f -a \! -name '.*' wc -l 384900 anarcat@angela:~(main)$ find Maildir/ -type f -a \! -name '.*' wc -l 385059
tmp/directories of the client's mail spool. Mysterious! On the server side, it's even more files, and not the same ones. Possible that those were mails that were left there during a failed delivery of some sort, during a power failure or some sort of crash? Who knows. It could be another race condition in SMD if it runs while mail is being delivered in
tmp/... The first thing to do with those is to cleanup a bunch of empty files (21 on angela):
As it turns out, they are all duplicates, in the sense that notmuch can easily find a copy of files with the same message ID in its database. In other words, this hairy command returns nothing
find .[^.]*/tmp -type f -empty -delete
... which is good. Or, to put it another way, this is safe:
find .[^.]*/tmp -type f while read path; do msgid=$(grep -m 1 -i ^message-id "$path" sed 's/Message-ID: //i;s/[<>]//g'); if notmuch count --exclude=false "id:$msgid" grep -q 0; then echo "$path <$msgid> not in notmuch" ; fi; done
Poof! 314 mails cleaned on the server side. Interestingly, SMD doesn't pick up on those changes at all and still sees files in
find .[^.]*/tmp -type f -delete
tmp/directories on the client side, so we need to operate the same twisted logic there.
Ha! 27 mails difference. Those are the really sticky, unclear ones. I was hoping a full sync might clear that up, but after deleting the entire directory and starting from scratch, I end up with:
anarcat@angela:~(main)$ find Maildir/ -type f -a \! -name '.*' wc -l 384928 anarcat@angela:~(main)$ find Maildir-mbsync/ -type f -a \! -name '.*' wc -l 384901
That is: even more messages missing (now 37). Sigh. Thankfully, this is something notmuch can help with: it can index all files by
anarcat@angela:~(main)$ find Maildir -type f -type f -a \! -name '.*' wc -l 385034 anarcat@angela:~(main)$ find Maildir-mbsync -type f -type f -a \! -name '.*' wc -l 384993
Message-ID(which I learned is case-insensitive, yay) and tell us which messages don't make it through. Considering the corruption I found in the mail spool, I wouldn't be the least surprised those messages are just skipped by the IMAP server. Unfortunately, there's nothing on the Dovecot server logs that would explain the discrepancy. Here again, notmuch comes to the rescue. We can list all message IDs to figure out that discrepancy:
And then we can see how many messages notmuch thinks are missing:
notmuch search --exclude=false --output=messages '*' pv -s 18M sort > Maildir-msgids notmuch --config=.notmuch-config-mbsync search --exclude=false --output=messages '*' pv -s 18M sort > Maildir-mbsync-msgids
That's 29 messages. Oddly, it doesn't exactly match the
$ wc -l *msgids 372723 Maildir-mbsync-msgids 372752 Maildir-msgids
That is 10 more messages. Ugh. But actually, I know what those are: more misfiled messages (in a
anarcat@angela:~(main)$ find Maildir-mbsync -type f -type f -a \! -name '.*' wc -l 385204 anarcat@angela:~(main)$ find Maildir -type f -type f -a \! -name '.*' wc -l 385241
.folder/draft/directory, bizarrely, so the totals actually match. In the notmuch output, there's a lot of stuff like this:
Those are messages without a valid Message-ID. Notmuch (presumably) constructs one based on the file's checksum. Because the files differ between the IMAP server and the local mail spool (which is unfortunate, but possibly inevitable), those do not match. There are exactly the same number of those on both sides, so I'll go ahead and assume those are all accounted for. What remains is:
ie. 21 missing from
anarcat@angela:~(main)$ diff -u Maildir-mbsync-msgids Maildir-msgids grep '^\-[^-]' grep -v sha1 wc -l 2 anarcat@angela:~(main)$ diff -u Maildir-mbsync-msgids Maildir-msgids grep '^\+[^+]' grep -v sha1 wc -l 21 anarcat@angela:~(main)$
mbsync, and, surprisingly, 2 missing from the original mail spool. Further inspection also showed they were all messages with some sort of "corruption": no body and only headers. I am not sure that is a legal email format in the first place. Since they were mostly spam or administrative emails ("You have been unsubscribed from mailing list..."), it seems fairly harmless to ignore those.
mbsyncprovide fast and reliable transport, including over SSH. See the next article for further discussion of the alternatives.
|Series:||Chronicles of Narnia #6|
All of the faces they could see were certainly nice. Both the men and women looked kind and wise, and they seemed to come of a handsome race. But after the children had gone a few steps down the room they came to faces that looked a little different. These were very solemn faces. You felt you would have to mind your P's and Q's, if you ever met living people who looked like that. When they had gone a little farther, they found themselves among faces they didn't like: this was about the middle of the room. The faces here looked very strong and proud and happy, but they looked cruel. A little further on, they looked crueller. Further on again, they were still cruel but they no longer looked happy. They were even despairing faces: as if the people they belonged to had done dreadful things and also suffered dreadful things.The last statue is of a fierce, proud woman that Digory finds strikingly beautiful. (Lewis notes in an aside that Polly always said she never found anything specially beautiful about her. Here, as in The Silver Chair, the girl is the sensible one and things would have gone better if the boy had listened to her, a theme that I find immensely frustrating because Susan was the sensible one in the first two books of the series but then Lewis threw that away.) There is a bell in the middle of this hall, and the pillar that holds that bell has an inscription on it that I think every kid who grew up on Narnia knows by heart.
Make your choice, adventurous Stranger;Polly has no intention of striking the bell, but Digory fights her and does it anyway, waking Jadis from where she sat as the final statue in the hall and setting off one of the greatest reimaginings of a villain in children's literature. Jadis will, of course, become the White Witch who holds Narnia in endless winter some thousand Narnian years later. But the White Witch was a mediocre villain at best, the sort of obvious and cruel villain common in short fairy tales where the author isn't interested in doing much characterization. She exists to be evil, do bad things, and be defeated. She has a few good moments in conflict with Aslan, but that's about it. Jadis in this book is another matter entirely: proud, brilliant, dangerous, and creative. The death of everything on Charn was Jadis's doing: an intentional spell, used to claim a victory of sorts from the jaws of defeat by her sister in a civil war. (I find it fascinating that Lewis puts aside his normally sexist roles here.) Despite the best attempts of the kids to lose her both in Charn and in the Wood (which is inimical to her, in another nice bit of world-building), she manages to get back to England with them. The result is a remarkably good bit of villain characterization. Jadis is totally out of her element, used to a world-spanning empire run with magic and (from what hints we get) vaguely medieval technology. Her plan to take over their local country and eventually the world should be absurd and is played somewhat for laughs. Her magic, which is her great weapon, doesn't even work in England. But Jadis learns at a speed that the reader can watch. She's observant, she pays attention to things that don't fit her expectations, she changes plans, and she moves with predatory speed. Within a few hours in London she's stolen jewels and a horse and carriage, and the local police seem entirely overmatched. There's no way that one person without magic should be a real danger to England around the turn of the 20th century, but by the time the kids manage to pull her back into the Wood, you're not entirely sure England would have been safe. A chaotic confrontation, plus the ability of the rings to work their magic through transitive human contact, ends up with the kids, Uncle Andrew, Jadis, a taxicab driver and his horse all transported through the Wood to a new world. In this case, literally a new world: Narnia at the point of its creation. Here again, Lewis translates Christian myth, in this case the Genesis creation story, into a more vivid and in many ways more beautiful story than the original. Aslan singing the world into existence is an incredible image, as is the newly-created world so bursting with life that even things that normally could not grow will do so. (Which, of course, is why there is a lamp post burning in the middle of the western forest of Narnia for the Pevensie kids to find later.) I think my favorite part is the creation of the stars, but the whole sequence is great. There's also an insightful bit of human psychology. Uncle Andrew can't believe that a lion is singing, so he convinces himself that Aslan is not singing, and thus prevents himself from making any sense of the talking animals later.
Strike the bell and bide the danger,
Or wonder, till it drives you mad,
What would have followed if you had.
Now the trouble about trying to make yourself stupider than you really are is that you very often succeed.As with a lot in Lewis, he probably meant this as a statement about faith, but it generalizes well beyond the religious context. What disappointed me about the creation story, though, is the animals. I didn't notice this as a kid, but this re-read has sensitized me to how Lewis consistently treats the talking animals as less than humans even though he celebrates them. That happens here too: the newly-created, newly-awakened animals are curious and excited but kind of dim. Some of this is an attempt to show that they're young and are just starting to learn, but it also seems to be an excuse for Aslan to set up a human king and queen over them instead of teaching them directly how to deal with the threat of Jadis who the children inadvertently introduced into the world. The other thing I dislike about The Magician's Nephew is that the climax is unnecessarily cruel. Once Digory realizes the properties of the newly-created world, he hopes to find a way to use that to heal his mother. Aslan points out that he is responsible for Jadis entering the world and instead sends him on a mission to obtain a fruit that, when planted, will ward Narnia against her for many years. The same fruit would heal his mother, and he has to choose Narnia over her. (It's a fairly explicit parallel to the Garden of Eden, except in this case Digory passes.) Aslan, in the end, gives Digory the fruit of the tree that grows, which is still sufficient to heal his mother, but this sequence made me angry when re-reading it. Aslan knew all along that what Digory is doing will let him heal his mother as well, but hides this from him to make it more of a test. It's cruel and mean; Aslan could have promised to heal Digory's mother and then seen if he would help Narnia without getting anything in return other than atoning for his error, but I suppose that was too transactional for Lewis's theology or something. Meh. But, despite that, the only reason why this is not the best Narnia book is because The Voyage of the Dawn Treader is the only Narnia book that also nails the ending. The Magician's Nephew, up through Charn, Jadis's rampage through London, and the initial creation of Narnia, is fully as good, perhaps better. It sags a bit at the end, partly because it tries to hard to make the Narnian animals humorous and partly because of the unnecessary emotional torture of Digory. But this still holds up as the second-best Narnia book, and one I thoroughly enjoyed re-reading. If anything, Jadis and Charn are even better than I remembered. Followed by the last book of the series, the somewhat notorious The Last Battle. Rating: 9 out of 10
arch/arm/boot/dts/bcm2711-rpi-4-b.dtsand then I did:
cpp -nostdinc -I include -I arch -undef -x assembler-with-cpp \ arch/arm/boot/dts/bcm2711-rpi-4-b.dts > rpi4.preprocessed dtc -I dts -O dtb rpi4.preprocessed -o bcm2711-rpi-4-b.dtb
bcm2711-rpi-4-b.dtbfile replaced the one in
/boot/firmware. This isn t a necessary step if you don t want to use the cooling fan in the case, or the front USB ports, and it s not really anyone s fault, but it was an annoying extra step to have to figure out. The DeskPi came with a microSD card that was supposed to have RaspiOS already on it. It didn t, it was blank. In my case that was fine, because I wanted to use Debian, but it was a minor niggle.
apt upgradeand got updated to the Buster 10.7 release, as well as the latest 5.9 backport kernel, and everything came back without effort after a reboot. It s lovely to be able to run Debian on this device without having to futz around with self-compiled kernels. The DeskPi makes a lot of effort to route things externally. The SD slot is brought out to the front, making it easy to fiddle with the card contents without having to open the case to replace it. All the important ports are brought out to the back either through orientation of the Pi, or extenders in the case. That means the built in Pi USB ports, the HDMI sockets (conveniently converted to full size internally), an audio jack and a USB-C power port. The aforementioned USB3 dongle for the bridge to the drive is the only external thing that s annoying. Thermally things seem good too. I haven t done a full torture test yet, but with the fan off the system is sitting at about 40 C while fairly idle. Some loops in bash that push load up to above 2 get the temperature up to 46 C or so, and turning the fan on brings it down to 40 C again. It s audible, but quieter than my laptop and not annoying. I liked the way the case came with everything I needed other than the Pi 4 and a suitable disk drive. There was an included PSU (a proper USB-C PD device, UK plug), the heatsink/fan is there, the USB/SATA converter is there and even an SD card is provided (though that s just because I had a pre-order). Speaking of the SD, I only needed it for initial setup. Recent Pi 4 bootloaders are capable of booting directly from USB mass storage devices. So I upgraded using the RPi EEPROM Recovery image (which just needs extracted to the SD FAT partition, no need for anything complicated - boot with it and the screen goes all green and you know it s ok), then created a FAT partition at the start of the drive for the kernel / bootloader config and a regular EXT4 partition for root. Copies everything over, updated paths, took out the SD and it all just works happily.
Why is my e-reader still getting regular OS updates, while Google stopped issuing security patches for my smartphone four years ago?To try to answer this, let us turn to economic incentives theory. Although not the be-all and end-all some think it is3, incentives theory is not a bad tool to analyse this particular problem. Executives at Google most likely followed a very business-centric logic when they decided to drop support for the Nexus 5. Likewise, Rakuten Kobo's decision to continue updating older devices certainly had very little to do with ethics or loyalty to their user base. So, what are the incentives that keep Kobo updating devices and why are they different than smartphone manufacturers'? A portrait of the current long-term software support offerings for smartphones and e-readers Before delving deeper in economic theory, let's talk data. I'll be focusing on 2 brands of e-readers, Amazon's Kindle and Rakuten's Kobo. Although the e-reader market is highly segmented and differs a lot based on geography, Amazon was in 2015 the clear worldwide leader with 53% of the worldwide e-reader sales, followed by Rakuten Kobo at 13%4. On the smartphone side, I'll be differentiating between Apple's iPhones and Android devices, taking Google as the barometer for that ecosystem. As mentioned below, Google is sadly the leader in long-term Android software support. Rakuten Kobo According to their website and to this Wikipedia table, the only e-readers Kobo has deprecated are the original Kobo eReader and the Kobo WiFi N289, both released in 2010. This makes their oldest still supported device the Kobo Touch, released in 2011. In my book, that's a pretty good track record. Long-term software support does not seem to be advertised or to be a clear selling point in their marketing. Amazon According to their website, Amazon has dropped support for all 8 devices produced before the Kindle Paperwhite 2nd generation, first sold in 2013. To put things in perspective, the first Kindle came out in 2007, 3 years before Kobo started selling devices. Like Rakuten Kobo, Amazon does not make promises of long-term software support as part of their marketing. Apple Apple has a very clear software support policy for all their devices:
Owners of iPhone, iPad, iPod or Mac products may obtain a service and parts from Apple or Apple service providers for five years after the product is no longer sold or longer, where required by law.This means in the worst-case scenario of buying an iPhone model just as it is discontinued, one would get a minimum of 5 years of software support. Android Google's policy for their Android devices is to provide software support for 3 years after the launch date. If you buy a Pixel device just before the new one launches, you could theoretically only get 2 years of support. In 2018, Google decided OEMs would have to provide security updates for at least 2 years after launch, threatening not to license Google Apps and the Play Store if they didn't comply. A question of cost structure From the previous section, we can conclude that in general, e-readers seem to be supported longer than smartphones, and that Apple does a better job than Android OEMs, providing support for about twice as long. Even Fairphone, who's entire business is to build phones designed to last and to be repaired was not able to keep the Fairphone 1 (2013) updated for more than a couple years and seems to be struggling to keep the Fairphone 2 (2015) running an up to date version of Android. Anyone who has ever worked in IT will tell you: maintaining software over time is hard work and hard work by specialised workers is expensive. Most commercial electronic devices are sold and developed by for-profit enterprises and software support all comes down to a question of cost structure. If companies like Google or Fairphone are to be expected to provide long-term support for the devices they manufacture, they have to be able to fund their work somehow. In a perfect world, people would be paying for the cost of said long-term support, as it would likely be cheaper then buying new devices every few years and would certainly be better for the planet. Problem is, manufacturers aren't making them pay for it. Economists call this type of problem externalities: things that should be part of the cost of a good, but aren't for one a reason or another. A classic example of an externality is pollution. Clearly pollution is bad and leads to horrendous consequences, like climate change. Sane people agree we should drastically cut our greenhouse gas emissions, and yet, we aren't. Neo-classical economic theory argues the way to fix externalities like pollution is to internalise these costs, in other words, to make people pay for the "real price" of the goods they buy. In the case of climate change and pollution, neo-classical economic theory is plain wrong (spoiler alert: it often is), but this is where band-aids like the carbon tax comes from. Still, coming back to long-term software support, let's see what would happen if we were to try to internalise software maintenance costs. We can do this multiple ways. 1 - Include the price of software maintenance in the cost of the device This is the choice Fairphone makes. This might somewhat work out for them since they are a very small company, but it cannot scale for the following reasons:
the program code would be doing this:int socket = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);
According to the ;login: article, converting a C program to use TLS would normally modify only 5-10 lines in the code, which is amazing when compared to using for example the OpenSSL API. The project has set up the https://securesocketapi.org/ web site to spread the idea, and the code for a kernel module and the associated system daemon is available from two github repositories: ssa and ssa-daemon. Unfortunately there is no explicit license information with the code, so its copyright status is unclear. A request to solve this about it has been unsolved since 2018-08-17. I love the idea of extending socket() to gain TLS support, and understand why it is an advantage to implement this as a kernel module and system wide service daemon, but can not help to think that it would be a lot easier to get projects to move to this way of setting up TLS if it was done with a user space approach where programs wanting to use this API approach could just link with a wrapper library. I recommend you check out this simple and powerful approach to more secure network connections. :) As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.int socket = socket(PF_INET, SOCK_STREAM, IPPROTO_TLS);
Once there was a brave and temeraryous (sp?) girl who found a Door. It was a magic Door that's why it has a capital D. She opened the Door.The Door led to a bluff over the sea and above a city, a place very far from Kentucky, and she almost stayed, but she came back through the Door when her guardian, Mr. Locke, called. The adventure cost her a diary, several lectures, days of being locked in her room, and the remnants of her strained relationship with her father. When she went back, the frame of the Door was burned to the ground. That was the end of Doors for January for some time, and the continuation of a difficult childhood. She was cared for by her father's employer as a sort of exotic pet, dutifully attempting to obey, grateful for Mr. Locke's protection, and convinced that he was occasionally sneaking her presents through a box in the Pharaoh Room out of some hidden kindness. Her father appeared rarely, said little, and refused to take her with him. Three things helped: the grocery boy who smuggled her stories, an intimidating black woman sent by her father to replace her nurse, and her dog.
Once upon a time there was a good girl who met a bad dog, and they became the very best of friends. She and her dog were inseparable from that day forward.I will give you a minor spoiler that I would have preferred to have had, since it would have saved me some unwarranted worry and some mental yelling at the author: The above story strains but holds. January's adventure truly starts the day before her seventeenth birthday, when she finds a book titled The Ten Thousand Doors in the box in the Pharaoh Room. As you may have guessed from the title, The Ten Thousand Doors of January is a portal fantasy, but it's the sort of portal fantasy that is more concerned with the portal itself than the world on the other side of it. (Hello to all of you out there who, like me, have vivid memories of the Wood between the Worlds.) It's a book about traveling and restlessness and the possibility of escape, about the ability to return home again, and about the sort of people who want to close those doors because the possibility of change created by people moving around freely threatens the world they have carefully constructed. Structurally, the central part of the book is told by interleaving chapters of January's tale with chapters from The Ten Thousand Doors. That book within a book starts with the framing of a scholarly treatment but quickly becomes a biography of a woman: Adelaide Lee Larson, a half-wild farm girl who met her true love at the threshold of a Door and then spent much of her life looking for him. I am not a very observant reader for plot details, particularly for books that I'm enjoying. I read books primarily for the emotional beats and the story structure, and often miss rather obvious story clues. (I'm hopeless at guessing the outcomes of mysteries.) Therefore, when I say that there are many things January is unaware of that are obvious to the reader, that's saying a lot. Even more clues were apparent when I skimmed the first chapter again, and a more observant reader would probably have seen them on the first read. Right down to Mr. Locke's name, Harrow is not very subtle about the moral shape of this world. That can make the early chapters of the book frustrating. January is being emotionally (and later physically) abused by the people who have power in her life, but she's very deeply trapped by false loyalty and lack of external context. Winning free of that is much of the story of the book, and at times it has the unpleasantness of watching someone make excuses for her abuser. At other times it has the unpleasantness of watching someone be abused. But this is the place where I thought the nested story structure worked marvelously. January escapes into the story of The Ten Thousand Doors at the worst moments of her life, and the reader escapes with her. Harrow uses the desire to switch scenes back to the more adventurous and positive story to construct and reinforce the emotional structure of the book. For me, it worked extremely well. It helps that the ending is glorious. The payoff is worth all the discomfort and tension-building in the first half of the book. Both The Ten Thousand Doors and the surrounding narrative reach deeply satisfying conclusions, ones that are entangled but separate in just the ways that they need to be. January's abilities, actions, and decisions at the end of the book were just the outcome that I needed but didn't entirely guess in advance. I could barely put down the last quarter of this story and loved every moment of the conclusion. This is the sort of book that can be hard to describe in a review because its merits don't rest on an original twist or easily-summarized idea. The elements here are all elements found in other books: portal fantasy, the importance of story-telling, coming of age, found family, irrepressible and indomitable characters, and the battle of the primal freedom of travel and discovery and belief against the structural forces that keep rulers in place. The merits of this book are in the small details: the way that January's stories are sparse and rare and sometimes breathtaking, the handling of tattoos, the construction of other worlds with a few deft strokes, and the way Harrow embraces the emotional divergence between January's life and Adelaide's to help the reader synchronize the emotional structure of their reading experience with January's.
She writes a door of blood and silver. The door opens just for her.The Ten Thousand Doors of January is up against a very strong slate for both the Nebula and the Hugo this year, and I suspect it may be edged out by other books, although I wouldn't be unhappy if it won. (It probably has a better shot at the Nebula than the Hugo.) But I will be stunned if Harrow doesn't walk away with the Mythopoeic Award. This seems like exactly the type of book that award was created for. This is an excellent book, one of the best I've read so far this year. Highly recommended. Rating: 9 out of 10
Sealioning (also spelled sea-lioning and sea lioning) is a type of trolling or harassment which consists of pursuing people with persistent requests for evidence or repeated questions, while maintaining a pretense of civility and sincerity. It may take the form of "incessant, bad-faith invitations to engage in debate".
Tone policing (also tone trolling, tone argument, and tone fallacy) is an ad hominem (personal attack) and antidebate tactic based on criticizing a person for expressing emotion. Tone policing detracts from the validity of a statement by attacking the tone in which it was presented rather than the message itself.
"A brawler who tattoos a message onto his knuckles does not throw every punch with the weight of First Amendment protection behind him," the brief stated. "Conduct like this does not constitute speech, nor should it. A deliberate attempt to cause physical injury to someone does not come close to the expression which the First Amendment is designed to protect."
It s no secret that times are changing. It used to be that men were men, jokes were jokes, and all facts came from one white guy in a suit who you trusted because he looked like your dad. Now I know I could get in a lot of trouble for just saying this, but I don t care because someone has to tell the truth: These days, you can t say anything racist at all without being called a racist.
Russia's neighbor has developed a plan for countering misinformation. Can it be exported to the rest of the world?
So, I wrote my BIFF Response but Marvin wrote me another angry email. Actually, he wrote 6 more this week, so what s up with that? Why didn t he stop after my first email?
The BIFF Response Method will teach you how to respond to angry emails, texts, or social media posts while maintaining your dignity and personal power.
The problem with crying in the woods, by the side of a white road that leads somewhere terrible, is that the reason for crying isn't inside your head. You have a perfectly legitimate and pressing reason for crying, and it will still be there in five minutes, except that your throat will be raw and your eyes will itch and absolutely nothing else will have changed.Lord Crevan, when Rhea finally reaches him, toys with her by giving her progressively more horrible puzzle tasks, threatening her with the promised marriage if she fails at any of them. The way this part of the book finally resolves is one of the best moments I've read in any book. Kingfisher captures an aspect of moral decisions, and a way in which evil doesn't work the way that evil people expect it to work, that I can't remember seeing an author capture this well. There are a lot of things here for Rhea to untangle: the nature of Crevan's power, her unexpected allies in his manor, why he proposed marriage to her, and of course how to escape his power. The plot works, but I don't think it was the best part of the book, and it tends to happen to Rhea rather than being driven by her. But I have rarely read a book quite this confident of its moral center, or quite as justified in that confidence. I am definitely reading everything Vernon has published under the T. Kingfisher name, and quite possibly most of her children's books as well. Recommended, particularly if you liked the excerpt above. There's an entire book full of paragraphs like that waiting for you. Rating: 8 out of 10
iptablesare generally consistent across distributions, even though each distribution has its own way of using them. Yet Linux is not dominant, why? Mukherjee identified the problem as "packaging issues" and listed a set of features he would like Linux to improve:
ethtoolinterface. The idea is to make
ethtoola de-facto standard to manage switches and ports. Mukherjee gave the example that data centers spend more money on cables than any other hardware and explained that making it easier to identify cables is therefore a key functionality. Getting consistent interface naming was also a key problem identified by numerous participants at the conference. While systemd tried to fix this with the predictable network interface names specification, the interface names are not necessarily predictable across virtual machines or on special hardware; in fact, this was the topic of the first talk of the conference.
ethtoolalso needs to support interfaces that run faster than 1 Gbps, something that still has limited support in Linux at the moment.
switchdev_ops, which we previously covered,
netdev_ops. Certain switches, however, also need distributed switch architecture (DSA) features to be properly handled. DSA is a more obscure part of the network stack that allows Linux to represent hardware switches or chains of switches using regular Linux tools like
ifconfig, and so on. While switchdev is a new layer, DSA has been in the kernel since 2.6.28 in 2008. Originally developed to support Marvell switches, DSA is now a generic layer deployed in WiFi access points, set-top boxes, on-board flight entertainment systems, trains, and other industrial equipment. Switches that have an Ethernet controller need DSA, whereas the kernel can support switches without Ethernet controllers directly with switchdev drivers. The first years of DSA's development consisted only of basic maintenance but, in the last three years, DSA has seen a resurgence of contributions, as part of Linux networking push to support hardware offloading and network switches. Between 2014 and 2015, DSA added support for Broadcom hardware, wake on LAN, and hardware port bridging, among other features. DSA's development was parallel to swconfig, written by the OpenWrt project to support the small office and home office (SOHO) routers that the project is targeting. The main difference between swconfig and DSA is that DSA-supported switches show one network interface per port, whereas swconfig-configured switches show up as a single port, which limits the amount of information that can be extracted from the switch. For example, you cannot have per-port traffic statistics with swconfig. That limitation is what led to the creation of the switchdev framework, when swconfig was proposed (then refused) for inclusion in mainline. Another goal of switchdev was to support bridge hardware offloading and network interface card (NIC) virtualization. Also, whereas swconfig uses virtual LAN (VLAN) tagging to address ports, DSA enables the use of device-specific tagging headers to address different ports, which enables DSA to have better control over the switches. This allows, for example, DSA to do internet group management protocol (IGMP) snooping or implement the spanning tree protocol, whereas swconfig doesn't have those features. Some switches are actually connected to the host CPU through an Ethernet interface instead of regular PCI-Express interface, and DSA supports this as well. One advantage that remains in the swconfig approach is that it treats the internal switch as a simple external switch, and addresses ports with standard VLAN tags. This is something DSA could do, as well, but no one has bothered implementing this just yet. For now, DSA drivers use device-specific tagging mechanisms that limit the number of supported devices. Other areas of future improvement for DSA are better multi-chip support, IGMP snooping, and bonding, as well as firewall, NAT, and TC offloading.
The author would like to thank the Netdev organizers for travel assistance. Also, thanks to Andrew Lunn for a technical review of this article. Note: this article first appeared in the Linux Weekly News.
Today marks the 13th anniversary since the last passenger flight from New York arrived in the UK. Every seat was filled, a feat that had become increasingly rare for a plane that was a technological marvel but a commercial flop .
kwbootutility. For some reason the SPL didn't properly detect UART booting on my device (wrong magic number) but patching the
if(in arch-mvebu's spl.c) and always assume UART boot is an easy way around. The plan then was to boot a Debian armhf rootfs with a
defconfigkernel from USB stick. and install U-Boot and the rootfs to eMMC from within that system. Unfortunately U-Boot seems to be unable to talk to the USB3 port so no kernel loading from there. One could probably make UART loading work but switching between screen for serial console and xmodem seemed somewhat fragile and I never got it working. However ethernet can be made to work, though you need to set
eth3addr(or just the right one of these) in U-Boot,
saveenvand reboot. After that TFTP works (but is somewhat slow). eMMC There's one last step required to allow U-Boot and Linux to access the eMMC. eMMC is wired to the same PINs as the SD card would be. However the SD card has an additional indicator pin showing whether a card is present. You might be lucky inserting a dummy card into the slot or go the clean route and remove the pin specification from the device tree.
Next Up is flashing the U-Boot to eMMC. This seems to work with the vendor U-Boot but proves to be tricky with mainline. The fun part boils down to the fact that the boot firmware reads the first block from eMMC, but the second from SD card. If you write the mainline U-Boot, which was written and tested for SD card, to eMMC the SPL will try to load the main U-Boot starting from it's second sector from flash -- obviously resulting in garbage. This one took me several tries to figure out and made me read most of the SPL code for the device. The fix however is trivial (apart from the question on how to support all different variants from one codebase, which I'll leave to the U-Boot developers):
--- a/arch/arm/dts/armada-388-clearfog.dts +++ b/arch/arm/dts/armada-388-clearfog.dts @@ -306,7 +307,6 @@ sdhci@d8000 bus-width = <4>; - cd-gpios = <&gpio0 20 GPIO_ACTIVE_LOW>; no-1-8-v; pinctrl-0 = <&clearfog_sdhci_pins &clearfog_sdhci_cd_pins>;
Linux Now we have a System booting from eMMC with mainline U-Boot (which is a most welcome speedup compared to the UART and TFTP combination from the beginning). Getting to fine-tune linux on the device -- we want to install the
--- a/include/configs/clearfog.h +++ b/include/configs/clearfog.h @@ -143,8 +143,7 @@ #define CONFIG_SPL_LIBDISK_SUPPORT #define CONFIG_SYS_MMC_U_BOOT_OFFS (160 << 10) #define CONFIG_SYS_U_BOOT_OFFS CONFIG_SYS_MMC_U_BOOT_OFFS -#define CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR ((CONFIG_SYS_U_BOOT_OFFS / 512)\ - + 1) +#define CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR (CONFIG_SYS_U_BOOT_OFFS / 512) #define CONFIG_SYS_U_BOOT_MAX_SIZE_SECTORS ((512 << 10) / 512) /* 512KiB */ #ifdef CONFIG_SPL_BUILD #define CONFIG_FIXED_SDHCI_ALIGNED_BUFFER 0x00180000 /* in SDRAM */
armmpDebian kernel and have it work. As all the drivers are build as modules for that kernel this also means initrd support. Funnily U-Boots
bootzallows booting a plain
vmlinuxkernel but I couldn't get it to boot a plain initrd. Passing a uImage initrd and a normal kernel however works pretty well. Back when I first tried there were some modules missing and ethernet didn't work with the PHY driver built as a module. In the meantime the PHY problem was fixed in the Debian kernel and almost all modules already added. Ben then only added the USB3 module on my suggestion and as a result, unstable's armhf
armmpkernel should work perfectly well on the device (you still need to patch the device tree similar to the patch above). Still missing is an updated
flash-kernelto automatically generate the initrd uImage which is work in progress but got stalled until I fixed the U-Boot on eMMC problem and everything should be fine -- maybe get debian u-boot builds for that board. Pro versus Base The main difference so far between the Pro and the Base version of the ClearFog is the switch chip which is included on the Pro. The Base instead "just" has two gigabit ethernet ports and a SFP. Both, linux' and U-Boot's device tree are intended for the Pro version which makes on of the ethernet ports unusable (it tries to find the switch behind the ethernet port which isn't there). To get both ports working (or the one you settled on earlier) there's a second patch to the device tree (my version might be sub-optimal but works), U-Boot -- the linux-kernel version is a trivial adaption:
Conclusion Apart from the mess with eMMC this seems to be a pretty nice device. It's now happily running with a M.2 SSD providing enough storage for now and still has a mSATA/mPCIe plug left for future journeys. It seems to be drawing around 5.5 Watts with SSD and one Ethernet connected while mostly idle and can feed around 500 Mb/s from disk over an encrypted ethernet connection which is, I guess, not too bad. My plans now include helping to finish
--- a/arch/arm/dts/armada-388-clearfog.dts +++ b/arch/arm/dts/armada-388-clearfog.dts @@ -89,13 +89,10 @@ internal-regs ethernet@30000 mac-address = [00 50 43 02 02 02]; + managed = "in-band-status"; + phy = <&phy1>; phy-mode = "sgmii"; status = "okay"; - - fixed-link - speed = <1000>; - full-duplex; - ; ; ethernet@34000 @@ -227,6 +224,10 @@ pinctrl-0 = <&mdio_pins>; pinctrl-names = "default"; + phy1: ethernet-phy@1 /* Marvell 88E1512 */ + reg = <1>; + ; + phy_dedicated: ethernet-phy@0 /* * Annoyingly, the marvell phy driver @@ -386,62 +386,6 @@ tx-fault-gpio = <&expander0 13 GPIO_ACTIVE_HIGH>; ; - dsa@0 - compatible = "marvell,dsa"; - dsa,ethernet = <ð1>; - dsa,mii-bus = <&mdio>; - pinctrl-0 = <&clearfog_dsa0_clk_pins &clearfog_dsa0_pins>; - pinctrl-names = "default"; - #address-cells = <2>; - #size-cells = <0>; - - switch@0 - #address-cells = <1>; - #size-cells = <0>; - reg = <4 0>; - - port@0 - reg = <0>; - label = "lan1"; - ; - - port@1 - reg = <1>; - label = "lan2"; - ; - - port@2 - reg = <2>; - label = "lan3"; - ; - - port@3 - reg = <3>; - label = "lan4"; - ; - - port@4 - reg = <4>; - label = "lan5"; - ; - - port@5 - reg = <5>; - label = "cpu"; - ; - - port@6 - /* 88E1512 external phy */ - reg = <6>; - label = "lan6"; - fixed-link - speed = <1000>; - full-duplex; - ; - ; - ; - ; - gpio-keys compatible = "gpio-keys"; pinctrl-0 = <&rear_button_pins>;
flash-kernelsupport, creating a nice case and probably get it deployed. I might bring it to FOSDEM first though. Working on it was really quite some fun (apart from the frustrating parts finding the one-block-offset ..) and people were really helpful. Big thanks here to Debian's arm folks, Ben Hutchings the kernel maintainer and U-Boot upstream (especially Tom Rini and Stefan Roese)
armelport. Debian now supports the Seagate Personal Cloud and Seagate NAS devices. They are based on Marvell's Armada 370, a platform which can run Debian's
armhfport. Unfortunately, even the Armada 370 is a bit dated now, so I would not recommend these devices for new purchases. If you have one already, however, you now have the option to run native Debian. There are some features I like about the Seagate NAS devices:
Adults worry a lot these days. Especially, they worry about how to make other people learn more about computers. They want to make us all computer-literate. Literacy means both reading and writing, but most books and courses about computers only tell you about writing programs. Worse, they only tell about commands and instructions and programming-language grammar rules. They hardly ever give examples. But real languages are more than words and grammar rules. There s also literature what people use the language for. No one ever learns a language from being told its grammar rules. We always start with stories about things that interest us.In a new paper titled Remixing as a pathway to Computational Thinking that was recently published at the ACM Conference on Computer Supported Collaborative Work and Social Computing (CSCW) conference, we used a series of quantitative measures of online behavior to try to uncover evidence that might support the theory that remixing in Scratch is positively associated with learning. Of course, because Scratch is an informal environment with no set path for users, no lesson plan, and no quizzes, measuring learning is an open problem. In our study, we built on two different approaches to measure learning in Scratch. The first approach considers the number of distinct types of programming blocks available in Scratch that a user has used over her lifetime in Scratch (there are 120 in total) something that can be thought of as a block repertoire or vocabulary. This measure has been used to model informal learning in Scratch in an earlier study. Using this approach, we hypothesized that users who remix more will have a faster rate of growth for their code vocabulary. Controlling for a number of factors (e.g. age of user, the general level of activity) we found evidence of a small, but positive relationship between the number of remixes a user has shared and her block vocabulary as measured by the unique blocks she used in her non-remix projects. Intriguingly, we also found a strong association between the number of downloads by a user and her vocabulary growth. One interpretation is that this learning might also be associated with less active forms of appropriation, like the process of reading source code described by Minksy. The second approach we used considered specific concepts in programming, such as loops, or event-handling. To measure this, we utilized a mapping of Scratch blocks to key programming concepts found in this paper by Karen Brennan and Mitchel Resnick. For example, in the image below are all the Scratch blocks mapped to the concept of loop . We looked at six concepts in total (conditionals, data, events, loops, operators, and parallelism). In each case, we hypothesized that if someone has had never used a given concept before, they would be more likely to use that concept after encountering it while remixing an existing project. Using this second approach, we found that users who had never used a concept were more likely to do so if they had been exposed to the concept through remixing. Although some concepts were more widely used than others, we found a positive relationship between concept use and exposure through remixing for each of the six concepts. We found that this relationship was true even if we ignored obvious examples of cutting and pasting of blocks of code. In all of these models, we found what we believe is evidence of learning through remixing. Of course, there are many limitations in this work. What we found are all positive correlations we do not know if these relationships are causal. Moreover, our measures do not really tell us whether someone has understood the usage of a given block or programming concept.However, even with these limitations, we are excited by the results of our work, and we plan to build on what we have. Our next steps include developing and utilizing better measures of learning, as well as looking at other methods of appropriation like viewing the source code of a project.
This blog post and the paper it describes are collaborative work with Sayamindu Dasgupta, Andr s Monroy-Hern ndez, and William Hale. The paper is released as open access so anyone can read the entire paper here. This blog post was also posted on Sayamindu Dasgupta s blog and on Medium by the MIT Media Lab.