Welcome to the 36th post of the really randomly reverberating R, or
R4 for short, write-ups. Today s post is about using
Redis, and especially
RcppRedis, for live or (near) real-time monitoring with
R.
There is an saying that
you can take the boy out of the valley, but you cannot the valley out of the boy so for those of us who spent a decade or two in finance and on trading floors, having some market price information available becomes second nature. And/or sometimes it is just good fun to program this.
A good while back
Josh posted a
gist on a simple-yet-robust while loop. It (very cleverly) uses his
quantmod package to access the SP500 in real-time . (I use quotes here because at the end of retail broadband one is not at the same market action as someone co-located in a New Jersey data center. It is however not delayed: as an index, it is not immediately tradeable as a stock, etf, or derivative may be all of which are only disseminated as delayed price information, usually by ten minutes.) I quite enjoyed the gist and used it and started tinkering with it. For example, it collects data but only saves (
i.e. persists ) it after market close. If for whatever reason one needs to restart recent history is gone. In any event, I used his code and generalized it a little and published this about a year ago as function
intradayMarketMonitor() in my
dang package. (See
this blog post announcing it.) The chart of the left shows this in action, the chart is a snapshot from a couple of days ago when the vignettes (more on them below) were written.
As lovely as
intradayMarketMonitor() is, it also limits itself to market hours. And sometimes you want to see, say, how the market opens on Sunday (futures usually restart at 17h Chicago time), or how news dissipates during the night, or where markets are pre-open, or . So I both wanted to complement this with futures, and also cache it locally so that, say, one machine might collect data and one (or several others) can visualize. For such tasks,
Redis is unparalleled.
(Yet I also always felt
Redis could do with another, simple, short and sweet
introduction stressing the key features of i) being multi-lingual: write in one language, consume in another and ii) loose coupling: no linking as one talks to
Redis via standard tcp/ip networking. So I
wrote a new intro vignette that is now in RcppRedis. I hope this comes in handy. Comments welcome!)
Our
RcppRedis package had long been used for such tasks, and it was easy to set it up. Standard use is to loop, fetch some data, push it to
Redis, sleep, and start over. Clients do the same: fetch most recent data, plot or report it, sleep, start over. That works, but it has a dual delay as the client sleeping may miss the data update!
The standard answer to this is called publish/pubscribe, or pub/sub. Libraries such as
0mq or zeromq specialise in this. But it turns out
Redis already has it. I had some initial difficulty adding it to
RcppRedis so for a trial I tested the marvellous
rredis package by
Bryan and simply instantiated two
Redis clients. Now the data getter simply publishes a new data point in a given channel, by convention named after the security it tracks. Clients register with the
Redis server which does all the actual work of keeping track of who listens to what. The clients now simply listen (which is a blocking operation) and as soon as data comes in receive it.
This is quite mesmerizing when you just run two command-line clients (in a
byobu session, say). As sone as the data is written (as shown on console log) it is consumed. No measruable overhead. Just lovely.
Bryan and I then talked a litte as he may or may not retire
rredis. Having implemented the pub/sub logic for both sides once, he took a good hard look at
RcppRedis and just like that added it there. With some really clever wrinkles for (optional) per-symbol callback as closure attached to the instance. Truly amazeballs And once we had it in there, generalizing from publishing or subscribing to just one symbol
easily generalizes to having one listener collect
and publish for multiple symbols, and having one or more clients subscribe and listen one, more or even all symbol. All with ease thanks tp
Redis. The second chart, also from a few days ago, shows four symbols for four (front-contract) futures for Bitcoin, Crude Oil, SP500, and Gold.
As all this can get a little technical, I wrote a
second vignette for RcppRedis on just this: market monitoring. Give this a read, if interested, feedback on this one is most welcome too! But all the code you need is included in the package just run a local
Redis instance.
Before closing, one sour note. I uploaded all this in a new and much improved updated RcppRedis 0.2.0 to CRAN on March 13 ten days ago. Not only is it still not there , but CRAN in their most delightful way also refuses to answer any emails of mine. Just lovely. The package exhibited just one compiler warning: a C++ compiler objected to the (embedded) C library
hiredis (included as a fallback) for using a C language construct. Yes. A C++ compiler complaining about C. It s a non-issue. Yet it s been ten days and we still have nothing. So irritating and demotivating. Anyway, you can get the package off
its GitHub repo.
If you like this or other open-source work I do, you can
sponsor me at GitHub.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
In February I was assigned 16 hours of work by Freexian's Debian LTS
initiative and carried over 8 hours from January. I worked 16
hours, and will carry over the remaining time to March.
I spent most of my time triaging security issues for Linux, working
out which of them were fixed upstream and which actually applied to
the versions provided in Debian 9 "stretch". I also rebased the
Linux 4.9 (linux) package on the latest stable update, but did not
make an upload this month.
Testing Picolibc with the glibc tests
Picolibc has a bunch of built-in tests, but more testing is always
better, right? I decided to see how hard it would be to run some of
the tests provided in the GNU C Library (glibc).
Parallel meson build files
Similar to how Picolibc uses meson build files to avoid modifying the
newlib autotools infrastructure, I decided to take the glibc code and
write meson build rules that would compile the tests against Picolibc
header files and link against Picolibc libraries.
I decided to select a single target for this project so I could focus
on getting things building and not worry too much about making it
portable. I wanted to pick something that had hardware floating point
so that I would have rounding modes and exception support, so I picked
the ARM Cortex M7 with hard float ABI:
As many of you may be aware, I work with 
As of this morning, 
At the beginning of this year I updated a hundred of 
I recently learned about the 
This has ended up longer than I expected. I ll write up posts about some of the individual steps with some more details at some point, but this is an overview of the yak shaving I engaged in. The TL;DR is:
This provided an opportunity to see just what the RB3011 could actually manage. In the configuration I had it turned out to be not much more than the 80Mb/s speeds I had previously seen. The upload jumped from a solid 20Mb/s to 75Mb/s, so I knew the regrade had actually happened. Looking at CPU utilisation clearly showed the problem; softirqs were using almost 100% of a CPU core.
Now, the way the hardware is setup on the RB3011 is that there are two separate 5 port switches, each connected back to the CPU via a separate GigE interface. For various reasons I had everything on a single switch, which meant that all traffic was boomeranging in and out of the same CPU interface. The IPQ8064 has dual cores, so I thought I d try moving the external connection to the other switch. That puts it on its own GigE CPU interface, which then allows binding the interrupts to a different CPU core. That helps; throughput to the outside world hits 140Mb/s+. Still a long way from the expected max, but proof we just need more grunt.
Which brings us to this past weekend, when, having worked out all the other bits, I tried the squashfs root image again on the RB3011. Success! The home automation bits connected to it, the link to the outside world came up, everything seemed happy. So I double checked my bootloader bits on the RB5009, brought it down to the comms room and plugged it in instead. And, modulo my failing to update the nftables config to allow it to do forwarding, it all came up ok. Some testing with iperf3 internally got a nice 912Mb/s sustained between subnets, and some less scientific testing with wget + speedtest-cli saw speeds of over 460Mb/s to the outside world.
Time from ordering the router until it was in service? Just under 8 weeks