Welcome to the 36th post of the really randomly reverberating R, or
R4 for short, write-ups. Today s post is about using
Redis, and especially
RcppRedis, for live or (near) real-time monitoring with
R.
There is an saying that
you can take the boy out of the valley, but you cannot the valley out of the boy so for those of us who spent a decade or two in finance and on trading floors, having some market price information available becomes second nature. And/or sometimes it is just good fun to program this.
A good while back
Josh posted a
gist on a simple-yet-robust while loop. It (very cleverly) uses his
quantmod package to access the SP500 in real-time . (I use quotes here because at the end of retail broadband one is not at the same market action as someone co-located in a New Jersey data center. It is however not delayed: as an index, it is not immediately tradeable as a stock, etf, or derivative may be all of which are only disseminated as delayed price information, usually by ten minutes.) I quite enjoyed the gist and used it and started tinkering with it. For example, it collects data but only saves (
i.e. persists ) it after market close. If for whatever reason one needs to restart recent history is gone. In any event, I used his code and generalized it a little and published this about a year ago as function
intradayMarketMonitor() in my
dang package. (See
this blog post announcing it.) The chart of the left shows this in action, the chart is a snapshot from a couple of days ago when the vignettes (more on them below) were written.
As lovely as
intradayMarketMonitor() is, it also limits itself to market hours. And sometimes you want to see, say, how the market opens on Sunday (futures usually restart at 17h Chicago time), or how news dissipates during the night, or where markets are pre-open, or . So I both wanted to complement this with futures, and also cache it locally so that, say, one machine might collect data and one (or several others) can visualize. For such tasks,
Redis is unparalleled.
(Yet I also always felt
Redis could do with another, simple, short and sweet
introduction stressing the key features of i) being multi-lingual: write in one language, consume in another and ii) loose coupling: no linking as one talks to
Redis via standard tcp/ip networking. So I
wrote a new intro vignette that is now in RcppRedis. I hope this comes in handy. Comments welcome!)
Our
RcppRedis package had long been used for such tasks, and it was easy to set it up. Standard use is to loop, fetch some data, push it to
Redis, sleep, and start over. Clients do the same: fetch most recent data, plot or report it, sleep, start over. That works, but it has a dual delay as the client sleeping may miss the data update!
The standard answer to this is called publish/pubscribe, or pub/sub. Libraries such as
0mq or zeromq specialise in this. But it turns out
Redis already has it. I had some initial difficulty adding it to
RcppRedis so for a trial I tested the marvellous
rredis package by
Bryan and simply instantiated two
Redis clients. Now the data getter simply publishes a new data point in a given channel, by convention named after the security it tracks. Clients register with the
Redis server which does all the actual work of keeping track of who listens to what. The clients now simply listen (which is a blocking operation) and as soon as data comes in receive it.
This is quite mesmerizing when you just run two command-line clients (in a
byobu session, say). As sone as the data is written (as shown on console log) it is consumed. No measruable overhead. Just lovely.
Bryan and I then talked a litte as he may or may not retire
rredis. Having implemented the pub/sub logic for both sides once, he took a good hard look at
RcppRedis and just like that added it there. With some really clever wrinkles for (optional) per-symbol callback as closure attached to the instance. Truly amazeballs And once we had it in there, generalizing from publishing or subscribing to just one symbol
easily generalizes to having one listener collect
and publish for multiple symbols, and having one or more clients subscribe and listen one, more or even all symbol. All with ease thanks tp
Redis. The second chart, also from a few days ago, shows four symbols for four (front-contract) futures for Bitcoin, Crude Oil, SP500, and Gold.
As all this can get a little technical, I wrote a
second vignette for RcppRedis on just this: market monitoring. Give this a read, if interested, feedback on this one is most welcome too! But all the code you need is included in the package just run a local
Redis instance.
Before closing, one sour note. I uploaded all this in a new and much improved updated RcppRedis 0.2.0 to CRAN on March 13 ten days ago. Not only is it still not there , but CRAN in their most delightful way also refuses to answer any emails of mine. Just lovely. The package exhibited just one compiler warning: a C++ compiler objected to the (embedded) C library
hiredis (included as a fallback) for using a C language construct. Yes. A C++ compiler complaining about C. It s a non-issue. Yet it s been ten days and we still have nothing. So irritating and demotivating. Anyway, you can get the package off
its GitHub repo.
If you like this or other open-source work I do, you can
sponsor me at GitHub.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Sony RX100-III, relegated to a webcam
N/B: The form checks all architecture on the load of the page
merge request:
As of this morning, 
About a month later than I probably should have posted it, here s a recap of my Free Software activities in 2021. For previous years see