MJ Ray: GPG Transition Statement
Rather late but I guess that just confirms it s really me, right?
Rather late but I guess that just confirms it s really me, right?
Gammu 1.34.0 has been just released. It has been more than year from last Gammu release and it's time to release all new features and bugfixes.
Full list of changes:
Filed under: English Gammu Wammu 0 comments Flattr this!
In wikis, redirects are special pages that silently take readers from the page they are visiting to another page. Although their presence is noted in tiny gray text (see the image below) most people use them all the time and never know they exist. Redirects exist to make linking between pages easier, they populate Wikipedia s search autocomplete list, and are generally helpful in organizing information. In the English Wikipedia, redirects make up more than half of all article pages.
Over the years, I ve spent some time contributing to to Redirects for Discussion (RfD). I think of RfD as like an ultra-low stakes version of Articles for Deletion where Wikipedians decide whether to delete or keep articles. If a redirect is deleted, viewers are taken to a search results page and almost nobody notices. That said, because redirects are almost never viewed directly, almost nobody notices if a redirect is kept either!
I ve told people that if they want to understand the soul of a Wikipedian, they should spend time participating in RfD. When you understand why arguing about and working hard to come to consensus solutions for how Wikipedia should handle individual redirects is an enjoyable way to spend your spare time where any outcome is invisible you understand what it means to be a Wikipedian.
That said, wiki researchers rarely take redirects into account. For years, I ve suspected that accounting for redirects was important for Wikipedia research and that several classes of findings were noisy or misleading because most people haven t done so. As a result, I worked with my colleague Aaron Shaw at Northwestern earlier this year to build a longitudinal dataset of redirects that can capture the dynamic nature of redirects. Our work was published as a short paper at OpenSym several months ago.
It turns out, taking redirects into account correctly (especially if you are looking at activity over time) is tricky because redirects are stored as normal pages by MediaWiki except that they happen to start with special redirect text. Like other pages, redirects can be updated and changed over time are frequently are. As a result, taking redirects into account for any study that looks at activity over time requires looking at the text of every revision of every page.
Using our dataset, Aaron and I showed that the distribution of edits across pages in English Wikipedia (a relationships that is used in many research projects) looks pretty close to log normal when we remove redirects and very different when you don t. After all, half of articles are really just redirects and, and because they are just redirects, these articles are almost never edited.
Another puzzling finding that s been reported in a few places and that I repeated myself several times is that edits and views are surprisingly uncorrelated. I ll write more about this later but the short version is that we found that a big chunk of this can, in fact, be explained by considering redirects.
We ve published our code and data and the article itself is online because we paid the ACM s open access fee to ransom the article.
This is the time of year when there are lots of adverts shown on TV solicating donations for charities, which frequently end with the two words "thank you".
I've always felt there were too many charities in the world, and that it was hard to half-heartedly give money to one charity this month, one the next, and still another next year. On that basis I decided long ago to give my money solely to three charities. If I had money that was spare, or I felt generous that month, I would give it to one of "my" charities. Any other appeals I just ignored (with minor exceptions for one-off events like tsunamis, etc).
I won't claim credit for this idea, it came directly from my mom who does the same thing. I've given money to the same three charities for twenty years now. Maybe not thousands, but hopefully enough to be useful. Certainly more than I'd have given if my donation were split between more recipients.
Now I'm changing. As of next year only one charitable organization will get my pennies. The other two haven't done anything bad, wrong, or failed/succeeded (sadly), but it feels better for me to stick to a single recipient.
Happy Christmas.
(Details shouldn't matter, but to answer the obvious question the charity I've kept is the RNLI.)
One way of working on reproducible
builds is to look at packages
which fail to build
reproducibly
on our continuous integration
platform. Looking at the output of
debbindiff often makes it possible to
spot common
problems.
Sometimes, issues are more singular.
sources.debian.net and
codesearch are amazing helps to skim over the
source code trying to figure out what is happening.
When I started to work on reproducible builds for Debian a year and a half ago,
I never thought it would be useful to find actual release critical bugs.
Until I had a look at libical debbindiff
output.
Can you understand the issue and how bad it is?
Answer is in bug #773916.
A bit of news on the development of OpenTAC the Open Hardware Test Automation Controller. I ve talked about this at the MiniDebConf 2014 in Cambridge. (Video available). The development is being tracked on the Vero-Apparatus wiki and as this is Open Hardware, the files are attached to the wiki (All files are CC BY-SA 4.0).
Andy completed the schematics at the start of November, allowing work to start on the routing. With routing completed, orders were placed for manufacture of the first PCBs on the 19th December 2014. Whilst waiting for the PCBs to arrive we re working on the device tree database (my first real experience with creating a device tree) which will underpin the PDU and serial console services available to the user as well as the admin interface for control of the USB subsystems, fan control, power control and thermal monitoring. The first thing we need to do is create a data dictionary a table to correlate software identifiers with the real hardware pins. We ll then follow that through with a default device tree overlay that will leave all the associated I/O lines in a safe initial state.
Once we have some code, I ll be pushing to a branch on GitHub. We ve also got an internal git repo on vero-apparatus for OpenTAC files which we will add to in due course.
Image of the board as rendered prior to prototype production:
More to come once we have the hardware on the desk
It seems that the 0.37 release was not that good as I hoped for, so here comes another bugfix release. So here comes Wammu 0.38.
The list of changes is not really huge:
Filed under: English Gammu Wammu 0 comments Flattr this!
A new version of rfoaas is now on CRAN. The rfoaas package provides an interface for R to the most excellent FOAAS service--which provides a modern, scalable and RESTful web service for the frequent need to tell someone to eff off.
This version aligns the rfoaas version number with the (at long last) updated upstream version number, and brings a change suggested by Richie Cotton to set the encoding of the returned object.
As usual, CRANberries provides a diff to the previous release. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Conrad produced another minor release 4.600 of Armadillo. As before, I had created a GitHub-only pre-release(s) of his pre-release(s), and tested a pre-release as well as the actual release against the now over one hundred CRAN dependents of our RcppArmadillo package. The tests passed fine as usual with less than a handful of checks not passing, all for known cases -- and results are as always in the rcpp-logs repository.
Changes are summarized below based on the NEWS.Rd file.
Changes in RcppArmadillo version 0.4.600.0 (2014-12-27)Courtesy of CRANberries, there is also a diffstat report for the most recent release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
- Upgraded to Armadillo release Version 4.600 ("Off The Reservation")
- added
.head()and.tail()to submatrix views- faster matrix transposes within compound expressions
- faster
accu()andnorm()when compiling with -O3 -ffast-math -march=native (gcc and clang)- workaround for a bug in GCC 4.4
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
kdb qt-gui. It is very important to note that the gui is in version 0.0.2, meaning it is an alpha project and shouldn t be considered stable yet! This GUI is a convenient tool that lets you mount, unmount, search, import and export configurations. We are very excited about this tool and we hope it makes Elektra more accessible to all users. Check out the screenshot below to see this new GUI in action:
A few other new features are that Felix added support for multi-line into the ini plugin. Users are able to turn this feature on or off for each configuration. Additionally, Kai-Uwe added support for Windows 7 to Elektra! The Windows 7 port relies on a new resolver called wresolver which is a simpler version of the regular resolver which is compatible with Windows. Additionally, there is a new Java binding, so now Elektra is compatible with applications and plugins written in Java! On top of these features, a new hosts plugin and a new rename plugin have been written. The new hosts in conjunction with improvements to the ini plugin allows for multiple styles of comments (because some ini dialects allow ; and # to be used to differentiate comments) to be used and preserved in configuration files. Additionally, this new hosts plugin is compatible with and differentiates between ipv4 and ipv6 entries.
There have also been a huge number of under-the-hood improvements, changes, and fixes; there have been so many in-fact that I am going to list them:
regexstore was added.add_plugin function which allows plugin developers to more easily and consistently add plugins when building Elektra.kdb mount-openicckdb list-tools which lists the available kdb tools.- prefix such as DPLUGINS=ALL;-xmltool" which would exclude xmltool.| Publisher: | William Morrow |
| Copyright: | June 2013 |
| ISBN: | 0-06-202444-2 |
| Format: | Trade paperback |
| Pages: | 336 |
A friend recently commented on my rather unusual portrait on my (out of date) page on the Berkman website. Here s the story.
I joined Berkman as a fellow with a fantastic class of fellows that included, among many other incredibly accomplished people, Vivek Kundra: first Chief Information Officer of the United States. At Berkman, all the fellows are all asked for photos and Vivek apparently sent in his official government portrait.
You are probably familiar with the genre. In the US at least, official government portraits are mostly pictures of men in dark suits, light shirts, and red or blue ties with flags draped blurrily in the background.
Not unaware of the fact that Vivek sat right below me on the alphabetically sorted Berkman fellows page, a small group that included Paul Tagliamonte very familiar with the genre from his work with government photos in Open States decided to create a government portrait of me using the only flag we had on hand late one night.
The result shown in the screenshot above and in the WayBack Machine was almost entirely unnoticed (at least to my knowledge) but was hopefully appreciated by those who did see it.
| Week | Squeeze | Wheezy | Jessie |
|---|---|---|---|
| 43 | 284 (213+71) | 468 (332+136) | 319 (240+79) |
| 44 | 261 (201+60) | 408 (265+143) | 274 (224+50) |
| 45 | 261 (205+56) | 425 (291+134) | 295 (229+66) |
| 46 | 271 (200+71) | 401 (258+143) | 427 (313+114) |
| 47 | 283 (209+74) | 366 (221+145) | 342 (260+82) |
| 48 | 256 (177+79) | 378 (230+148) | 274 (189+85) |
| 49 | 256 (180+76) | 360 (216+155) | 226 (147+79) |
| 50 | 204 (148+56) | 339 (195+144) | ??? |
| 51 | 178 (124+54) | 323 (190+133) | 189 (134+55) |
| 52 | 115 (78+37) | 289 (190+99) | 147 ((112+35)) |
| 1 | 93 (60+33) | 287 (171+116) | |
| 2 | 82 (46+36) | 271 (162+109) | |
| 3 | 25 (15+10) | 249 (165+84) | |
| 4 | 14 (8+6) | 244 (176+68) | |
| 5 | 2 (0+2) | 224 (132+92) | |
| 6 | release! | 212 (129+83) | |
| 7 | release+1 | 194 (128+66) | |
| 8 | release+2 | 206 (144+62) | |
| 9 | release+3 | 174 (105+69) | |
| 10 | release+4 | 120 (72+48) | |
| 11 | release+5 | 115 (74+41) | |
| 12 | release+6 | 93 (47+46) | |
| 13 | release+7 | 50 (24+26) | |
| 14 | release+8 | 51 (32+19) | |
| 15 | release+9 | 39 (32+7) | |
| 16 | release+10 | 20 (12+8) | |
| 17 | release+11 | 24 (19+5) | |
| 18 | release+12 | 2 (2+0) |
I have been hard at work on the
shell-monad
ever since it was born on Christmas Eve. It's now up to 820 lines of code,
and has nearly comprehensive coverage of all shell features.
Time to use it for something interesting! Let's make a shell script and a
haskell program that both speak a simple protocol. This kind of thing could
be used by propellor when it's deploying itself to a new host. The
haskell program can ssh to a remote host and run the shell program, and
talk back and forth over stdio with it, using the protocol they both speak.
abstract beginnings
First, we'll write a data type for the commands in the protocol.
data Proto
= Foo String
Bar
Baz Integer
deriving (Show)
Now, let's go type class crazy!
class Monad t => OutputsProto t where
output :: Proto -> t ()
instance OutputsProto IO where
output = putStrLn . fromProto
So far, nothing interesting; this makes the IO monad an instance of the
OutputsProto type class, and gives a simple implementation to output a
line of the protocol.
instance OutputsProto Script where
output = cmd "echo" . fromProto
Now it gets interesting. The Script monad is now also a member of the
OutputsProto. To output a line of the protocol, it just uses echo.
Yeah -- shell code is a member of a haskell type class. Awesome -- most
abstract shell code evar!
Similarly, we can add another type class for monads that can input the
protocol:
class Monad t => InputsProto t p where
input :: t p
instance InputsProto IO Proto where
input = toProto <$> readLn
instance InputsProto Script Var where
input = do
v <- newVar ()
readVar v
return v
While the IO version reads and deserializes a line back to a Proto,
the shell script version of this returns a Var, which has the newly
read line in it, not yet deserialized. Why the difference? Well, Haskell
has data types, and shell does not ...
speaking the protocol
Now we have enough groundwork to write haskell code in the IO monad that
speaks the protocol in arbitrary ways. For example:
protoExchangeIO :: Proto -> IO Proto
protoExchangeIO p = do
output p
input
fooIO :: IO ()
fooIO = do
resp <- protoExchangeIO (Foo "starting up")
-- etc
But that's trivial and uninteresting. Anyone who has read to here certianly
knows how to write haskell code in the IO monad. The interesting part is
making the shell program speak the protocol, including doing various things
when it receives the commands.
foo :: Script ()
foo = do
stopOnFailure True
handler <- func (NamedLike "handler") $
handleProto =<< input
output (Foo "starting up")
handler
output Bar
handler
handleFoo :: Var -> Script ()
handleFoo v = toStderr $ cmd "echo" "yay, I got a Foo" v
handleBar :: Script ()
handleBar = toStderr $ cmd "echo" "yay, I got a Bar"
handleBaz :: Var -> Script ()
handleBaz num = forCmd (cmd "seq" (Val (1 :: Int)) num) $
toStderr . cmd "echo" "yay, I got a Baz"
serialization
I've left out a few serialization functions. fromProto is used in both
instances of OutputsProto. The haskell program and the script will both use
this to serialize Proto.
fromProto :: Proto -> String
fromProto (Foo s) = pFOO ++ " " ++ s
fromProto Bar = pBAR ++ " "
fromProto (Baz i) = pBAZ ++ " " ++ show i
pFOO, pBAR, pBAZ :: String
(pFOO, pBAR, pBAZ) = ("FOO", "BAR", "BAZ")
And here's the haskell function to convert the other direction, which was also
used earlier.
toProto :: String -> Proto
toProto s = case break (== ' ') s of
(w, ' ':rest)
w == pFOO -> Foo rest
w == pBAR && null rest -> Bar
w == pBAZ -> Baz (read rest)
otherwise -> error $ "unknown protocol command: " ++ w
(_, _) -> error "protocol splitting error"
We also need a version of that written in the Script monad. Here it is.
Compare and contrast the function below with the one above. They're
really quite similar. (Sadly, not so similar to allow refactoring out a common
function..)
handleProto :: Var -> Script ()
handleProto v = do
w <- getProtoCommand v
let rest = getProtoRest v
caseOf w
[ (quote (T.pack pFOO), handleFoo =<< rest)
, (quote (T.pack pBAR), handleBar)
, (quote (T.pack pBAZ), handleBaz =<< rest)
, (glob "*", do
toStderr $ cmd "echo" "unknown protocol command" w
cmd "false"
)
]
Both toProto and handleProto split the incoming line apart into the
first word, and the rest of the line, then match the first word against the
commands in the protocol, and dispatches to appropriate actions. So,
how do we split a variable apart like that in the Shell monad? Like this...
getProtoCommand :: Var -> Script Var
getProtoCommand v = trimVar LongestMatch FromEnd v (glob " *")
getProtoRest :: Var -> Script Var
getProtoRest v = trimVar ShortestMatch FromBeginning v (glob "[! ]*[ ]")
(This could probably be improved by using a DSL to generate the globs
too..)
conclusion
And finally, here's a main to generate the shell script!
main :: IO ()
main = T.writeFile "protocol.sh" $ script foo
The pretty-printed shell script that produces is not very interesting,
but I'll include it at the end for completeness. More interestingly for the
purposes of sshing to a host and running the command there, we can use
linearScript to generate a version of the script that's all contained on
a single line. Also included below.
I could easily have written the pretty-printed version of the shell script
in twice the time that it took to write the haskell program that generates
it and also speaks the protocol itself.
I would certianly have had to test the hand-written shell script repeatedly.
Code like for _x in $(seq 1 "$ _v#[!\ ]*[\ ] ") doesn't just write and
debug itself. (Until now!)
But, the generated scrpt worked 100% on the first try! Well, it worked as
soon as I got the Haskell program to compile...
But the best part is that the Haskell program and the shell script
don't just speak the same protocol. They both rely on the same
definition of Proto. So this is fairly close to the kind of type-safe
protocol serialization that Fay provides, when compiling Haskell to
javascript.
I'm getting the feeling that I won't be writing too many nontrivial shell
scripts by hand anymore! :)
the complete haskell program
Is here,
all 99 lines of it.
the pretty-printed shell program
#!/bin/sh
set -x
_handler () :
_v=
read _v
case "$ _v%%\ * " in FOO) :
echo 'yay, I got a Foo' "$ _v#[!\ ]*[\ ] " >&2
: ;; BAR) :
echo 'yay, I got a Bar' >&2
: ;; BAZ) :
for _x in $(seq 1 "$ _v#[!\ ]*[\ ] ")
do :
echo 'yay, I got a Baz' "$_x" >&2
done
: ;; *) :
echo 'unknown protocol command' "$ _v%%\ * " >&2
false
: ;; esac
echo 'FOO starting up'
_handler
echo 'BAR '
_handler
the one-liner shell program
set -p; _handler () :; _v=; read _v; case "$ _v%%\ * " in FOO) :; echo 'yay, I got a Foo' "$ _v#[!\ ]*[\ ] " >&2; : ;; BAR) :; echo 'yay, I got a Bar' >&2; : ;; BAZ) :; for _x in $(seq 1 "$ _v#[!\ ]*[\ ] "); do :; echo 'yay, I got a Baz' "$_x" >&2; done; : ;; *) :; echo 'unknown protocol command' "$ _v%%\ * " >&2; false; : ;; esac; ; echo 'FOO starting up'; _handler; echo 'BAR '; _handler
for x in $(ps -eo pid h); do s=/proc/$ x /smaps; [ -e $ s ] && awk -vp=$ x 'BEGIN sum=0 /Swap:/ sum+=$2 END if (sum!=0) print sum " PID " p ' $ s ; done sort -rg
After I shared this one with some friends V. came up with a faster
version (and properly formatted output :), that relies on the "VMSwap"
value in /proc/[pid]/status. Since the one-liner above has to add up one
"Swap" value per memory segment it's obvious why it's very slow on
systems with many processes and a lot of memory.
awk 'BEGIN printf "%-7s %-16s %s (KB)\n", "PID","COMM","VMSWAP"
if($1 == "Name:") n=$2
if($1 == "Pid:") p=$2
if($1 == "VmSwap:" && $2 != "0") printf "%-7s %-16s %s\n", p,n,$2 "sort -nrk3"
' /proc/*/status
Drawback of this second version is that it relies on the "VmSwap" value
which is only part of Linux
2.6.34
and later. It also ended up in the 2.6.32 based kernel of RHEL 6, so
this one should work from RHEL 6 and Debian/wheezy onwards.
The first version also works on RHEL 5 and Debian/squeeze
since both have a (default) kernel with "CONFIG_PROC_PAGE_MONITOR" enabled,
which is what you need to enable /proc/[pid]/smaps.
Update: C.R. pointed out via mail that there is a versatile Python script
smem. It's available in Debian
and at least EPEL 6 and 7. Thanks for the hint.
The usefulness of swap and why there is no automatic ressource assignment
check and bookkeeping, based on calculations around things like the JVM -Xmx
settings and other input, is a story on its own. There is room for improvement.
A CMDB (Chapter 10.7) would be a starting
point.
rrs@learner:~$ sudo cat /var/lib/lxc/deb-template/config
[sudo] password for rrs:
# Template used to create this container: /usr/share/lxc/templates/lxc-debian
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)
# CPU
lxc.cgroup.cpuset.cpus = 0,1
lxc.cgroup.cpu.shares = 1234
# Mem
lxc.cgroup.memory.limit_in_bytes = 2000M
lxc.cgroup.memory.soft_limit_in_bytes = 1500M
# Network
lxc.network.type = veth
lxc.network.hwaddr = 00:16:3e:0c:c5:d4
lxc.network.flags = up
lxc.network.link = lxcbr0
# Root file system
lxc.rootfs = /var/lib/lxc/deb-template/rootfs
# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf
# Container specific configuration
lxc.mount = /var/lib/lxc/deb-template/fstab
lxc.utsname = deb-template
lxc.arch = amd64
# For apt
lxc.mount.entry = /var/cache/apt/archives var/cache/apt/archives none defaults,bind 0 0
23:07
rrs@learner:~$Some of the important settings to have in the templace are the mount point, to point to your local apt cache, and CPU and Memory limits.
If there was one feature request to ask the LXC developers, I'd ask them to provide a util-lxc tools suite. Currently, to know the memory (soft/hard) allocation for the container, one needs to do the following:
rrs@learner:/sys/fs/cgroup/memory/lxc/deb-template$ cat memory.soft_limit_in_bytes memory.limit_in_bytes
1572864000
2097152000
23:21
rrs@learner:/sys/fs/cgroup/memory/lxc/deb-template$ bc
bc 1.06.95
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type warranty'.
1572864000/1024/1024
1500
quit
23:21
rrs@learner:/sys/fs/cgroup/memory/lxc/deb-template$
Tools like lxc-cpuinfo, lxc-free would be much better.
Finally, there's been a lot of buzz about Docker. Docker is an alternate product offering, like LXC, for Linux Containers. From what I have briefly looked at, docker doesn't seem to be providing any ground breaking new interface than what is already possible with LXC. It does take all the tidbit tools, and presents you with a unified docker interface. But other than that, I couldn't find it much appealing. And the assumption that the profiles should be pulled off the internet (Github ?) is not very exciting. I am hoping they do have other options, where dependence on the network is not really required.
It Will Never Work in Theory is a web
site that blogs, though slowly, of important research and findings
about software development. It's one of the most interesting sites
I've found recently, possibly for a long time.
I disagree with the term "software engineering" to describe the
software development that happens today. I don't think it's accurate,
and indeed I think the concept's too much of a fantasy for the term to
be used seriously about practicing developers do. For software
development to be an engineering discipline, it needs a strong
foundation based on actual research. In short, we need to know what
works, what doesn't work, and preferably why in both cases. We don't
have much of that.
This website is one example of how that's now changing, and that's
good. As a practicing software developer, I want to know, for example,
whether code review actually helps improve software quality, the speed
of software development, and the total cost of a software project, and
also under what the limits of code review are, how it should be done
well, and what kind of review doesn't work. Once I know that, I can
decide whether and how to do reviews in my development teams.
The software development field is full of anecdotal evidence about
these things. It's also full of people who've done something once, and
then want to sell books, seminars, and lectures about it. That's not
been working too well: it makes research be mostly about fads, and
that's no way to build a strong foundation.
Now I just need the time to read everything, and the brain to
understand big words.
Next.