Mike Hommey: Why is the git-cinnabar master branch slower to clone?
Apart from the memory considerations, one thing that the data presented in the When the memory allocator works against you post that I haven t touched in the followup posts is that there is a large difference in the time it takes to clone mozilla-central with git-cinnabar 0.4.0 vs. the master branch.
One thing that was mentioned in the first followup is that reducing the amount of
On master:
(Note: the data gathering is flawed in some ways, which explains why the
realloc and substring copies made the cloning more than 15 minutes faster on master. But the same code exists in 0.4.0, so this isn t part of the difference.
So what s going on? Looking at the CPU usage during the clone is enlightening.
On 0.4.0:
On master:
(Note: the data gathering is flawed in some ways, which explains why the git-remote-hg process goes above 100%, which is not possible for this python process. The data is however good enough for the high level analysis that follows, so I didn t bother to get something more acurate)
On 0.4.0, the git-cinnabar-helper process was saturating one CPU core during the File import phase, and the git-remote-hg process was saturating one CPU core during the Manifest import phase. Overall, the sum of both processes usually used more than one and a half core.
On master, however, the total of both processes barely uses more than one CPU core.
What happened?
This and that happened.
Essentially, before those changes, git-remote-hg would send instructions to git-fast-import (technically, git-cinnabar-helper, but in this case it s only used as a wrapper for git-fast-import), and use marks to track the git objects that git-fast-import created.
After those changes, git-remote-hg asks git-fast-import the git object SHA1 of objects it just asked to be created. In other words, those changes replaced something asynchronous with something synchronous: while it used to be possible for git-remote-hg to work on the next file/manifest/changeset while git-fast-import was working on the previous one, it now waits.
The changes helped simplify the python code, but made the overall clone process much slower.
If I m not mistaken, the only real use for that information is for the mapping of mercurial to git SHA1s, which is actually rarely used during the clone, except at the end, when storing it. So what I m planning to do is to move that mapping to the git-cinnabar-helper process, which, incidentally, will kill not 2, but 3 birds with 1 stone:
- It will restore the asynchronicity, obviously (at least, that s the expected main outcome).
- Storing the mapping in the
git-cinnabar-helperprocess is very likely to take less memory than what it currently takes in thegit-remote-hgprocess. Even if it doesn t (which I doubt), that should still help stay under the 2GB limit of 32-bit processes. - The whole thing that spikes memory usage during the finalization phase, as seen in previous post, will just go away, because the
git-cinnabar-helperprocess will just have prepared the git notes-like tree on its own.
systemd 233 is scheduled to be released next week, and there is only
... but that doesn't mean the work is over.
One big job that needs to happen after the conference is to review and
release the video recordings that were made. With several hundreds of
videos to be checked and only a handful of people with the ability to do
so, review was a massive job that for the past three editions took
several months; e.g., in 2016 the last video work was done in July, when
the preparation of the 2017 edition had already started.
Obviously this is suboptimal, and therefore another solution was
required. After working on it for quite a while (in my spare time), I
came up with
Today,
So that was 2016! Here s a summary of what I got up to on my computer(s) in December, a check of how I went against my plan, and the TODO list for the next month or so.
With a short holiday to Oslo, Christmas holidays, Christmas parties (at work and with Alexander at school, football etc.), travelling to Brussels with work, birthdays (Alexander & Antje), I missed a lot of deadlines, and failed to reach most of my Free Software goals (including my goals for new & updated packages in Debian Stretch the soft freeze is in a couple of days). To top it all off, I lost my grandmother at the ripe old age of 93. Rest in peace Nana. I wish I could have made it to the funeral, but it is sometimes tough living on the other side of the world to your family.
Debian
Here is my monthly update covering what I have been doing in the free software world (