Demonstration of a Lapdance Wikipedia
Illustration of an open jeep, sadly its a military one wikipedia
We needed a router and wifi access point in the office, and
simultaneously both I and my co-worker Ivan needed such a thing at our
respective homes. After some discussion, and after reading articles in
Ars Technica about building PCs to act as routers, we decided to do
just that.
| Component | Model | Cost |
|---|---|---|
| Barebone | Qotom Q190G4, VGA, 2x USB 2.0, 134x126x36mm, fanless | 130 |
| CPU | Intel J1900, 2-2.4GHz quad-core | - |
| NIC | Intel WG82583, 4x 10/100/1000 | - |
| Memory | Crucial CT102464BF160B, 8GB DDR3L-1600 SODIMM 1.35V CL11 | 40 |
| SSD | Kingston SSDNow mS200, 60GB mSATA | 42 |
| WLAN | AzureWave AW-NU706H, Ralink RT3070L, 300M 802.11b/g/n, half mPCIe | 17 |
| mPCIe adapter | Half to full mPCIe adapter | 3 |
| Antennas | 2x 2.4/5GHz 6dBi, RP-SMA, U.FL Cables | 7 |
/home/ansible/.ssh/authorized_keys file
with one that contains only your public ssh key.ansible@10.0.0.4, and verify that sudo id
works without password. Except you can't do this, unless you put in
your ssh key in the authorized keys file above.minipc-router.yml as wanted, and run the playbook. Then reboot the
router again.roles/router/files/ferm.conf as
you wish.ansible
user's authorized keys file for ssh. This is because we built this
for ourselves first. If there's interest by others in using the
images, we'll solve this.
So I had been toying with the idea of getting back to the blog and more regularly writing / posting little tips and tricks. I even started taking some notes but because perfect is always the enemy of the good it never quite materialized.
But the relatively broad discussion spawned by last week's short rant on Suggests != Depends made a few things clear. There appears to be an audience. It doesn't have to be long. And it doesn't have to be too polished.
So with that, let's get the blogging back from micro-blogging.
This note forms post zero of what will a new segment I call R4 which is shorthand for relatively random R rambling.
Stay tuned.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
http://social.gl-como.it/photos/valhalla/image/81b600789aa02a91fdf62f54a71b1ba0
Dear lazyweb,
I am trying to get a good way to present the categorization of several cases studied with a fitting graph. I am rating several vulnerabilities / failures according to James Cebula et. al.'s paper, A taxonomy of Operational Cyber Security Risks; this is a somewhat deep taxonomy, with 57 end items, but organized in a three levels deep hierarchy. Copying a table from the cited paper (click to display it full-sized):
My categorization is binary: I care only whether it falls within a given category or not. My first stab at this was to represent each case using a star or radar graph. As an example:
As you can see, to a "bare" star graph, I added a background color for each top-level category (blue for actions of people, green for systems and technology failures), red for failed internal processes and gray for external events), and printed out only the labels for the second level categories; for an accurate reading of the graphs, you have to refer to the table and count bars. And, yes, according to the Engineering Statistics Handbook:
Star plots are helpful for small-to-moderate-sized multivariate data sets. Their primary weakness is that their effectiveness is limited to data sets with less than a few hundred points. After that, they tend to be overwhelming.I strongly agree with the above statement And stating that "a few hundred points" can be understood is even an overstatement. 50 points are just too much. Now, trying to increase usability for this graph, I came across the Sunburst diagram. One of the proponents for this diagram, John Stasko, has written quite a bit about it. Now... How to create my beautiful Sunburst diagram? That's a tougher one. Even though the page I linked to in the (great!) Data visualization catalogue presents even some free-as-in-software tools to do this... They are Javascript projects that will render their beautiful plots (even including an animation)... To the browser. I need them for a static (i.e. to be printed) document. Yes, I can screenshot and all, but I want them to be automatically generated, so I can review and regenerate them all automatically. Oh, I could just write JSON and use SaaS sites such as Aculocity to do the heavy-lifting, but if you know me, you will understand why I don't want to. So... I set out to find a Gunnar-approved way to display the information I need. Now, as the Protovis documentation says, an icicle is simply a sunburst transformed from polar to cartesian coordinates... But I came to a similar conclusion: The tools I found are not what I need. OK, but an icicle graph seems much simpler to produce I fired up my Emacs, and started writing using Ruby, RMagick and RVG... I decided to try a different way. This is my result so far:
So... What do you think? Does this look right to you? Clearer than the previous one? Worst? Do you have any idea on how I could make this better?
Oh... You want to tell me there is something odd about it? Well, yes, of course! I still need to tweak it quite a bit. Would you believe me if I told you this is not really a left-to-right icicle graph, but rather a strangely formatted Graphviz non-directed graph using the dot formatter?
I can assure you you don't want to look at my Graphviz sources... But in case you insist... Take them and laugh. Or cry. Of course, this file comes from a hand-crafted template, but has some autogenerated bits to it. I have still to tweak it quite a bit to correct several of its usability shortcomings, but at least it looks somewhat like what I want to achieve.
Anyway, I started out by making a "dear lazyweb" question. So, here it goes: Do you think I'm using the right visualization for my data? Do you have any better suggestions, either of a graph or of a graph-generating tool?
Thanks!
[update] Thanks for the first pointer, Lazyweb! I found a beautiful solution; we will see if it is what I need or not (it is too space-greedy to be readable... But I will check it out more thoroughly). It lays out much better than anything I can spew out by myself Writing it as a mindmap using TikZ directly from within LaTeX, I get the following result:
$ cat > hello.c
#include <stdio.h>
int main(void)
printf("Hello, world!\n");
$ gcc-5 hello.c -o hello-5
$ md5sum hello-5
a00416d7392442321bad4afc5a461321 hello-5
$ gcc-5 hello.c -o hello-5
$ md5sum hello-5
a00416d7392442321bad4afc5a461321 hello-5
$ gcc-6 hello.c -o hello-6
$ md5sum hello-6
f7f52c2f5f82fe2a95061a771a6c5acd hello-6
$ hexcompare hello-5 hello-6
[lots of red]
...
$ gcc-6 hello.c -lc -o hello-6
$ gcc-6 -lc hello.c -o hello-6b
$ md5sum hello-6 hello-6b
f7f52c2f5f82fe2a95061a771a6c5acd hello-6
f73ee6d8c3789fd8f899f5762025420e hello-6b
$ hexcompare hello-6 hello-6b
[lots of red]
...
$ cd ..
$ cp -a repro/ repro2/
$ cd repro2/
$ gcc-6 hello.c -o hello-6
$ md5sum hello-6
f7f52c2f5f82fe2a95061a771a6c5acd hello-6
$ md5sum freedink/native/src/freedink freedink2/native/src/freedink
839ccd9180c72343e23e5d9e2e65e237 freedink/native/src/freedink
6d5dc6aab321fab01b424ac44c568dcf freedink2/native/src/freedink
$ hexcompare freedink2/native/src/freedink freedink/native/src/freedink
[lots of red]
$ strip freedink/native/src/freedink freedink2/native/src/freedink
$ md5sum freedink/native/src/freedink freedink2/native/src/freedink
415e96bb54456f3f2a759f404f18c711 freedink/native/src/freedink
e0702d798807c83d21f728106c9261ad freedink2/native/src/freedink
$ hexcompare freedink/native/src/freedink freedink2/native/src/freedink
[1 single red spot]
$ diffoscope freedink/native/src/freedink freedink2/native/src/freedink
--- freedink/native/src/freedink
+++ freedink2/native/src/freedink
readelf --wide --notes
@@ -3,8 +3,8 @@
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 2.6.32
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
- Build ID: a689574d69072bb64b28ffb82547e126284713fa
+ Build ID: d7be191a61e84648a58c18e9c108b3f3ce500302
$ (cd repro/ && gcc -g -O2 hello.c -o hello)
$ (cd repro/ && gcc -g -O2 hello.c -o hello-b)
$ md5sum repro/hello repro/hello-b
6b2cd79947d7c5ed2e505ddfce167116 repro/hello
6b2cd79947d7c5ed2e505ddfce167116 repro/hello-b
# => OK for now
$ (cd repro2/ && gcc -g -O2 hello.c -o hello)
$ md5sum repro2/hello
20b4d09d94de5840400be05bc76e4172 repro2/hello
$ strip repro/hello repro2/hello
$ diffoscope repro/hello repro2/hello
--- repro/hello
+++ repro2/hello2
readelf --wide --notes
@@ -3,8 +3,8 @@
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 2.6.32
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
- Build ID: 462a3c613537bb57f20bd3ccbe6b7f6d2bdc72ba
+ Build ID: b4b448cf93e7b541ad995075d2b688ef296bd88b
# => issue reproduced with -g -O2 and different build directories
$ (cd repro/ && gcc -O2 hello.c -o hello)
$ (cd repro2/ && gcc -O2 hello.c -o hello)
$ md5sum repro/hello repro2/hello
1571d45eb5807f7a074210be17caa87b repro/hello
1571d45eb5807f7a074210be17caa87b repro2/hello
# => culprit is not -O2, so culprit is -g
$ /opt/mxe/usr/bin/i686-w64-mingw32.static-gcc hello.c -o hello.exe
$ md5sum hello.exe
e0fa685f6866029b8e03f9f2837dc263 hello.exe
$ /opt/mxe/usr/bin/i686-w64-mingw32.static-gcc hello.c -o hello.exe
$ md5sum hello.exe
df7566c0ac93ea4a0b53f4af83d7fbc9 hello.exe
$ /opt/mxe/usr/bin/i686-w64-mingw32.static-gcc hello.c -o hello.exe
$ md5sum hello.exe
bbf4ab22cbe2df1ddc21d6203e506eb5 hello.exe
There are 7 different phases happening during a git clone using git-cinnabar, most of which can easily be identified on the graph above:
getbundle request is emitted, which response is read in the next three phases. This phase is essentially invisible on the graph.
getbundle request is changesets. They are sent in the RevChunk format. Translated to git, they become commit objects. But to create commit objects, we need the entire corresponding trees and files (blobs), which we don t have yet. So we keep this data in memory.
In the git clone analyzed here, there are 345643 changesets loaded in memory. Their raw size in RawChunk format is 237MB. I think by the end of this phase, we made 20 million allocator calls, have about 300MB of live data in about 840k allocations. (No certainty because I don t actually have definite data that would allow to correlate between the phases and allocator calls, and the memory usage change between this phase and next is not as clear-cut as with other phases). This puts us at less than 3 live allocations per changeset, with only about 60MB overhead over the raw data.
git clone analyzed here, there are 345398 manifests loaded in memory. Their raw size in RawChunk format is 1.18GB. By the end of this phase, we made 23 million more allocator calls, and have about 1.52GB of live data in about 1.86M allocations. We re still at less than 3 live allocations for each object (changeset or manifest) we re keeping in memory, and barely over 100MB of overhead over the raw data, which, on average puts the overhead at 150 bytes per object.
The three phases so far are relatively fast and account for a small part of the overall process, so they don t appear clear-cut to each other, and don t take much space on the graph.
.cargo/.gitignore, followed by all the file revisions of .cargo/config.in, .clang-format, and so on. The data here doesn t depend on anything else, so we can finally directly import the data.
This means that for each revision, we actually expand the RawChunk into the full file data (RawChunks contain patches against a previous revision), and don t keep the RawChunk around. We also don t keep the full data after it was sent to the git-cinnabar-helper process (as far as cloning is concerned, it s essentially a wrapper for git-fast-import), except for the previous revision of the file, which is likely the patch base for the next revision.
We however keep in memory one or two things for each file revision: a mapping of its mercurial SHA1 and the corresponding git SHA1 of the imported data, and, when there is one, the file metadata (containing information about file copy/renames) that lives as a header in the file data in mercurial, but can t be stored in the corresponding git blobs, otherwise we d have irrelevant data in checkouts.
On the graph, this is where there is a steady and rather long increase of both live allocations and memory usage, in stairs for the latter.
In the git clone analyzed here, there are 2.02M file revisions, 78k of which have copy/move metadata for a cumulated size of 8.5MB of metadata. The raw size of the file revisions in RawChunk format is 3.85GB. The expanded data size is 67GB. By the end of this phase, we made 622 million more allocator calls, and peaked at about 2.05GB of live data in about 6.9M allocations. Compared to the beginning of this phase, that added about 530MB in 5 million allocations.
File metadata is stored in memory as python dicts, with 2 entries each, instead of raw form for convenience and future-proofing, so that would be at least 3 allocations each: one for each value, one for the dict, and maybe one for the dict storage ; their keys are all the same and are probably interned by python, so wouldn t count.
As mentioned above, we store a mapping of mercurial to git SHA1s, so for each file that makes 2 allocations, 4.04M total. Plus the 230k or 310k from metadata. Let s say 4.45M total. We re short 550k allocations, but considering the numbers involved, it would take less than one allocation per file on average to go over this count.
As for memory size, per this answer on stackoverflow, python strings have an overhead of 37 bytes, so each SHA1 (kept in hex form) will take 77 bytes (Note, that s partly why I didn t particularly care about storing them as binary form, that would only save 25%, not 50%). That s 311MB just for the SHA1s, to which the size of the mapping dict needs to be added. If it were a plain array of pointers to keys and values, it would take 2 * 8 bytes per file, or about 32MB. But that would be a hash table with no room for more items (By the way, I suspect the stairs that can be seen on the requested and in-use bytes is the hash table being realloc()ed). Plus at least 290 bytes per dict for each of the 78k metadata, which is an additional 22M. All in all, 530MB doesn t seem too much of a stretch.
git-cinnabar-helper process. Once we re done pushing data about a manifest to that process, we drop the corresponding data, except when we know it will be required later as the delta base for a subsequent RevChunk (which can happen in bundle2).
As with file revisions, for each manifest, we keep track of the mapping of SHA1s between mercurial and git. We also keep a DAG of the manifests history (contrary to git trees, mercurial manifests track their ancestry ; files do too, but git-cinnabar doesn t actually keep track of that separately ; it just relies on the manifests data to infer file ancestry).
On the graph, this is where the number of live allocations increases while both requested and in-use bytes decrease, noisily.
By the end of this phase, we made about 1 billion more allocator calls. Requested allocations went down to 1.02GB, for close to 7 million live allocations. Compared to the end of the dip at the beginning of this phase, that added 1.4 million allocations, and released 900MB. By now, we expect everything from the Reading manifests phase to have been released, which means we allocated around 620MB (1.52GB 900MB), for a total of 3.26M additional allocations (1.4M + 1.86M).
We have a dict for the SHA1s mapping (345k * 77 * 2 for strings, plus the hash table with 345k items, so at least 60MB), and the DAG, which, now that I m looking at memory usage, I figure has the one of the possibly worst structure, using 2 sets for each node (at least 232 bytes per set, that s at least 160MB, plus 2 hash tables with 345k items). I think 250MB for those data structures would be largely underestimated. It s not hard to imagine them taking 620MB, because really, that DAG implementation is awful. The number of allocations expected from them would be around 1.4M (4 * 345k), but I might be missing something. That s way less than the actual number, so it would be interesting to take a closer look, but not before doing something about the DAG itself.
Fun fact: the amount of data we re dealing with in this phase (the expanded size of all the manifests) is close to 2.9TB (yes, terabytes). With about 4700 seconds spent on this phase on a real clone (less with the release branch), we re still handling more than 615MB per second.
git ls-tree -r refs/cinnabar/metadata^3 if you re curious about the details). If I remember correctly, it s faster if the tree is created with an ordered list of paths, so the code created a list of paths, and then sorted it to send commands to create the tree. The former creates a new str of length 42 and a tuple of 3 elements for each and every one of the 2.7M mappings. With the 37 bytes overhead by str instance and the 56 + 3 * 8 bytes per tuple, we have at least 429MB wasted. Creating the tree itself keeps the corresponding fast-import commands in a buffer, where each command is going to be a tuple of 2 elements: a pointer to a method, and a str of length between 90 and 93. That s at least another 440MB wasted.
I already fixed the first half, but the second half still needs addressing.
malloc, realloc, calloc, etc.malloc, calloc, etc. except realloc. For realloc, we only count the delta in size between what the size was before and after the realloc.realloc was ever in-place), while the the latter gives us a lower bound (the amount that would be filled or copied if all reallocs were in-place).
Ideally, we d want the upper and lower bounds to be close to each other (indicating few realloc calls), and the total amount at the end of the process to be as close as possible to the amount of data we re handling (which we ve seen is around 3TB).
and this is clearly bad. Like, really bad. But we already knew that from the previous post, although it s nice to put numbers on it. The lower bound is about twice the amount of data we re handling, and the upper bound is more than 10 times that amount. Clearly, we can do better.
We ll see how things evolve after the necessary code changes happen. Stay tuned.
/dev/sdb) and copy the image onto it:
$ wget https://people.debian.org/~stapelberg/raspberrypi3/2017-03-22/2017-03-22-raspberry-pi-3-stretch-PREVIEW.img $ sudo dd if=2017-03-22-raspberry-pi-3-stretch-PREVIEW.img of=/dev/sdb bs=5MIf resolving client-supplied DHCP hostnames works in your network, you should be able to log into the Raspberry Pi 3 using SSH after booting it:
$ ssh root@rpi3 # Password is raspberry
git clone of the mozilla-central mercurial repository using git-cinnabar, and how the glibc memory allocator is using more than one would expect. This post is going to explore how/why it s happening.
I happen to have written a basic memory allocation logger for Firefox, so I used it to log all the allocations happening during a git clone exhibiting the runaway memory increase behavior (using a python that doesn t use its own allocator for small allocations).
The result was a 6.5 GB log file (compressed with zstd ; 125 GB uncompressed!) with 2.7 billion calls to malloc, calloc, free, and realloc, recorded across (mostly) 2 processes (the python git-remote-hg process and the native git-cinnabar-helper process ; there are other short-lived processes involved, but they do less than 5000 calls in total).
The vast majority of those 2.7 billion calls is done by the python git-remote-hg process: 2.34 billion calls. We ll only focus on this process.
Replaying those 2.34 billion calls with a program that reads the log allowed to reproduce the runaway memory increase behavior to some extent. I went an extra mile and modified glibc s realloc code in memory so it doesn t call memcpy, to make things faster. I also ran under setarch x86_64 -R to disable ASLR for reproducible results (two consecutive runs return the exact same numbers, which doesn t happen with ASLR enabled).
I also modified the program to report the number of live allocations (allocations that haven t been freed yet), and the cumulated size of the actually requested allocations (that is, the sum of all the sizes given to malloc, calloc, and realloc calls for live allocations, as opposed to what the memory allocator really allocated, which can be more, per malloc_usable_size).
RSS was not tracked because the allocations are never filled to make things faster, such that pages for large allocations are never dirty, and RSS doesn t grow as much because of that.
Full disclosure: it turns out the system bytes and in-use bytes numbers I had been collecting in the previous post were smaller than what they should have been, and were excluding memory that the glibc memory allocator would have mmap()ed. That however doesn t affect the trends that had been witnessed. The data below is corrected.
(Note that in the graph above and the graphs that follow, the horizontal axis represents the number of allocator function calls performed)
While I was here, I figured I d check how mozjemalloc performs, and it has a better behavior (although it has more overhead).
What doesn t appear on this graph, though, is that mozjemalloc also tells the OS to drop some pages even if it keeps them mapped (madvise(MADV_DONTNEED)), so in practice, it is possible the actual RSS decreases too.
And jemalloc 4.5:
(It looks like it has better memory usage than mozjemalloc for this use case, but its stats are being thrown off at some point, I ll have to investigate)
Going back to the first graph, let s get a closer look at what the allocations look like when the system bytes number is increasing a lot. The highlights in the following graphs indicate the range the next graph will be showing.
So what we have here is a bunch of small allocations (small enough that they don t seem to move the requested line ; most under 512 bytes, so under normal circumstances, they would be allocated by python, a few between 512 and 2048 bytes), and a few large allocations, one of which triggers a bump in memory use.
What can appear weird at first glance is that we have a large allocation not requiring more system memory, later followed by a smaller one that does. What the allocations actually look like is the following:
As it turns out, inspecting all the live allocations at that point, while there was a hole large enough to do the first two reallocs (the second actually happens in place), at the point of the third one, there wasn t a large enough hole to fit 8.8MB. What inspecting the live allocations also reveals, is that there is a large number of large holes between all the allocated memory ranges, presumably coming from previous similar patterns. There are, in fact, 91 holes larger than 1MB, 24 of which are larger than 8MB. It s the accumulation of those holes that can t be used to fulfil larger allocations that makes the memory use explode. And there aren t enough small allocations happening to fill those holes. In fact, the global trend is for less and less memory to be allocated, so, smaller holes are also being poked all the time. Really, it s all a straightforward case of memory fragmentation. The reason it tends not to happen with jemalloc is that jemalloc groups allocations by sizes, which the glibc allocator doesn t seem to be doing. The following is how we got a hole that couldn t fit the 8.8MB allocation in the first place:void *ptr0 = malloc(4850928); // #1391340418 void *ptr1 = realloc(some_old_ptr, 8000835); // #1391340419 free(ptr0); // #1391340420 ptr1 = realloc(ptr1, 8000925); // #1391340421 /* ... */ void *ptrn = malloc(879931); // #1391340465 ptr1 = realloc(ptr1, 8880819); // #1391340466 free(ptrn); // #1391340467
All would go well ifptr1 = realloc(ptr1, 8880467); // #1391324989; ptr1 is 0x5555de145600 /* ... */ void *ptrx = malloc(232); // #1391325001; ptrx is 0x5555de9bd760 ; that is 13 bytes after the end of ptr1. /* ... */ free(ptr1); // #1391325728; this leaves a hole of 8880480 bytes at 0x5555de145600.
ptrx was free()d, but it looks like it s long-lived. At least, it s still allocated by the time we reach the allocator call #1391340466. And since the hole is 339 bytes too short for the requested allocation, the allocator has no other choice than request more memory to the system.
What s bothering, though, is that the allocator chose to allocate ptrx in the space following ptr1, when it allocated similarly sized buffers after allocating ptr1 and before allocating ptrx in completely different places, and while there are plenty of holes in the allocated memory where it could fit.
Interestingly enough, ptrx is a 232 bytes allocation, which means under normal circumstances, python itself would be allocating it. In all likeliness, when the python allocator is enabled, it s allocations larger than 512 bytes that become obstacles to the larger allocations. Another possibility is that the 256KB fragments that the python allocator itself allocates to hold its own allocations become the obstacles (my original hypothesis). I think the former is more likely, though, putting back the blame entirely on glibc s shoulders.
Now, it looks like the allocation pattern we have here is suboptimal, so I re-ran a git clone under a debugger to catch when a realloc() for 8880819 bytes happens (the size is peculiar enough that it only happened once in the allocation log). But doing that with a conditional breakpoint is just too slow, so I injected a realloc wrapper with LD_PRELOAD that sends a SIGTRAP signal to the process, so that an attached debugger can catch it.
Thanks to the support for python in gdb, it was then posible to pinpoint the exact python instructions that made the realloc() call (it didn t come as a surprise ; in fact, that was one of the places I had in mind, but I wanted definite proof):
What happens here is that we re creating a mercurial manifest we got from the server in patch form against a previous manifest. Sonew = '' end = 0 # ... for diff in RevDiff(rev_patch): new += data[end:diff.start] new += diff.text_data end = diff.end # ... new += data[end:]
data contains the original manifest, and rev_patch the patch. The patch essentially contains instructions of the form replace n bytes at offset o with the content c .
The code here just does that in the most straightforward way, implementation-wise, but also, it turns out, possibly the worst way.
So let s unroll this loop over a couple iterations:
This allocates an emptynew = ''
str object. In fact, this doesn t actually allocate anything, and only creates a pointer to an interned representation of an empty string.
This is where things start to get awful.new += data[0:diff.start]
data is a str, so data[0:diff.start] creates a new, separate, str for the substring. One allocation, one copy.
Then appends it to new. Fortunately, CPython is smart enough, and just assigns data[0:diff.start] to new. This can easily be verified with the CPython REPL:
(and this is not happening because the example string is small here ; it also happens with larger strings, like>>> foo = '' >>> bar = 'bar' >>> foo += bar >>> foo is bar True
'bar' * 42000)
Back to our loop:
Now,new += diff.text_data
new is realloc()ated to have the right size to fit the appended text in it, and the contents of diff.text_data is copied. One realloc, one copy.
Let s go for a second iteration:
Here again, we re doing an allocation for the substring, and one copy. Thennew += data[diff.end:new_diff.start]
new is realloc()ated again to append the substring, which is an additional copy.
new += new_diff.text_data
new is realloc()ated yet again to append the contents of new_diff.text_data.
We now finish with:
which, again creates a substring from thenew += data[new_diff.end:]
data, and then proceeds to realloc()ate new one freaking more time.
That s a lot of malloc()s and realloc()s to be doing
realloc()s by using new = bytearray() instead of new = ''. I haven t looked in the CPython code what the growth strategy is, but, for example, appending a 4KB string to a 500KB bytearray makes it grow to 600KB instead of 504KB, like what happens when using str.realloc()s completely by preallocating the right size for the bytearray (with bytearray(size)), but that requires looping over the patch once first to know the new size, or using an estimate (the new manifest can t be larger than the size of the previous manifest + the size of the patch) and truncating later (although I m not sure it s possible to truncate a bytearray without a realloc()). As a downside, this initializes the buffer with null bytes, which is a waste of time.bytearrays previously allocated for previous manifests.''.join(). CPython is smart enough to create a single allocation for the total size in that case. That would be the most convenient solution, but see below.memoryview.''.join() on a list of memoryviews before Python 3.4.git clone of mozilla-central looks like the following (with the python allocator enabled):
(Note this hasn t actually landed on the master branch yet)
Compared to what it looked like before, this is largely better. But that s not the only difference: the clone was also about 1000 seconds faster. That s more than 15 minutes! But that s not all so surprising when you know the volumes of data handled here. More insight about this coming in an upcoming post.
But while the changes outlined above make the glibc allocator behavior less likely to happen, it doesn t totally obliviate it. In fact, it seems it is still happening by the end of the manifest import phase. We re still allocating increasingly large temporary buffers because the size of the imported manifests grows larger and larger, and every one of them is the result of patching a previous one.
The only way to avoid those large allocations creating holes would be to avoid doing them in the first place. My first attempt at doing that, keeping manifests as lists of lines instead of raw buffers, worked, but was terribly slow. So slow, in fact, that I had to stop a clone early and estimated the process would likely have taken a couple days. Iterating over multiple generators at the same time, a lot, kills performance, apparently. I ll have to try with significantly less of that.
Apologies the first blog post got sent out by mistake.
Weather comparisons between the two countries
Last year, I had come to know that this year s debconf is happening in Canada, a cold country. Hence, few weeks/month back, I started trying to find information online when I stumbled across few discussion boards where people were discussing about Innerwear and Outerwear and I couldn t understand what that was all about. Then somehow stumbled across this Video, which is of a game called the Long Dark and just seeing couple of episodes it became pretty clear to me why the people there were obsessing with getting the right clothes and everything about it. Couple of Debconf people were talking about the weather in Montreal, and surprise, surprise it was snowing there, in fact supposed to be near the storm of the century. Was amazed to see that they have a website to track how much snow has been lifted.
If we compare that to Pune, India weather-wise we are polar opposites. There used to be a time, when I was very young, maybe 5 yrs. old that once the weather went above 30 degree celsius, rains would fall, but now its gonna touch 40 degrees soon. And April and May, the two hottest months are yet to come.
China Gate
Before I venture further, I was gifted the book China Gate written by an author named William Arnold. When I read the cover and the back cover, it seemed the story was set between China and Taiwan, later when I started reading it, it shares history of Taiwan going back 200 or so odd years. This became relevant as next year s Debconf, Debconf 2018 will be in Taiwan, yes in Asia very much near to India. I am ashamed to say that except for the Tiananmen Square Massacre and the Chinese High-Speed Rail there wasn t much that I knew. According to the book, and I m paraphrasing here the gist I got was that for a long time, the Americans promised Taiwan it will be an Independent country forever, but due to budgetary and other political constraints, the United States took the same stand as China from 1979. Interestingly, now it seems Mr. Trump wants to again recognize Taiwan as a separate entity from China itself but as is with Mr. Trump you can t be sure of why he does, what he does. Is it just a manoeuvrer designed to out-smart the chinese and have a trade war or something else, only time will tell.
One thing which hasn t been shared in the book but came to know via web is that Taiwan calls itself Republic of China . If Taiwan wants to be independent then why the name Republic of China ? Doesn t that strengthen China s claim that Taiwan is an integral part of China. I don t understand it.
The book does seduce you into thinking that the events are happening in real-time, as in happening now.
That s enough OT for now.
Population Density
As well in the game and whatever I could find on the web, Canada seems to be on the lower side as far as population is concerned. IIRC, few years back, Canadians invited Indian farmers and gave them large land-holdings for over 100 years on some small pittance. While the link I have shared is from 2006, I read it online and in newspapers even as late as in 2013/2014. The point being there seems to be lot of open spaces in Canada, whereas in India we fight for even one inch literally, due to overpopulation. This sharing reminded me of Mark of Gideon . While I was young, I didn t understand the political meaning of it and still struggle to understand about whom the show was talking about. Was it India, Africa or some other continent they were talking about ?
This also becomes obvious when you figure out the surface area of the two countries. When I had started to learn about Canada, I had no idea, nor a clue that Canada is three times the size of India. And this is when I know India is a large country. but seeing that Canada is thrice larger just boggled my mind. As a typical urbanite, would probably become mad if in a rural area in Canada. Montreal, however seems to be something like Gwalior or Bangalore before IT stormed in, seems to be a place where people can work, play and have quite a few gardens as well.
Rail
This is one thing that is similar in both the great countries. India has Indian Railways and while the Canadians have their own mountain railway called viarail. India chugs on its 68k kilometre network, Canada is at fourth position with 52k network. With thrice the land size, it should have been somewhere where Russia is or even better than them. It would be interesting if a Canadian/s comment about their railway network and why it is so bad in terms of reach.
As far as food is concerned, somebody shared this
Also, have no idea if Canadian trains are as entertaining as Indian ones, in terms of diverse group of people as well as variety of food to eat as also shared a bit in the video. I am not aware whether Via Rail is the only network operator and there are other network operators unlike Indian Railways which has monopoly on most of the operations.
Business houses, Political Families
This is again something that is similar in both the countries, it seems (from afar) that its only few business houses and more importantly political families which have governed for years. From what little I could understand, both India and Canada have first past the post system which as shared by its critics is unfair to new and small parties. It would be interesting to see if Canada does a re-think. For India, it would need a massive public education outreach policy and implementation. We just had elections in 5 states of India with U.P. (with respect to area-size and population density) and from the last several years, the EVM s (Electronic Voting Machines) tries to make sure that nobody could know which area which party got the most votes. This is to make sure the winning party is not able to take revenge on people or areas which did not vote for them. Instead you have general region counting of votes with probably even the Election Commission not knowing which EVM went to what area and what results are there in sort of double-blind methodology.
As far as Business houses are concerned, I am guessing it s the same world-over, only certain people hold the wealth while majority of us are in hard-working, non-wealthy status.
Northern Lights, Aurora Borealis
Apart from all the social activities that Montreal is famous for, somebody told/shared with me that it is possible to see the Northern Lights, Aurora Borealis can be seen in Canada. I dunno how true or not it is, while probably in Montreal it isn t possible to see due to light pollution, but maybe around 40-50 kms. from the city ? Can people see it from Canada ? IF yes, how far would you have to go ? Are there any companies or people who take people to see the Northern Lights.
While I still have to apply for bursary, and if that gets ok, then try getting the visa, but if that goes through, apart from debconf and social activities happening in and around Montreal, Museums, Music etc. , this would be something I would like to experience if it s possible. While I certainly would have to be prepared for the cold that would be, if it s possible, no offence to debconf or anybody else but it probably would be the highlight of the entire trip if its possible. This should be called/labelled as the greatest show on earth TM. 
Weather comparisons between the two countries
Few days/weeks back, I had come to know that Canada, where this year s debconf is happening is cold country. I started trying to find information online when I stumbled across few boards where people were discussing about innerwear and outerwear and I couldn t understand what that was all about. Then somehow stumbled across this game, it s called the Long Dark and just seeing couple of episodes it became pretty clear to me why the people there were obsessing with getting the right clothes and everything about it. Couple of Debconf people were talking about weather in Montreal, and surprise, surprise it was snowing there, in fact supposed to be near the storm of the century. Was amazed to see that they have a website to track how much snow has been lifted.
If we compare that to Pune, India weather-wise we are polar opposites. There used to be a time, when I was very young, maybe 5 yrs. old that once the weather went above 30 degree celcius, rains would fall, but now its gonna touch 40 degrees soon. And April and May, the two hottest months are yet to come.
China Gate
Before I venture further, I was gifted the book China Gate written by an author named William Arnold. When I read the cover and the backcover, it seemed the story was set between China and Taiwan, later when I started reading it, it shares history of Taiwan going back 200 or so odd years. This became relevant as next year s Debconf, Debconf 2018 will be in Taiwan, yes in Asia very much near to India. I am ashamed to say that except for the Tiananmen Square Massacre and the Chinese High-Speed Rail there wasn t much that I knew. According to the book, and I m paraphrasing here the gist I got was that for a long time, the Americans promised Taiwan it will be an Independent country forever, but due to budgetary and other political constraints, the United States took the same stand as China from 1979 and now it seems Mr. Trump wants to again recognize Taiwan as a separate entity from China itself.
One thing which hasn t been shared in the book but came to know via web is that Taiwan calls itself Republic of China . If Taiwan wants to be independent then why the name Republic of China ? Doesn t that strengthen China s claim that Taiwan is an integral part of China. I don t understand it.
The book does seduce you into thinking that the events are happening in real-time, as in happening now.
That s enough OT for now.
Population Density
As well in the game and whatever I could find on the web, Canada seems to be on the lower side as far as population is concerned. IIRC, few years back, Canadians invited Indian farmers and gave them large land-holdings for over 100 years on some small pittance. While the link I have shared is from 2006, I read it online and in newspapers even as late as in 2013/2014. The point being there seems to be lot of open spaces in Canada, whereas in India we fight for even one inch literally, due to overpopulation. This sharing reminded me of Mark of Gideon . While I was young, I didn t understand the political meaning of it and still struggle to understand about whom the show was talking about. Was it India, Africa or some other continent they were talking about ?
This also becomes obvious when you figure out the surface area of the two countries. When I had started to learn about Canada, I had no idea, nor a clue that Canada is three times the size of India. And this is when I know India is a large country. but seeing that Canada is thrice larger just boggled my mind. As a typical urbanite, would probably become mad if in a rural area in Canada. Montreal, however seems to be something like Gwalior or Bangalore before IT stormed in, seems to be a place where people can work, play and have quite a few gardens as well.
Rail
This is one thing that is similar in both the great countries. India has Indian Railways and while the Canadians have their own mountain railway called viarail. India chugs on its 68k kilometre network, Canada is at fourth position with 52k network. With thrice the land size, it should have been somewhere where Russia is or even better than them. It would be interesting if Canadians comment about their railway network and why it is so bad in terms of reach.
Business houses, Political Families
This is again something that is similar in both the countries, it seems (from afar) that its only few business houses and more importantly political families which have governed for years. 
DebConf17 will take place in Montreal, Canada in August 2017. We are working hard
to provide fuel for hearts and minds, to make this conference once again a
fertile soil for the Debian Project flourishing. Please join us and support
this landmark in the Free Software calendar.
Eighteen companies have already committed to sponsor DebConf17! With a warm
welcome, we'd like to introduce them to you.
Our first Platinum sponsor is Savoir-faire Linux,
a Montreal-based Free/Open-Source Software company which offers Linux and
Free Software integration solutions and actively contributes to many free
software projects. "We believe that it's an essential piece [Debian], in a
social and political way, to the freedom of users using modern technological
systems", said Cyrille B raud, president of Savoir-faire Linux.
Our first Gold sponsor is Valve,
a company developing games, social entertainment platform, and game engine technologies.
And our second Gold sponsor is Collabora,
which offers a comprehensive range of services to help its clients to
navigate the ever-evolving world of Open Source.
As Silver sponsors we have
credativ
(a service-oriented company focusing on open-source software and also a
Debian development partner),
Mojatatu Networks
(a Canadian company developing Software Defined Networking (SDN) solutions),
the Bern University of Applied Sciences
(with over 6,600 students enrolled,
located in the Swiss capital),
Microsoft
(an American multinational technology company),
Evolix
(an IT managed services and support company located in Montreal),
Ubuntu
(the OS supported by Canonical)
and Roche
(a major international pharmaceutical provider and research company
dedicated to personalized healthcare).
ISG.EE,
IBM,
Bluemosh,
Univention
and Skroutz
are our Bronze sponsors so far.
And finally, The Linux foundation,
R seau Koumbit
and adte.ca
are our supporter sponsors.
Become a sponsor too!
Would you like to become a sponsor? Do you know of or work in a company or
organization that may consider sponsorship?
Please have a look at our
sponsorship brochure
(or a summarized flyer),
in which we outline all the details and describe the sponsor benefits.
For further details, feel free to contact us through sponsors@debconf.org, and
visit the DebConf17 website at https://debconf17.debconf.org.
The above picture is the blend of the two communities from foss community and mozilla India. And unless you were there you wouldn t know who is from which community which is what FOSS is all about. But as always I m getting a bit ahead of myself.
Akshat, who works at NCRA as a programmer, the standing guy on the left shared with me in January this year that this year too, we should have two stalls, foss community and mozilla India stalls next to each other. While we had the banners, we were missing stickers and flyers. Funds were and are always an issue and this year too, it would have been emptier if we didn t get some money saved from last year minidebconf 2016 that we had in Mumbai. Our major expenses included printing stickers, stationery and flyers which came to around INR 5000/- and couple of LCD TV monitors which came for around INR 2k/- as rent. All the labour was voluntary in nature, but both me and Akshat easily spending upto 100 hours before the event. Next year, we want to raise to around INR 10-15k so we can buy 1 or 2 LCD monitors and we don t have to think for funds for next couple of years. How will we do that I have no idea atm.
Me and Akshat did all the printing and stationery runs and hence had not been using my lappy for about 3-4 days.
Come to the evening before the event and the laptop would not start. Coincidentally, or not few months or even last at last year s Debconf people had commented on IBM/Lenovo s obsession with proprietary power cords and adaptors. I hadn t given it much thought but when I got no power even after putting it on AC power for 3-4 hours, I looked up on the web and saw that the power cord and power adaptors were all different even in T440 and even that under existing models. In fact I couldn t find mine hence sharing it via pictures below.
I knew/suspected that thinkpads would be rare where I was going, it would be rarer still to find the exact power cord and I was unsure whether it was the power cord at fault or adaptor or whatever goes for SMPS in laptop or memory or motherboard/CPU itself. I did look up the documentation at support.lenovo.com and was surprised at the extensive documentation that Lenovo has for remote troubleshooting.
I did the usual take out the battery, put it back in, twiddle with the little hole in the bottom of the laptop, trying to switch on without the battery on AC mains, trying to switch on with battery power only but nothing worked. Couple of hours had gone by and with a resigned thought went to bed, convincing myself that anyways it s good I am not taking the lappy as it is extra-dusty there and who needs a dead laptop anyways.
Update After the event was over, I did contact Lenovo support and within a week, with one visit from a service engineer, he was able to identify that it was a faulty cable which was at fault and not the the other things which I was afraid of. Another week gone by and lenovo replaced the cable. Going by service standards that I have seen of other companies, Lenovo deserves a gold star here for the prompt service they provided. I probably would end up subscribing to their extended 2-year warranty service when my existing 3 year warranty is about to be over.
Next day, woke up early morning, two students from COEP hostel were volunteering and we made our way to NCRA, Pune University Campus. Ironically, though we were under the impression that we would be the late arrivals, it turned out we were the early birds. 5-10 minutes passed by and soon enough we were joined by Aniket and we played catch-up for a while. We hadn t met each other for a while so it was good to catch-up. Then slowly other people starting coming in and around 07:10-07:15 we started for GMRT, Khodad.
Now I had been curious as had been hearing for years that the Pune-Nashik NH-50 highway would be concreted and widened to six-lane highways but the experience was below par. Came back and realized the proposal has now been pushed back to 2020.
From the mozilla team, only Aniket was with us, the rest of the group was coming straight from Nashik. Interestingly, all the six people who came, came on bikes which depending upon how you look at it was either brave or stupid. Travelling on bikes on Indian highways you either have to be brave or stupid or both, we have more than enough accidents due to quality of road construction, road design, lane-changing drivers and many other issues. This is probably not the place for it hence will use some other blog post to rant about that.
We reached around 10:00 hrs. IST and hung around till lunch as Akshat had all the marketing material, monitors etc. The only thing we had were couple of lappies and couple of SBC s, an RPI 3 and a BBB.
Our find for the event was Aarti Kashyap who you can see above. She is a third-year student at COEP and one of the rare people who chose to interact with hardware rather than software. From last several years, we had been trying, successfully and unsuccessfully to get more Indian women and girls interested into technology. It is a vicious circle as till a girl/woman doesn t volunteer we are unable to share our knowledge to the extent we can which leads them to not have much interest in FOSS or even technology in general.
While there are groups are djangogirls, Pyladies and railgirls and even Outreachy which tries to motivate getting girls into computing but it s a long road ahead.
We are short of both funds and ideas as to how to motivate more girls to get into computing and then to get into playing with hardware. I don t know where to start and end for whoever wants to play with hardware. From SBC s, routers to blade servers the sky is the limit. Again this probably isn t the place for it, hence probably we can chew it on more at some other blog post.
This year, we had a lowish turnout due to the fact that the 12th board exams 1st paper was on the day we had opened. So instead of 20-25k, we probably had 5-7k fewer people pass through. There were two-three things that we were showing, we were showing Debian on one of the systems, we were showing the output from the SBC s on the other monitor but the glare kept hitting the monitors.
While the organizers had done exemplary work over last year. They had taped the carpets on the ground so there was hardly any dust moving around. However, I wished the organizers had taken the pains to have two cloth roofs over our head instead of just one, the other roof head could be say 2 feet up, this would have done two things
a. It probably would have cooled the place a bit more as
b. We could get diffused sunlight which would have lessened the glare and reflection the LCD s kept throwing back. At times we also got people to come to our side as can be seen in Aarti s photo as can be seen above.
If these improvements can be made for next year, this would result in everybody in our Pandal would benefit, not just us and mozilla. This would be benefiting around 10-15 organizations which were within the same temporary structure.
Of course, it depends very much on the budget they are able to have and people who are executing, we can just advise.
The other thing which had been missing last year and this year is writing about Single Board Computers in Marathi. If we are to promote them as something to replace a computer or something for a younger brother/sister to learn computing upon at a lower cost, we need leaflets written in their language to be more effective. And this needs to be in the language and mannerisms that people in that region understand. India, as probably people might have experienced is a dialect-prone country. Which means every 2-5 kms, the way the language is spoken is different from anywhere else. The Marathi spoken by somebody who has lived in Ravivar Peth for his whole life and a person who has lived in say Kothrud are different. The same goes from any place and this place, Khodad, Narayangaon would have its own dialect, its own mini-codespeak.
Just to share, we did have one in English but it would have been a vast improvement if we could do it in the local language. Maybe we can discuss about this and ask for help from people.
Outside, Looking in
What had been interesting about the whole journey were the new people who were bringing all their passion and creativity to the fore. From the mozilla community, we had Akshay who is supposed to be a wizard on graphics, animation, editing anything to do with the visual medium. He shared some of the work he had done and also shared a bit about how blender works with people who wanted to learn about that.
Mayur, whom you see in the picture pointing out something about FOSS and this was the culture that we strove to have. I know and love and hate the browser but haven t been able to fathom the recklessness that Mozilla has been doing the last few years, which has just been having one mis-adventure after another.
For instance, mozstumbler was an effort which I thought would go places. From what little I understood, it served/serves as a user-friendly interface to a potential user while still sharing all the data with OSM . They (Mozilla) seems/seemed to have a fatalistic take as it provided initial funding but then never fully committing to the project.
Later, at night we had the whole free software and open-source sharings where I tried to emphasize that without free software, the term open-source would not have come into existence. We talked and talked and somewhere around 02:00 I slept, the next day was an extension of the first day itself where we ribbed each other good-naturedly and still shared whatever we could share with each other.
I do hope that we continue this tradition for great many years to come and engage with more and more people every passing year. 
Large software ecosystems, such as GNU/Linux distributions, demand a large amount of effort to make sure all of its components work correctly invidually, and also integrate correctly with each other to form a coherent system. Automated Quality Assurance techniques can prevent issues from reaching end users. This paper presents a pattern language originated in the Debian project for automated software testing in production-like environments. Such environments are closer in similarity to the environment where software will be actually deployed and used, as opposed to the development environment under which developers and regular Continuous Integration mechanisms usually test software products. The pattern language covers the handling of issues arising from the difference between development and production-like environments, as well as solutions for writing new, exclusive tests for as-installed functional tests. Even though the patterns are documented here in the context of the Debian project, they can also be generalized to other contexts.In practical terms, the paper documents a set of patterns I have noticed in the last few years, when I have been pushing the Debian Continous Integration project. It should be an interesting read for people interested in the testing of Debian packages in their installed form, as done with autopkgtest. It should also be useful for people from other distributions interested in the subject, as the issues are not really Debian-specific. I have recently finished the final version of the paper, which should be published in the ACM Digital Library at any point now. You can download a copy of the paper in PDF. Source is also available, if you are into markdown, LaTeX, makefiles and this sort of thing. If everything goes according to plan, I should be presenting a talk on this at the next Debconf in Montreal.
(Note, those memory sizes are RSS, minus shared )
It also was taking an unexpected long time, but then, I hadn t cloned a large repository like mozilla-central from scratch in a while, so I wasn t sure if it was just related to its recent growth in size or otherwise. So I collected data on 0.4.0 as well.
Less time spent, less memory usage ok. There s definitely something wrong on master. But wait a minute, that slope from ~2GB to ~4GB on the git-remote-hg process doesn t actually make any kind of sense. I mean, I d understand it if it were starting and finishing with the Import manifest phase, but it starts in the middle of it, and ends long before it finishes. WTH?
First things first, since RSS can be a variety of things, I checked /proc/$pid/smaps and confirmed that most of it was, indeed, the heap.
That s the point where you reach for Google, type something like python memory profile and find various tools. One from the results that I remembered having used in the past is guppy s heapy.
Armed with pdb, I broke execution in the middle of the slope, and tried to get memory stats with heapy. SIGSEGV. Ouch.
Let s try something else. I reached out to objgraph and pympler. SIGSEGV. Ouch again.
Tried working around the crashes for a while (too long while, retrospectively, hindsight is 20/20), and was somehow successful at avoiding them by peaking at a smaller set of objects. But whatever I did, despite being attached to a process that had 2.6GB RSS, I wasn t able to find more than 1.3GB of data. This wasn t adding up.
It surely didn t help that getting to that point took close to an hour each time. Retrospectively, I wish I had investigated using something like Checkpoint/Restore in Userspace.
Anyways, after a while, I decided that I really wanted to try to see the whole picture, not smaller peaks here and there that might be missing something. So I resolved myself to look at the SIGSEGV I was getting when using pympler, collecting a core dump when it happened.
Guess what? The Debian python-dbg package does not contain the debug symbols for the python package. The core dump was useless.
Since I was expecting I d have to fix something in python, I just downloaded its source and built it. Ran the command again, waited, and finally got a backtrace. First Google hit for the crashing function? The exact (unfixed) crash reported on the python bug tracker. No patch.
Crashing code is doing:
((f)->f_builtins != (f)->f_tstate->interp->builtins)
(f)->f_tstate is NULL. Classic NULL deref.
Added a guard (assessing it wouldn t break anything). Ran the command again. Waited. Again. SIGSEGV.
Facedesk. Another crash on the same line. Did I really use the patched python? Yes. But this time (f)->f_tstate->interp is NULL. Sigh.
Same player, shoot again.
Finally, no crash but still stuck on only 1.3GB accounted for. Ok, I know not all python memory profiling tools are entirely reliable, let s try heapy again. SIGSEGV. Sigh. No debug info on the heapy module, where the crash happens. Sigh. Rebuild the module with debug info, try again. The backtrace looks like heapy is recursing a lot. Look at %rsp, compare with the address space from /proc/$pid/maps. Confirmed. A stack overflow. Let s do ugly things and increase the stack size in brutal ways.
Woohoo! Now heapy tells me there s even less memory used than the 1.3GB I found so far. Like, half less. Yeah, right.
I m not clear on how I got there, but that s when I found gdb-heap, a tool from Red Hat s David Malcolm, and the associated talk Dude, where s my RAM? A deep dive into how Python uses memory (slides).
With a gdb attached, I would finally be able to rip python s guts out and find where all the memory went. Or so I thought. The gdb-heap tool only found about 600MB. About as much as heapy did, for that matter, but it could be coincidental. Oh. Kay.
I don t remember exactly what went through my mind then, but, since I was attached to a running process with gdb, I typed the following on the gdb prompt:
gdb> call malloc_stats()
Arena 0:
system bytes = some number above (but close to) 2GB
in use bytes = some number above (but close to) 600MB
There are two things of note on this data:
malloc() instead of mmap() for its arenas.
Aha! I thought. That definitely looks much better. Less gap between what glibc says it requested from the system and the RSS size. And, more importantly, no runaway increase of memory usage in the middle of nowhere.
I was preparing myself to write a post about how mixing allocators could have unintended consequences. As a comparison point, I went ahead and ran another test, with the python allocator entirely disabled, this time.
Heh. It turns out glibc was acting up all alone. So much for my (plausible) theory. (I still think mixing allocators can have unintended consequences.)
(Note, however, that the reason why the python allocator exists is valid: without it, the overall clone took almost 10 more minutes)
And since I had been getting all this data with 0.4.0, I gathered new data without the python allocator with the master branch.
This paints a rather different picture than the original data on that branch, with much less memory use regression than one would think. In fact, there isn t much difference, except for the spike at the end, which got worse, and some of the noise during the Import manifests phase that got bigger, implying larger amounts of temporary memory used. The latter may contribute to the allocation patterns that throw glibc s memory allocator off.
It turns out tracking memory usage in python 2.7 is rather painful, and not all the tools paint a complete picture of it. I hear python 3.x is somewhat better in that regard, and I hope it s true, but at the moment, I m stuck with 2.7. The most reliable tool I ve used here, it turns out, is pympler. Or rebuilding the python interpreter without its allocator, and asking the system allocator what is allocated.
With all this data, I now have some defined problems to tackle, some easy (the spike at the end of the clone), and some less easy (working around glibc allocator s behavior). I have a few hunches as to what kind of allocations are causing the runaway increase of RSS. Coincidentally, I m half-way through a refactor of the code dealing with manifests, and it should help dealing with the issue.
But that will be the subject of a subsequent post.
ZFS has detected a checksum error: eid: 138 class: checksum host: alexandria time: 2017-01-29 18:08:10-0600 vtype: diskThis means there was a data error on the drive. But it s worse than a typical data error this is an error that was not detected by the hardware. Unlike most filesystems, ZFS and btrfs write a checksum with every block of data (both data and metadata) written to the drive, and the checksum is verified at read time. Most filesystems don t do this, because theoretically the hardware should detect all errors. But in practice, it doesn t always, which can lead to silent data corruption. That s why I use ZFS wherever I possibly can. As I looked into this issue, I saw that ZFS repaired about 400KB of data. I thought, well, that was unlucky and just ignored it. Then a week later, it happened again. Pretty soon, I noticed it happened every Sunday, and always to the same drive in my pool. It so happens that the highest I/O load on the machine happens on Sundays, because I have a cron job that runs zpool scrub on Sundays. This operation forces ZFS to read and verify the checksums on every block of data on the drive, and is a nice way to guard against unreadable sectors in rarely-used data. I finally swapped out the drive, but to my frustration, the new drive now exhibited the same issue. The SATA protocol does include a CRC32 checksum, so it seemed (to me, at least) that the problem was unlikely to be a cable or chassis issue. I suspected motherboard. It so happened I had a 9211-8i SAS card. I had purchased it off eBay awhile back when I built the server, but could never get it to see the drives. I wound up not filling it up with as many drives as planned, so the on-board SATA did the trick. Until now. As I poked at the 9211-8i, noticing that even its configuration utility didn t see any devices, I finally started wondering if the SAS/SATA breakout cables were a problem. And sure enough I realized I had a reverse cable and needed a forward one. $14 later, I had the correct cable and things are working properly now. One other note: RAM errors can sometimes cause issues like this, but this system uses ECC DRAM and the errors would be unlikely to always manifest themselves on a particular drive. So over the course of this, had I not been using ZFS, I would have had several megabytes of reads with undetected errors. Thanks to using ZFS, I know my data integrity is still good.
You grant us and our legal successors the right to store and display your Content and make incidental copies as necessary to render the Website and provide the Service.Section D.5 then goes on to say:
[...] You grant each User of GitHub a nonexclusive, worldwide license to access your Content through the GitHub Service, and to use, display and perform your Content, and to reproduce your Content solely on GitHub as permitted through GitHub's functionality
The new TOS is potentially very bad for copylefted Free Software. It potentially neuters it entirely, so GPL licensed software hosted on Github has an implicit BSD-like licenseHess has since removed all his content (mostly mirrors) from GitHub. Others disagree. In a well-reasoned blog post, Debian developer Jonathan McDowell explained the rationale behind the changes:
My reading of the GitHub changes is that they are driven by a desire to ensure that GitHub are legally covered for the things they need to do with your code in order to run their service.This seems like a fair point to make: GitHub needs to protect its own rights to operate the service. McDowell then goes on to do a detailed rebuttal of the arguments made by Glaser, arguing specifically that section D.5 "does not grant [...] additional rights to reproduce outside of GitHub". However, specific problems arise when we consider that GitHub is a private corporation that users have no control over. The "Services" defined in the ToS explicitly "refers to the applications, software, products, and services provided by GitHub". The term "Services" is therefore not limited to the current set of services. This loophole may actually give GitHub the right to bypass certain provisions of licenses used on GitHub. As Hess detailed in a later blog post:
If Github tomorrow starts providing say, an App Store service, that necessarily involves distribution of software to others, and they put my software in it, would that be allowed by this or not? If that hypothetical Github App Store doesn't sell apps, but licenses access to them for money, would that be allowed under this license that they want to my software?However, when asked on IRC, Bradley M. Kuhn of the Software Freedom Conservancy explained that "ultimately, failure to comply with a copyleft license is a copyright infringement" and that the ToS do outline a process to deal with such infringement. Some lawyers have also publicly expressed their disagreement with Glaser's assessment, with Richard Fontana from Red Hat saying that the analysis is "basically wrong". It all comes down to the intent of the ToS, as Kuhn (who is not a lawyer) explained:
any license can be abused or misused for an intent other than its original intent. It's why it matters to get every little detail right, and I hope Github will do that.He went even further and said that "we should assume the ambiguity in their ToS as it stands is favorable to Free Software". The ToS are in effect since February 28th; users "can accept them by clicking the broadcast announcement on your dashboard or by continuing to use GitHub". The immediacy of the change is one of the reasons why certain people are rushing to remove content from GitHub: there are concerns that continuing to use the service may be interpreted as consent to bypass those licenses. Hess even hosted a separate copy of the ToS [PDF] for people to be able to read the document without implicitly consenting. It is, however, unclear how a user should remove their content from the GitHub servers without actually agreeing to the new ToS.
[...] unless there is a Contributor License Agreement to the contrary, whenever you make a contribution to a repository containing notice of a license, you license your contribution under the same terms, and agree that you have the right to license your contribution under those terms.I was concerned this would establish the controversial practice of forcing CLAs on every GitHub user. I managed to find a post from a lawyer, Kyle E. Mitchell, who commented on the draft and, specifically, on the CLA. He outlined issues with wording and definition problems in that section of the draft. In particular, he noted that "contributor license agreement is not a legal term of art, but an industry term" and "is a bit fuzzy". This was clarified in the final draft, in section D.6, by removing the use of the CLA term and by explicitly mentioning the widely accepted norm for licenses: "inbound=outbound". So it seems that section D.6 is not really a problem: contributors do not need to necessarily delegate copyright ownership (as some CLAs require) when they make a contribution, unless otherwise noted by a repository-specific CLA. An interesting concern he raised, however, was with how GitHub conducted the drafting process. A blog post announced the change on February 7th with a link to a form to provide feedback until the 21st, with a publishing deadline of February 28th. This gave little time for lawyers and developers to review the document and comment on it. Users then had to basically accept whatever came out of the process as-is. Unlike every software project hosted on GitHub, the ToS document is not part of a Git repository people can propose changes to or even collaboratively discuss. While Mitchell acknowledges that "GitHub are within their rights to update their terms, within very broad limits, more or less however they like, whenever they like", he sets higher standards for GitHub than for other corporations, considering the community it serves and the spirit it represents. He described the process as:
[...] consistent with the value of CYA, which is real, but not with the output-improving virtues of open process, which is also real, and a great deal more pleasant.Mitchell also explained that, because of its position, GitHub can have a major impact on the free-software world.
And as the current forum of preference for a great many developers, the knock-on effects of their decisions throw big weight. While GitHub have the wheel and they ve certainly earned it for now they can do real damage.In particular, there have been some concerns that the ToS change may be an attempt to further the already diminishing adoption of the GPL for free-software projects; on GitHub, the GPL has been surpassed by the MIT license. But Kuhn believes that attitudes at GitHub have begun changing:
GitHub historically had an anti-copyleft culture, which was created in large part by their former and now ousted CEO, Preston-Warner. However, recently, I've seen people at GitHub truly reach out to me and others in the copyleft community to learn more and open their minds. I thus have a hard time believing that there was some anti-copyleft conspiracy in this ToS change.
While we are confident that these Terms serve the best needs of the community, we take our users' feedback very seriously and we are looking closely at ways to address their concerns.Regardless, free-software enthusiasts have other concerns than the new ToS if they wish to use GitHub. First and foremost, most of the software running GitHub is proprietary, including the JavaScript served to your web browser. GitHub also created a centralized service out of a decentralized tool (Git). It has become the largest code hosting service in the world after only a few years and may well have become a single point of failure for free software collaboration in a way we have never seen before. Outages and policy changes at GitHub can have a major impact on not only the free-software world, but also the larger computing world that relies on its services for daily operation. There are now free-software alternatives to GitHub. GitLab.com, for example, does not seem to have similar licensing issues in its ToS and GitLab itself is free software, although based on the controversial open core business model. The GitLab hosting service still needs to get better than its grade of "C" in the GNU Ethical Repository Criteria Evaluations (and it is being worked on); other services like GitHub and SourceForge score an "F". In the end, all this controversy might have been avoided if GitHub was generally more open about the ToS development process and gave more time for feedback and reviews by the community. Terms of service are notorious for being confusing and something of a legal gray area, especially for end users who generally click through without reading them. We should probably applaud the efforts made by GitHub to make its own ToS document more readable and hope that, with time, it will address the community's concerns.
Note: this article first appeared in the Linux Weekly News.
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF echo "deb http://download.mono-project.com/repo/debian wheezy main" sudo tee /etc/apt/sources.list.d/mono-xamarin.list sudo apt-get update sudo apt-get upgrade sudo apt-get install mono-runtimeThe install instruction comes from mono-project and the initial pointer was found on raspberrypi forums, stackoverflow and Benny Michielsen s blog.
apt-get install mono-complete.
Compiling gcc
Using the Raspberry Pi 3, I recovered an old project based on buildroot for the raspberry pi 2. And just for building the tool-chain I had a few issues.
First the compilation would stop during mnp compilation:
/usr/bin/gcc -std=gnu99 -c -DHAVE_CONFIG_H -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_divrem_1 -O2 -Wa,--noexecstack tmp-divrem_1.s -fPIC -DPIC -o .libs/divrem_1.o tmp-divrem_1.s: Assembler messages: tmp-divrem_1.s:129: Error: selected processor does not support ARM mode mls r1,r4,r8,r11' tmp-divrem_1.s:145: Error: selected processor does not support ARM mode mls r1,r4,r8,r11' tmp-divrem_1.s:158: Error: selected processor does not support ARM mode mls r1,r4,r8,r11' tmp-divrem_1.s:175: Error: selected processor does not support ARM mode mls r1,r4,r3,r8' tmp-divrem_1.s:209: Error: selected processor does not support ARM mode mls r11,r4,r12,r3' Makefile:768: recipe for target 'divrem_1.lo' failed make[]: *** [divrem_1.lo] Error 1I Googled the error and found this post on the Raspberry Pi forum not really helpful
source.list with plain debian Jessie, make sure to add theses lines before the raspbian lines:
# add official debian jessie (real armhf gcc) deb http://ftp.fr.debian.org/debian/ jessie main contrib non-free deb-src http://ftp.fr.debian.org/debian/ jessie main deb http://security.debian.org/ jessie/updates main deb-src http://security.debian.org/ jessie/updates main deb http://ftp.fr.debian.org/debian/ jessie-updates main deb-src http://ftp.fr.debian.org/debian/ jessie-updates mainThen run the following to get the debian gpg keys, but don t yet upgrade your system:
apt update apt install debian-archive-keyringNow, let s add the pinning.
APT::Default-Release "stable"; in your apt.conf (as I did) remove it. It does not mix well with fine grained pinning we will then implement.
Then, fill your /etc/apt/preferences file with the following:
# Debian Package: * Pin: release o=Debian,a=stable,n=jessie Pin-Priority: 700 # Raspbian Package: * Pin: release o=Raspbian,a=stable,n=jessie Pin-Priority: 600 Package: * Pin: release o=Raspberry Pi Foundation,a=stable,n=jessie Pin-Priority: 600 # Mono Package: * Pin: release v=7.0,o=Xamarin,a=stable,n=wheezy,l=Xamarin-Stable,c=main Pin-Priority: 800Note: You can use
apt-cache policy (no parameter) to debug pinning.o=)apt update apt-cache policy gcc rm /var/cache/apt/archives/* apt upgrade apt-cache policy gccNote: Removing the cache ensure we download the packages from debian as raspbian is using the exact same naming but we now they are not compiled with a real armhf tool-chain. Second issue with gcc The build stopped on
recipe for target 's-attrtab' failed. There are many references on the web, that one was easy, it just need more memory, so I added some swap on the external SSD I was already using to work on buildroot.
Conclusion
That s it for today, not bad considering my last post was more that 3 years ago
Next.