We've just put the new PostgreSQL minor releases live on apt.postgresql.org.
Building 5 major versions for 10 distributions produces quite a lot of stuff:
$ readelf --segments /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Elf file type is DYN (Shared object file)
Entry point 0x77c00
There are 7 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x00000000001b5ea4 0x00000000001b5ea4 R E 200000
LOAD 0x00000000001b6b60 0x00000000003b6b60 0x00000000003b6b60
0x0000000000028dc4 0x000000000002c998 RW 200000
The permissions to load the code and data segments are given the
Flags output. Code has execute-permissions, and data has
write-permissions. These flags are mapped into flags for the mmap
call, which the loader then uses to map the various segments of the
file into memory.
So, do you actually need execute-permissions on the underlying file
to mmap that segment as executable? No, because you can read it.
If I can read it, then I can copy the segment to another area of
memory I have already mapped with PROT_EXEC and execute it there
anyway.
Googling suggests that some systems do require execute-permissions
on a file if you want to directly mmap pages from it with
PROT_EXEC (and if you dig into the kernel source, there's an
ancient system call uselib that looks like it comes from a.out
days, given it talks about loading libraries at fixed addresses, that
also wants this). This doesn't sound like a terrible hardening step;
I wouldn't be surprised if some hardening patches require it. Maximum
compatability and historical-features such as a.out also probably
explains why gcc creates shared libraries with execute permissions
by default.
Thus, should you feel like it, you can run a shared-library.
Something trivial will suffice:
int function(void)
return 100;
$ gcc -fPIC -shared -o libfoo.so foo.c $ ./libfoo.so Segmentation faultThis is a little more interesting (to me anyway) to dig into. At a first pass, why does this even vaguely work? That's easy -- an ELF file is an ELF file, and the kernel is happy to map those PT_LOAD segments in and jump to the entry point for you:
$ readelf --segments ./libfoo.so
Elf file type is DYN (Shared object file)
Entry point 0x570
There are 6 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x000000000000070c 0x000000000000070c R E 200000
LOAD 0x0000000000000710 0x0000000000200710 0x0000000000200710
0x0000000000000238 0x0000000000000240 RW 200000
What's interesting here is that the shared-library has an entry point
(e_entry) at all. ELF defines it as:
This member gives the virtual address to which the system first transfers control, thus starting the process. If the file has no associated entry point, this member holds zero.First things first, where did that entry point come from? The ld manual tells us that the linker will set the entry point based upon the following hierarchy:
We know we're not specifying an entry point. The ENTRY command is interesting; we can check our default-link script to see if that is specified:
- the -e entry command-line option;
- the ENTRY(symbol) command in a linker script;
- the value of a target specific symbol
- the address of the first byte of the .text section, if present;
- The address 0.
$ ld --verbose grep ENTRY ENTRY(_start)Interesting. Obviously we didn't specify a _start; so do we have one? A bit of digging leads to the crt files, for C Run-Time. These are little object files automatically linked in by gcc that actually do the support-work to get a program to the point that main is ready to run. So, if we go and check out the crt files, one can find a definition of _start in crt1.o
$ nm /usr/lib/x86_64-linux-gnu/crt1.o
crt1.o:
0000000000000000 R _IO_stdin_used
0000000000000000 D __data_start
U __libc_csu_fini
U __libc_csu_init
U __libc_start_main
0000000000000000 T _start
0000000000000000 W data_start
U main
But do we have that for our little shared-library? We can get a feel
for what gcc is linking in by examining the output of
-dumpspecs. Remembering gcc is mostly just a driver that
calls out to other things, a specs file is what gcc uses to
determine which arguments pass around to various stages of a compile:
$ gcc -dumpspecs ... *startfile: % !shared: % pg p profile:gcrt1.o%s;pie:Scrt1.o%s;:crt1.o%s crti.o%s % static:crtbeginT.o%s;shared pie:crtbeginS.o%s;:crtbegin.o%sThe format isn't really important here (of course you can read about it); but the gist is that various flags, such as -static or -pie get passed different run-time initailisation helpers to link-in. But we can see that if we're creating a shared library we won't be getting crt1.o. We can double-confirm this by checking the output of gcc -v (cut down for clarity).
$ gcc -v -fPIC -shared -o libfoo.so foo.c Using built-in specs. ... /usr/lib/gcc/x86_64-linux-gnu/4.4.5/collect2 -shared -o libfoo.so /usr/lib/gcc/x86_64-linux-gnu/4.4.5/../../../../lib/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.4.5/crtbeginS.o /tmp/ccRpsQU3.o /usr/lib/gcc/x86_64-linux-gnu/4.4.5/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.4.5/../../../../lib/crtn.oSo this takes us further down ld's entry-point logic to pointing to the first bytes of .text, which is where the entry-point comes from. So that solves the riddle of the entry point. There's one more weird thing you notice when you run the library, which is the faulting address in kern.log:
libfoo.so[8682]: segfault at 1 ip 0000000000000001 sp 00007fffcd63ec48 error 14 in libfoo.so[7f54c51fa000+1000]The first thing is decoding error; 14 doesn't seem to have any relation to anything. Of course everyone has the Intel 64 Architecture Manual (or mm/fault.c that also mentions the flags) to decode this into 1110 which means "no page found for a user-mode write access with reserved-bits found to be set" (there's another post in all that someday!). So why did we segfault at 0x1, which is an odd address to turn up? Let's disassemble what actually happens when this starts.
00000000000004a0 <call_gmon_start>:
4a0: 48 83 ec 08 sub $0x8,%rsp
4a4: 48 8b 05 2d 03 20 00 mov 0x20032d(%rip),%rax # 2007d8 <_DYNAMIC+0x190>
4ab: 48 85 c0 test %rax,%rax
4ae: 74 02 je 4b2 <call_gmon_start+0x12>
4b0: ff d0 callq *%rax
4b2: 48 83 c4 08 add $0x8,%rsp
4b6: c3 retq
$ readelf --relocs ./libfoo.so Relocation section '.rela.dyn' at offset 0x3f0 contains 4 entries: Offset Info Type Sym. Value Sym. Name + Addend 000000200810 000000000008 R_X86_64_RELATIVE 0000000000200810 0000002007d8 000200000006 R_X86_64_GLOB_DAT 0000000000000000 __gmon_start__ + 0 0000002007e0 000300000006 R_X86_64_GLOB_DAT 0000000000000000 _Jv_RegisterClasses + 0 0000002007e8 000400000006 R_X86_64_GLOB_DAT 0000000000000000 __cxa_finalize + 0 Relocation section '.rela.plt' at offset 0x450 contains 1 entries: Offset Info Type Sym. Value Sym. Name + Addend 000000200808 000400000007 R_X86_64_JUMP_SLO 0000000000000000 __cxa_finalize + 0Thus call_gmon_start, rather unsurprisingly, checks the value of __gmon_start__ and calls it if it is set. Presumably this is set as part of profiling and called during library initialisation -- but it is clearly not an initialiser by itself. The retq ends up popping a value off the stack and jumping to it, which in this case just happens to be 0x1 -- which we can confirm with gdb by putting a breakpoint on the first text address and examining the stack-pointer:
(gdb) x/2g $rsp 0x7fffffffe7d8: 0x0000000000000000 0x0000000000000001So that gives us our ultimate failure. Of course, if you're clever, you can get around this and initalise yourself correctly and actually make your shared-library do something when executed. The canonical example of this is libc.so itself:
$ /lib/x86_64-linux-gnu/libc-2.13.so GNU C Library (Debian EGLIBC 2.13-37) stable release version 2.13, by Roland McGrath et al. Copyright (C) 2011 Free Software Foundation, Inc. This is free software; see the source for copying conditions. ...You can trace through how this actually does work in the same way as we traced through why the trivial example doesn't work. If you wondering my opinion on executable-bits for shared-libraries; I would not give them execute permissions. I can't see it does anything but open the door to confusion. However, understanding exactly why the library segfaults the way it does actually ends up being a fun little tour around various parts of the toolchain!
Interesting stuff is happening again and I m doing a bit of travelling where I m not driving much, so I can write some blog posts. If this train stops bouncing quite so much!
I think most readers are interested in technology and collaborative work, so it makes sense to alternate those two themes most of the time. So that s what I ll aim for, probably a few posts each week for the next few weeks.
Let me know in the comments or our co-op s contract form if there s anything in particular you d like us to cover, else I ll start with my recent experiences installing Debian 6 on a new laptop and the fun of running a business at tax return time.
Update: I apologise for spamming all the nice people on Planet Debian with this unrelated post in foreign language it got mis-tagged and (although I tried classifying it correctly and removing it from the separate Debian-only RSS feed going to the Debian Planet) Planet software just keeps it around until admins get a spare moment to remove it manually. The basic idea of the post is that there is a e-government service by the Latvian government that provides any citizen a way to check what their pension would be, but it is kinda useless, because it calculates what you pension would be if you decided to retire right now. So in the post below I provide a few simple steps on how to estimate what the real pension could be (in todays currency values) if a person kept paying into the system at the same rate until the normal retirement age.
Latvija.lv ir r ks savas n kotnes pensijas prognoz anai, ta u tas ir, maigi izsakoties, bezj dz gs tiem, kas nepl no iet pensij tuv k gada laik .
tri apskatoties atteic gos likumus un noteikumus es nonacu pie lietder g kas metodes. Vispirms m s varam dro i ignor t pensiju indeks ciju un visas ar to saist t s pensiju kapit la p rr in anas vai iemaksu palielin anos ir skaidrs, ka gadiem ejot uz priek u b s kaut k ds infl cijas procents, kura d palielin sies algas, palielin sies iemaksas pensiju kapit l un tiks p rr in tas vis das izmaksas, ta u t pati infl cija tie i ietekm s ar to cik v rt gs b s katrs gala pensijas lats taj br d , kad pensija tiks sa emta, l dz ar to infl cijas efekts tiek neitraliz ts, ja m s v rt jam gala ciparu br a cenu kontekst t.i. ja gal san ks, ka pensija b s 400Ls, tad to vai tas ir daudz vai maz j dom br a cen s, nevis m inot izdom t k das b cenas p c 30 gadiem.
To visu emot v r pensijas apr ins stipri vienk r ojas:
sudo qemu-debootstrap --arch=armhf --foreign wheezy armhf http://ftp.de.debian.org/debian sudo LC_ALL=C chroot armhf /bin/bash echo 'deb http://ftp.de.debian.org/debian wheezy main' > /etc/apt/sources.list echo 'nameserver 8.8.8.8' > /etc/resolv.conf apt-get update apt-get install clangAlso, here are UDD queries which I prefer to those posted by RichiH. Note that they don t display all bugs, but ignore those which were created in the last 7 days.
Edward Tufte s Speech From Aaron Swartz s Memorial
Speech starts 41:00 [video cuts out in beginning]
We would then meet over the years for a long talk every now and then, and my responsibility was to provide him with a reading list, a reading list for life and then about two years ago Quinn had Aaron come to Connecticut and he told me about the four and a half million downloads of scholarly articles and my first question is, Why isn t MIT celebrating this? .
[Video cuts out again]
Obviously helpful in my career there, he then became president of the Mellon foundation, he then retired from the Mellon foundation, but he was asked by the Mellon foundation to handle the problem of JSTOR and Aaron. So I wrote Bill Bullen(sp?) an email about it, I said first that Aaron was a treasure and then I told a personal story about how I had done some illegal hacking and been caught at it and what happened. In 1962, my housemate and I invented the first blue box, that s a device that allows for free, undetectable, unbillable long distance telephone calls. And we got this up and played around with it and the end of our research came when we concluded what was the longest long distance call ever made, which was from Palo Alto to New York time-of-day via Hawaii, well during our experimentation, AT&T, on the second day it turned out, had tapped our phone and uh but it wasn t until about 6 months later when I got a call from the gentleman, AJ Dodge, senior security person at AT&T and I said, I know what you re calling about. and so we met and he said You what you are doing is a crime that would , you know all that. But I knew it wasn t serious because he actually cared about the kind of engineering stuff and complained that the tone signals we were generating were not the standard because they record them and play them back in the network to see what numbers they we were that you were trying to reach, but they couldn t break though the noise of our signal. The upshot of it was that uh oh and he asked why we went off the air after about 3 months, because this was to make long distance telephone calls for free and I said this was because we regarded it as an engineering problem and we made the longest long distance call and so that was it. So the deal was, as I explained in my email to Bill Bullen, that we wouldn t try to sell this and we were told, I was told that crime significance would pay a great deal for this, we wouldn t do any more of it and that we would turn our equipment over to AT&T, and so they got a complete vacuum tube isolator kit for making long distance phone calls. But I was grateful for AJ Dodge and I must say, AT&T that they decided not to wreck my life. And so I told Bill Bullen that he had a great opportunity here, to not wreck somebody s life, course he thankfully did the right thing.
Aaron s unique quality was that he was marvelously and vigorously different. There is a scarcity of that. Perhaps we can be all a little more different too.
Thank you very much.
It s been a long time I wanted to blog about my recent work on the Openstack packaging. I finally can find a bit of time to do it.
For those who don t know yet Openstack, it s a fairly recent (less than 3 years) cloud computing suite, which is becoming quite huge. If you plan on deploying a private cloud, you should definitively have a look into it.
When the Openstack project started, I immediately was interested. I started packaging the Cactus release (that is, the 3rd version of Openstack), created the Alioth group (at the exact moment when Alioth was migrated to new hardware which added some fun so lucky I was!), and began to work on the Debian version (Openstack used to be an Ubuntu only project). After some success in integrating some Debian specific patches into the Ubuntu packages, I left it a bit aside, and skipped completely the Diablo release (which was never uploaded in Debian). Then, I worked a bit on the Essex release before the freeze of Wheezy last June, together with other DDs (big up to Julien, Loic, Ghe, Sileth!).
I (re-)started serious packaging work for Openstack just right after Folsom (eg: version 2012.2) was released early last October. I literally worked days and nights on it, in order to provide more automation so that it could become more easy to install. Indeed, it used to be very complicated and painful, with lots of manual tasks to perform on the shell. It s hopefully a lot easier now, with most of the manual boring shell work replaced by debconf and scripts. But I m sure there s more that can be done still.
After 4 months of effort, I finally pressed the red button and uploaded everything at once to Debian Experimental since Openstack Essex (version 2012.1) is in Wheezy, I can t of course upload to SID unless Wheezy is out -. This represent 32 sources packages in total (some of them are already uploaded and approved by the FTP masters: sorry to give you so much more work guys ), and 104 binary packages (and counting ). So this isn t exactly small And I already have some fixes for what s currently waiting in the NEW queue: CVE fixes, missing build-depends which we ve found using the Jenkins server of eNovance who sponsors the packaging work, etc. I probably will post again here to announce when Folsom is completely approved by the FTP masters and reaches Debian experimental. In the mean while, it s available on GPLHost non-official repositories (see the howto linked bellow).
If you would like to test the latest Openstack release (called Folsom, or 2012.2, if you are following well ), you can read the quite verbose howto I wrote here:
https://wiki.debian.org/OpenStackHowto/Folsom
If you do test it (don t be afraid, it s not that hard ), I will be very happy to hear from you, and receive any feedback / critics you may have. Note that I would strongly recommend to use the Folsom release, rather than what is currently in Wheezy, for many reasons which would be too long to list in this blog post (let me still drop a few reasons here: less bugs, more automations and it s more easy to install).
Now, Openstack is a constantly evolving software, with so many companies and developers involved. I don t think my work on this beast will ever be finished. But it doesn t mater, as it s been quite some fun, and I will enjoy to do more.
The next release of Openstack is schedule for next April (they release at the same time as Ubuntu). I hope to be able to join the Portland summit, and see all the persons I have worked with over IRC and mailing list. This time, I hope to be able to release Openstack packages at the same time as the upstream source code is released (Debian packages have historically been lagging a few months behind). The pre-version is by the way already in Alioth.
Last, a (private) message for our famous cheese-master: please wait until everything leaves the NEW queue before bugging everyone with translations. We probably should have a serious talk about how to make it less redundant, easier to translate, and probably find a way to avoid duplication of messages across all packages.
Next round of phpMyAdmin 4.0 translation status report is coming.
So let's look at which translations are at 100% right now (no change this week):
There are also several languages which need just few strings to be complete:
As you can see, there is still lot of languages missing, this might be your opportunity to contribute to phpMyAdmin. Also you are welcome to translate phpMyAdmin 4.0 using translation server.
If your language is already fully translated and you want to help as well, you can translate our documentation as well.
Filed under: English Phpmyadmin 0 comments Flattr this!
I have just uploaded a new version of ghc-heap-view to Hackage that provides Evaluation state assertions in the module GHC.AssertNF.
Imagine you are writing a web application in Haskell that sports a global number-of-visitors counter in an IORef Int. For every request, you call modifyIORef (+1). Eventually, you notice your very popular web site to hog more and more memory. So you browse to the internal page that shows the counter, and you have to wait for a long time until you eventually see the result (or get a stack overflow). The reason: The applications of (+1) were not performed until you looked at the number; instead, a long chain of such computation first filled your heap and then your stack.
So you have learned the hard way that you might want to avoid space leaks, and want calculations to be done during the request that caused them, and want the IORef to always contain fully evaluated data. So you stumble about modifyIORef' in Data.IORef and indeed, this fixes your problem.
Later, you notice that you want to count POST and GET requests separately. You change the type to IORef (Int, Int) and call modifyIORef' (first (+1)) or modifyIORef' (second (+1)). And suddenly, the space leak is back (which you only notice after the next push to the real site, because your local tests never caused enough requests to make it noticeable). So you not only want to fix it, you also want to ensure that it does not break again.
In other words, you want to ensure the policy that values stored in an IORef are always in normal form. You achieve this with the following alternative to modifyIORef':
modifyIORef'Assert :: IORef a -> (a -> a) -> IO ()
modifyIORef'Assert ref f = do
x <- readIORef ref
let x' = f x
x' seq return ()
assertNF x'
writeIORef ref x'
Using this instead of modifyIORef' will print this warning to standard error output right the first time you call modifyIORef'Assert (first (+1)):
Parameter not in normal form: 2 thunks found: let x1 = (S# 1,S# 1) in _bh (_thunk x1 (_bco (S# 1)),_sel x1)(Otherwise, the program runs as usual.) So obviously, you need to use a strict variant of first (or strict pairs):
first' :: (a -> b) -> (a, c) -> (b, c) first' f (x,y) = let x' = f x; r = (x', y) in x' seq r seq rWith this, the warning goes away. Whenever you now change the type of the IORef or modify it in a too-lazy-way, you can be sure that you ll be warned about it, before the space leak itself becomes noticeable. In the production code, you might want to disable the check. For that, simply put disableAssertNF somewhere in your main function. Why is this better than just calling deepseq in modifyIORef'Assert? Because this way, the code still creates unwanted thunks that are then evaluated before storing them in the IORef, whereas with assertNF you are told about the thunks and can prevent them from being created in the first place. Also, assertNF does not add a type class constraint. This is just one example application for assertNF (and its variants assertNFNamed, which includes a name in the warning to better spot the cause, and $assertNFHere, which uses Template Haskell to include the current source code position in the warning), and I hope that there are more. If you happen to make use of it, I d like to hear your story.
Some little birds had already been whispering about it, but I didn't want to jinx it
and told myself I would wait with an announcement until the booksellers have (at least) placeholder pages. And as I learned
from Duncan Murdoch via email earlier today, at least
Chapters/Indigo had a page up, presumably scraped from the publisher page, so here it goes:
I have in fact handed a complete draft of my book about Seamless R and C++ Integration with Rcpp to Springer a few weeks ago. With a bit of luck on the production side, we could be seeing physical copies by May of a new title in their popular UseR! Series series.And to the slowly growing new Rcpp site, I have added a a formal page about the Rcpp book where one can find information about it, including a link to the Springer page, links to a few bookseller's pages --- as well as a few wonderfully flattering endorsements. Eventually, errata and other support material should be available via this page too. Can't wait til I hold a physical copy in hand...
This post is a quick status update on DebConf13, for those who aren t following the debconf-team mailing list.
As you may know, this year s DebConf will be held at Le Camp, in Vaumarcus, Switzerland. Our concept for this DebConf is to hold it in a natural environment, away from distractions. We hope you will enjoy spending time together with other Debian collaborators in this beautiful part of Switzerland, on the shore of Lake Neuch tel.
Dates
DebConf13 will take place from Sunday 11. August 2013 to Sunday 18. August 2013. (We will use Saturday 10. August 2013 to prepare the venue for the conference.)
You may notice that the dates don t cover two weeks like in the last few years (there is no separate DebCamp week). For budget reasons, current plans are to merge the two weeks activities into an 8 day period. If you think that s a pity, it s not too late to change it just join the fundraising team and start working quickly!
Travel
If you want to start arranging your travel to attend DebConf, some initial travel suggestions may be useful:
FOSDEM 2013 is over and it's time to look what interesting I have seen there on Sunday.
Sunday was supposed to start for me with L20N, but it was postponed to 13:00 as the presenters weren't on time. I could have used one more hour of sleep, but at least I spent some time on coding.
Detect merge conflicts in realtime was quite interesting talk, though I was pretty surprised that the conflict detecting does not at all care about underlying version control system, but does purely file based guesses.
The Hardening MySQL talk pretty much described why security in MySQL sucks and what you should do to make it secure. Quite good introduction to the topic, but not much new information for me.
Introduction of Firefox OS, was quite nice demo showing they have something working, though it had some problems with flaky network on FOSDEM. Looking forward to see phone being sold, though it will probably not be something I'd buy.
To add some fun, I've stayed on systemd, Two Years Later presentation, which gave some summary of what is currently in systemd and where it wants to go. Still it did not move systemd from category of "I don't care as long as it works".
Now followed delayed L20N talk - it showed new Mozilla's effort for localization. Which is quite powerful and has nice features, on the other side it put's quite more load to translators - now they would have to understand some basics of programming as well to be able to use the new features (or not so new ones as plurals). Their motivation is to remove localization effort from developers, but I'm not really convinced it will work nicely.
After some meetings and lunch, I went to LibreOffice: cleaning and re-factoring a giant code-base, which showed some challenges LibreOffice has to take and how they dealt with that. I think it's pretty great job done and I'm looking forward to new releases.
Being GNOME user, I could not skip Has the GNOME community gone crazy?. It of course tried to tell that they did not :-).
Last, but not least my friend Dieter from phpMyAdmin had talk Present and future of phpMyAdmin. He listed some of the new features, demoed 3.5 and 4.0 version (of course the demo of 4.0 version broke due to some caching). Even when the talk had quite unpleasant timing, it has attracted some people and they even asked few questions.
This years FOSDEM was again great and looking forward to be there next year.
Filed under: Debian English Phpmyadmin Suse Weblate 2 comments Flattr this!
Lots of exciting things are happening with and around Rcpp.
I just added a new page about Upcoming Events to the recently-created
Rcpp site. This events page has lots to cover: an upcoming talk at Columbia on March 8 (details still TBD),
a day-long workshop in New York on March 9, a possible participation at a CERN / ROOT conference in Switzerland on May
11-14, an upcoming talk in May in Milwaukee, and last but not least the tutorial by Romain and Hadley at UseR!
2013 in Spain. Phew!
With that, a few quick words about the
upcoming master class in New York.
It will be a full day, covering an introduction and motivation, details about the core data types, tools for working with and
and extending Rcpp and of course applications galore, including RcppArmadillo and RInside. I have done the same one day
class format a few times before, most recently (with Revolution Analytics) in San Francisco in late 2011, and also as a two-part seminar at UseR!
2012. This time, we plan on providing cloud-hosted RStudio instances for participants. Better still, RStudio's own JJ
Allaire will be on deck as well for RStudio --- and Rcpp Attributes --- questions.
Details and registration information for the New York class are at
this page.
Since some people asked for it, and since it's (unfortunately) fairly
nontrivial, here is the script that I'm using currently to transcode a
.dv file into webm:
#!/bin/bash set -e newfile=$(basename $1 .dv).webm wavfile=$(basename $1 .dv).wav normalfile=$(basename $1 .dv)-normal.wav normalfile=$(readlink -f $normalfile) oldfile=$(readlink -f $1) echo "Audio split" gst-launch-0.10 uridecodebin uri=file://$oldfile ! progressreport ! audioconvert ! audiorate ! wavenc ! filesink location=$wavfile echo "Audio normalize" sox --norm $wavfile $normalfile echo "Pass 1" gst-launch-0.10 webmmux name=mux ! fakesink \ uridecodebin uri=file://$oldfile name=demux \ demux. ! ffmpegcolorspace ! deinterlace ! vp8enc multipass-cache-file=/tmp/vp8-multipass multipass-mode=1 threads=2 ! queue ! mux.video_0 \ demux. ! progressreport ! audioconvert ! audiorate ! vorbisenc ! queue ! mux.audio_0 echo "Pass 2" gst-launch-0.10 webmmux name=mux ! filesink location=$newfile \ uridecodebin uri=file://$oldfile name=video \ uridecodebin uri=file://$normalfile name=audio \ video. ! ffmpegcolorspace ! deinterlace ! vp8enc multipass-cache-file=/tmp/vp8-multipass multipass-mode=2 threads=2 ! queue ! mux.video_0 \ audio. ! progressreport ! audioconvert ! audiorate ! vorbisenc ! queue ! mux.audio_0 rm $wavfile $normalfileIt's fairly non-optimal, because many of these command-line A/V tools are either fairly badly documented, or have a horrific interface, or something similar. avconv is supposed to be able to do audio normalisation, but I haven't been able to figure out exactly how it's done. The options that are specified in the manpage seemingly have no effect. gstreamer has a 'ReplayGain' plugin which can do audio normalisation. Audio normalisation requires two passes; I could in theory just add the element in the gstreamer first-pass pipeline, add a '-t' to the gst-launch-0.10 invocation, and parse out the required gain value that I can then add to the second-pass pipeline. However, the elements that I can then pass that gain value to either have a different range for the gain (rgvolume's fallback-gain parameter) or expect a completely different unit of values with no obvious way to translate between the two (the 'volume' element's 'volume' parameter expects a multiplier, the replaygain plugin produces a dB value; a simple conversion seems wrong as it produces a value way out of range). The 'volume' element reportedly has an interface that takes a dB value, except you can't reach it from gst-launch-0.10. Bummer. So instead we split out the audio, do normalisation in sox, and mux that back in during the second pass of the video transcoding. Sox is good. Sox is easy. If all A/V command-line tools were like sox, I would be smiling now. What's missing from the above script is how to throw away the start or end of a file in case there's several minutes of uninterestingness there. This is easiest (in my experience) with avconv's -t and -ss options. I suspect gstreamer is capable of doing that, too; but I haven't figured out how, and this works. You may want to play with some of options of the vp8enc element. For instance, the threads= value can be increased (or decreased) depending on how many cores your transcoding machine has. You could probably also add a "target-bitrate" value if you find the quality is too low or the system uses too much diskspace. VP8 has several more options; the documentation lists them all. If you've done video recordings for a devroom at FOSDEM outside of the FOSDEM video team, note that FOSDEM is more than willing to host the videos. To do so, the easiest way is if the videos are online somewhere (this does not have to be a public system) that we can wget or scp them from; if you've done that, contact me with the location to the files and we'll put them on the FOSDEM video server.
After 27 hours of travel, I'm finally back from my
first Linux.conf.au.
This was a great conference! Some of my favorite highlights:
git pull --hard
and pulled GitHub's sever room across the street.")
 |
Next.