Search Results: "spa"

13 November 2015

Daniel Pocock: Building teams around SIP and XMPP in Debian and Fedora

I've recently started a discussion on the Fedora devel mailing list about building a team to collaborate on RTC services (SIP, XMPP, TURN and WebRTC) for the Fedora community. We already started a similar team within Debian. This isn't only for developers or package maintainers and virtually anybody with a keen interest in free software can help. Testing different softphones and putting screenshots on the wiki can help a lot (the Debian wiki already provides some examples). The FedRTC.org site is not intended to be an advertisement for my web design skills and anybody with expertise in design would be very welcome to contribute. Teamwork in this endeavor can provide many benefits:
  • Sharing knowledge about RTC, for use within our communities and also for other communities using the free and open technology
  • Engaging with collaborators who are not involved in packaging teams, for example, the Debian RTC team has also had interest from upstream developers who are not on other Debian or Fedora mailing lists
  • Minimizing the effort required by the system administrators (the DSA team in Debian or Infrastructure team in Fedora) by triaging user problems and planning and testing any proposed changes.
  • Freeing up developer time to work on new features, such as the exciting work I'm doing on telepathy-resiprocate.
There are also many opportunities for project work that go beyond traditional packaging responsibilities. Wouldn't it be interesting to find ways to integrate the publish/subscribe capabilities of SIP and XMPP with the Fedmsg infrastructure? Bringing XMPP to FedoraProject.org We recently launched XMPP for debian.org and it would not be hard to replicate for FedoraProject.org users. Sure, some people are happy running their own XMPP servers. There are just as many people who prefer to focus on development and have something like XMPP provided for them. With the strong emphasis on building a roster/buddy-list, XMPP can also help to facilitate long-term engagement in the community and users may identify more closely with the project. I haven't offered XMPP on the FedRTC.org trial service because it would be inconvenient for people to migrate buddy lists to the FedoraProject.org domain when the service is officially adopted. Collaboration across communities There are various other places where we can share knowledge between teams in different communities and people are invited to participate. The Free-RTC mailing list is a great place to discuss free RTC strategies and initiatives. The XMPP operators mailing list provides a forum to discuss operational issues in the XMPP space, such as keeping out the spammers. Would you like to participate? Please consider joining some of the mailing lists I've mentioned, replying to the thread on the Fedora devel mailing list, volunteering for the Debian RTC team or emailing me personally.

Raphaël Hertzog: Freexian s report about Debian Long Term Support, October 2015

A Debian LTS logoLike each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In September, 85.50 work hours have been dispatched among 8 paid contributors. Their reports are available: Evolution of the situation November crossed a new record with 114.5 hours funded. This is mainly thanks to our first Platinum sponsor: TOSHIBA (through Toshiba Software Development Vietnam). They don t know yet if they can sponsor us in the long term (they hope so), but it s still a nice news as we jumped from 50% to 65% of the objective of the equivalent of a full-time position with a single new sponsor. Currently no change is expected for next month as we don t have any other new sponsor in the process of joining us. We still need more support to be able to support all the packages we could not afford to support during the squeeze cycle. We are currently discussing which package we can or cannot support on the LTS list, see the thread Unsupported packages for Wheezy LTS for the current situation. In terms of security updates waiting to be handled, the situation is close to last month: the dla-needed.txt file lists 21 packages awaiting an update (6 more than last month), the list of open vulnerabilities in Squeeze shows about 23 affected packages in total (exactly like last month). Thanks to our sponsors The new sponsors are in bold.

No comment Liked this article? Click here. My blog is Flattr-enabled.

12 November 2015

Carl Chenet: db2twitter 0.2 released

db2twitter 0.2 was just released! If you missed my last post about it, db2twitter automatically extracts fields from your database, use them to feed a template of tweet and send the tweet. db2twitter is developed by and run for LinuxJobs.fr, the job board of th french-speaking Free Software and Opensource community. banner-linuxjobs-small The main new feature allows to send tweets only during authorized days/hours with the following content in your db2twitter.ini file :
[timer]
days=mon-fri,
hours=0-11,14-17,
Given the fact db2twitter will mostly be launched via cron, it allows sending tweets detected since the last db2twitter run during user-defined hours. And that s cool! db2twitter is coded in Python 3.4, uses SQlAlchemy (see supported database types) and Tweepy. The official documentation is available on readthedocs.

11 November 2015

Kees Cook: evolution of seccomp

I m excited to see other people thinking about userspace-to-kernel attack surface reduction ideas. Theo de Raadt recently published slides describing Pledge. This uses the same ideas that seccomp implements, but with less granularity. While seccomp works at the individual syscall level and in addition to killing processes, it allows for signaling, tracing, and errno spoofing. As de Raadt mentions, Pledge could be implemented with seccomp very easily: libseccomp would just categorize syscalls. I don t really understand the presentation s mention of Optional Security , though. Pledge, like seccomp, is an opt-in feature. Nothing in the kernel refuses to run unpledged programs. I assume his point was that when it gets ubiquitously built into programs (like stack protector), it s effectively not optional (which is alluded to later as comprehensive applicability ~= mandatory mitigation ). Regardless, this sensible (though optional) design gets me back to his slide on seccomp, which seems to have a number of misunderstandings: OpenBSD has some interesting advantages in the syscall filtering department, especially around sockets. Right now, it s hard for Linux syscall filtering to understand why a given socket is being used. Something like SOCK_DNS seems like it could be quite handy. Another nice feature of Pledge is the path whitelist feature. As it s still under development, I hope they expand this to include more things than just paths. Argument inspection is a weak point for seccomp, but under Linux, most of the arguments are ultimately exposed to the LSM layer. Last year I experimented with creating a seccomp LSM for path matching where programs could declare whitelists, similar to standard LSMs. So, yes, Linux could match this API on seccomp . It d just take some extensions to libseccomp to implement pledge(), as I described at the top. With OpenBSD doing a bunch of analysis work on common programs, it d be excellent to see this usable on Linux too. So far on Linux, only a few programs (e.g. Chrome, vsftpd) have bothered to do this using seccomp, and it could be argued that this is ultimately due to how fine grained it is.

2015, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

Ritesh Raj Sarraf: apt-offline 1.7

Hello World, In this part of the world, today is a great day. Today is Diwali - the festival of lights On this day, I am very happy to announce the release of apt-offline, version 1.7. This release brings in a large number of fixes and is a recommended update. Thanks to Bernd Dietzel for uncovering the shell injection bug which could be exploited by carefully crafting the signature file. Since apt-offline could be run as 'root', this one was an important bug. Also thanks to him for the fix. During my tests, I also realized that apt-offline's --install-src-packages implementation had broken over time. --install-src-packages option can be useful to users who would like the offline ability to download a [Debian] source package, along with all its Build Dependencies. For further details on many of the other fixes, please refer to the git repository at the homepage. Packages for Debian (and derivatives) are already in the queue. Wishing You and Your Loved Ones a Very Happy Diwali.

Categories:

Keywords:

Images:

Like:

10 November 2015

Norbert Preining: Music check: Google versus Apple Is that all? You can do better, Google!

Ok, I have been an iPhone user since I moved to Japan and got my first smart-phone ever. First a 3s, then a 4s, then a 5s that I dropped into the toilet, so I switched the SIM back into the 4s to have a working phone. Furthermore, I am a heavy music listener and used iTunes Radio for 3 months. Since I am planning to change phones to an Android phone (being fed up with Apple s super-closed environment), I tried out Google Music Plus for about 3 months, too. Here is my verict Google Music is a big pain, far from iTunes in comfort and user friendliness. music-check-google-vs-apple My move to Android is not very much in danger. General description of the service In principle, iTunes Radio and Google Music Subscription do the same things: There are slight differences, that often create confusion, especially with Apple s services: If you are only signed in, you can listen to music stations and can skip some music (limited), but you cannot listen to arbitrary songs from the whole Apple Music library. Then there is another service from Apple, called iTunes Match, that only allows you to upload your music library to the cloud, but other than that again only listening to normal radio stations. Google Music is much simpler, there are only two options: By default it is free to have your music (up to 25000 songs) in the cloud, but if you want to listen to radio stations or any other music in the Google music store, you need a subscription. Prizes and features overview:
Apple Google
No subscription iTunes Match iTunes Radio No subscription With subscription
Prize 0 25$/year 10$/month 0 10$/month
(early access 8$)
Cloud space 0 100000 25000
Extra service Radio stations Radio stations Radio/all music Nothing Radio/all music
Music on the go the application Let us first consider the applications provided to listen to music on your smart phone: Apple Music can of course only be used on Apple devices, and uses the built-in Music application. Start up time is about a few seconds after a cold boot (all on my 4s), and music streaming starts with hardly any delay. Responsiveness is good, and the user interface is clear and easy. Google s application is available on iOS and Android. I have tested the latest version on iOS, but it is a pain in the butt: Managing your library Here iTunes is the way to go, bit of a pain when using Linux, but there are other reasons I have a Windows installation in parallel, so I don t mind to boot now and then into Windows. iTunes gives you very powerful tools to change all kind of data. Google gives you a few options: Use your iTunes library in this case all your playlists and ratings (but see later) are also uploaded. I am not sure what happens if I retag a song in iTunes, or change anything else in there. I guess the song will not be re-uploaded, but who knows for sure. Furthermore, you can edit your library via the web interface, but this is rather poor. Searching your library Over times your library grows and there are hundreds if not thousands of artists. So you want to search them. The natural way would be to scroll to the first letter of the artist you are searching, and then look it up. Well, that works perfectly in both Apple s and Google s application unless you are having artists written in some strange script like Japanese or Korean. iTunes allows you to set a field called something like Artist name for sorting , which allows me to put for example or into the artist field, and into the sort field Tomokawa Kazuki and Kim Doo Soo . This way I find the artists in the correct place. Google on the other hand uses simple Unicode order sorting how could you do that? This is simply plain wrong, and everyone should know that by now. Japanese people will never be able to find anything in their list. And in contrast to Goole Contacts Google Music does not support phonetic name fields (similar to the order sort artist) for artists. What I had to do now is to rename all the artists to include first the phonetic name, followed by the proper name, like in Tomokawa Kazuki ( ) . Something I strongly detest! Radio stations I might have a slightly peculiar music taste, but the radio stations mixed by Google are simply a pain. The reason is easy to explain: I live in Japan, and so what Google does is mixing about 80% of J-Pop into the radio, all those happy yodeling girlies I really dislike (see my Anti J-Pop campaign for alternatives yes, they do exist also in Japan!). iTunes radio is here more relaxed it seems. I appreciate Google s trial to cater to local (dis)taste, but besides voting down each and every song I hear I don t see any other option. And honestly, I cannot go through all this voting down without dying from pain inflicted by J-Pop. Other than this, the two radio stations are probably more or less the same but as I said, due to the local colorit it is hard to compare. Rating Ohhh, what a dire point. So there you have you well curated iTunes library with 5-star ratings. I used the ratings in a way that those songs I like get 1 star, those I even like more get 2 stars, and my absolute favorites got 5 stars. I didn t do any negative ratings. iTunes/iOS Music app allows you to easily adjust rating, and they are synced between devices. All as you would expect. Now for Google they did have a 5-star rating system at some time, but:
Thank you for your feedback. We ve decided to remove the 5-star Rating lab. This decision wasn t made lightly, but Thumbs Up/Down is integral for the future vision for Play Music and will be a central design point for our future releases. Please note that we ll continue to store your ratings that you ve set via the star lab or via iTunes, and we ll translate them to thumbs up/down (1-2 stars = thumbs down, 4-5 stars = thumbs up).
Here we go with the move to a new design they threw away the 5-star system, without any reason but integral for future vision rubbish sales speak. Besides being a very very poor rating system to have only an up and down (good-bad-don t care), the translation from my rating system to the thumbs up-down is just plain wrong. <rant>
Why on earth is the current movement to reduce functionality and rob users of control? Gnome 3 is the prime example of how we stupiditize users by taken any freedom from them in the name of simple design. Google now does the same. I am so sick of getting patronized this way, so Gnome3 was completely banished from my computer and replaced by Cinnamon, which uses the same (good) underlying technique, but takes users seriously!
</rant> Overall verdict As much as I would like to see Google Music a valid alternative to iTunes, by now it makes the impression of a quickly hacked together rotten piece of code that tries to get a share of the market without providing equivalent service. Google is using its market presence and convenience to convert people, not features and quality. I can only hope that this changes in the future. That still leaves me with the question move to Android or remain with iOS

9 November 2015

Vasudev Kamath: Forwarding host port to service in container

Problem I've lxc container running distcc daemon and I would like to forward the distcc traffic coming to my host system to container.
Solution Following simple script which uses iptable did the job.
 #!/bin/sh
 set -e
usage()  
    cat <<EOF
    $(basename $0) [options] <in-interface> <out-interface> <port> <destination>
    --clear           Clear the previous rules before inserting new
                      ones
    --protocol        Protocol for the rules to use.
    in-interface      Interface on which incoming traffic is expected
    out-interface     Interface to which incoming traffic is to be
                      forwarded.
    port              Port to be forwarded. Can be integer or string
                      from /etc/services.
    destination       IP and port of the destination system to which
                      traffic needs to be forwarded. This should be in
                      form <destination_ip:port>
(C) 2015 Vasudev Kamath - This program comes with ABSOLUTELY NO
WARRANTY. This is free software, and you are welcome to redistribute
it under the GNU GPL Version 3 (or later) License
EOF
 
setup_portforwarding ()  
    local protocol="$1"
    iptables -t nat -A PREROUTING -i $IN_INTERFACE -p "$protocol" --dport $PORT \
           -j DNAT --to $DESTINATION
    iptables -A FORWARD -p "$protocol" -d $ DESTINATION%%\:*  --dport $PORT -j ACCEPT
    # Returning packet should have gateway IP
    iptables -t nat -A POSTROUTING -s $ DESTINATION%%\:*  -o \
           $IN_INTERFACE -j SNAT --to $ IN_IP%%\/* 
 
if [ $(id -u) -ne 0 ]; then
    echo "You need to be root to run this script"
    exit 1
fi
while true; do
    case "$1" in
      --clear)
          CLEAR_RULES=1
          shift
          ;;
      --protocol --protocol=*?)
      if [ "$1" = "--protocol" -a -n "$2" ];then
          PROTOCOL="$2"
          shift 2
      elif [ "$ 1#--protocol= " != "$1" ]; then
          PROTOCOL="$ 1$--protocol= "
          shift 1
      else
          echo "You need to specify protocl (tcp udp)"
          exit 2
      fi
      ;;
      *)
          break
          ;;
    esac
done
if [ $# -ne 4 ]; then
    usage $0
    exit 2
fi
IN_INTERFACE="$1"
OUT_INTERFACE="$2"
PORT="$3"
DESTINATION="$4"
# Get the incoming interface IP. This is used for SNAT.
IN_IP=$(ip addr show $IN_INTERFACE \
             perl -nE '/inet\s(.*)\sbrd/ and print $1')
if [ -n "$CLEAR_RULES" ]; then
    iptables -t nat -X
    iptables -t nat -F
    iptables -F
fi
if [ -n "$PROTOCOL" ]; then
    setup_portforwarding $PROTOCOL
else
    setup_portforwarding tcp
    setup_portforwarding udp
fi
Coming to systemd-nspawn I see there is --port option which takes argument in form proto:hostport:destport where proto can be either tcp or udp, hostport and destport are number from 1-65535. This option assumes private networking enabled in the container. I've not tried this option yet but that simplifies quite a lot of thing, its like -p switch used by docker.

Vasudev Kamath: Taming systemd-nspawn for running containers

I've been trying to run containers using systemd-nspawn for quite some time. I was always bumping to one or other dead end. This is not systemd-nspawn's fault, rather my impatience stopping me from reading manual pages properly lack of good tutorial like article available online. Compared to this LXC has a quite a lot of good tutorials and howto's available online. This article is my effort to create a notes putting all required information in one place.
Creating a Debian Base Install First step is to have a minimal Debian system some where on your hard disk. This can be easily done using debootsrap. I wrote a custom script to avoid reading manual every time I want to run debootstrap. Parts of this script (mostly packages and the root password generation) is stolen from lxc-debian template provided by lxc package.
#!/bin/sh
set -e
set -x
usage ()  
    echo "$ 0##/*  [options] <suite> <target> [<mirror>]"
    echo "Bootstrap rootfs for Debian"
    echo
    cat <<EOF
    --arch         set the architecture to install
    --root-passwd  set the root password for bootstrapped rootfs
    EOF
 
# copied from the lxc-debian template
packages=ifupdown,\
locales,\
libui-dialog-perl,\
dialog,\
isc-dhcp-client,\
netbase,\
net-tools,\
iproute,\
openssh-server,\
dbus
if [ $(id -u) -ne 0 ]; then
    echo "You must be root to execute this command"
    exit 2
fi
if [ $# -lt 2 ]; then
   usage $0
fi
while true; do
    case "$1" in
        --root-passwd --root-passwd=?*)
            if [ "$1" = "--root-passwd" -a -n "$2" ]; then
                ROOT_PASSWD="$2"
                shift 2
            elif [ "$1" != "$ 1#--root-passwd= " ]; then
                ROOT_PASSWD="$ 1#--root-passwd= "
                shift 1
            else
                # copied from lxc-debian template
                ROOT_PASSWD="$(dd if=/dev/urandom bs=6 count=1 2>/dev/null base64)"
                ECHO_PASSWD="yes"
            fi
            ;;
        --arch --arch=?*)
            if [ "$1" = "--arch" -a -n "$2" ]; then
                ARCHITECTURE="$2"
                shift 2
            elif [ "$1" != "$ 1#--arch= " ]; then
                ARCHITECTURE="$ 1#--arch= "
                shift 1
            else
                ARCHITECTURE="$(dpkg-architecture -q DEB_HOST_ARCH)"
            fi
            ;;
        *)
            break
            ;;
    esac
done
release="$1"
target="$2"
if [ -z "$1" ]   [ -z "$2" ]; then
    echo "You must specify suite and target"
    exit 1
fi
if [ -n "$3" ]; then
    MIRROR="$3"
fi
MIRROR=$ MIRROR:-http://httpredir.debian.org/debian 
echo "Downloading Debian $release ..."
debootstrap --verbose --variant=minbase --arch=$ARCHITECTURE \
             --include=$packages \
             "$release" "$target" "$MIRROR"
if [ -n "$ROOT_PASSWD" ]; then
    echo "root:$ROOT_PASSWD"   chroot "$target" chpasswd
    echo "Root password is '$ROOT_PASSWRD', please change!"
fi
It just gets my needs done, if you don't like it feel free to modify or use debootstrap directly. !NB Please install dbus package in the minimal base install, otherwise you will not be able to control the container using machinectl
Manually Running Container and then persisting it Next we need to run the container manually. This is done by using following command.
systemd-nspawn -bD   /path/to/container --network-veth \
     --network-bridge=natbr0 --machine=Machinename
--machine option is not mandatory, if not specified systemd-nspawn will take the directory name as machine name, and if you have characters like - in the directory name it translates to hexcode x2d and controlling container with name becomes difficult. --network-veth specifies the systemd-nspawn to enable virtual ethernet based networking and --network-bridge tells the bridge interface on host system to be used by systemd-nspawn. These options together constitutes private networking for container. If not specified container can use host systems interface there by removing network isolation of container. Once you run this command container comes up. You can now run machinectl to control the container. Container can be persisted using following command
machinectl enable container-name
This will create a symbolic link of /lib/systemd/system/systemd-nspawn@service to /etc/systemd/system/machine.target.wants/. This allows you to start or stop container using machinectl or systemctl command. Only catch here is your base install should be in /var/lib/machines/. What I do in my case is create a symbolic link from my base container to /var/lib/machines/container-name. !NB Note that symbolic link name under /var/lib/machines should be same as the container name you gave using --machine switch or the directory name if you didn't specify --machine
Persisting Container Networking We did persist the container in above step, but this doesn't persist the networking options we provided in command line. systemd-nspawn@.service provides following command to invoke container.
ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth --settings=override --machine=%I
To persist the bridge networking configuration we did in command line, we need the help of systemd-networkd. So first we need to enable the systemd-networkd.service on both container and the host system.
systemctl enable systemd-networkd.service
Now inside the container, interfaces will be named as hostN. Depending on how many interfaces we have N increments. In our example case we had single interface, hence it will named as host0. By default network interfaces will be down inside container, hence systemd-networkd is needed to put it up. We put the following in /etc/systemd/network/host0.network file inside the container.
[Match]
Name=host0
[Network]
Description=Container wired interface host0
DHCP=yes
And in the host system we just configure the bridge interface using systemd-nspawn. I put following in natbr0.netdev in /etc/systemd/network/
[NetDev]
Description=Bridge natbr0
Name=natbr0
Kind=bridge
In my case I already had configured the bridge using /etc/network/interfaces file for lxc. I think its not really needed to use systemd-networkd in this case. Since systemd-networkd doesn't do anything if network / virtual device is already present I safely put above configuration and enabled systemd-networkd. Just for the notes here is my natbr0 configuration in interfaces file.
auto natbr0
iface natbr0 inet static
   address 172.16.10.1
   netmask 255.255.255.0
   pre-up brctl addbr natbr0
   post-down brctl delbr natbr0
   post-down sysctl net.ipv4.ip_forward=0
   post-down sysctl net.ipv6.conf.all.forwarding=0
   post-up sysctl net.ipv4.ip_forward=1
   post-up sysctl net.ipv6.conf.all.forwarding=1
   post-up iptables -A POSTROUTING -t mangle -p udp --dport bootpc -s 172.16.0.0/16 -j CHECKSUM --checksum-fill
   pre-down iptables -D POSTROUTING -t mangle -p udp --dport bootpc -s 172.16.0.0/16 -j CHECKSUM --checksum-fill
Once this is done just reload the systemd-networkd and make sure you have dnsmasq or any other DHCP server running in your system. Now the last part is to tell systemd-nspawn to use the bridge networking interface we have defined. This is done using container-name.nspawn file. Put this file under /etc/systemd/nspawn folder.
[Exec]
Boot=on
[Files]
Bind=/home/vasudev/Documents/Python/upstream/apt-offline/
[Network]
VirtualEthernet=yes
Bridge=natbr0
Here you can specify networking, and files mounting section of the container. For full list please refer the systemd.nspawn manual page. Now all this is done you can happily do
machinectl start container-name
#or
systemctl start systemd-nspawn@container-name
Resource Control Now all things said and done, one last part remains. Yes what is the point if we can't control how much resource does the container use. Atleast it is more important for me, because I use old and bit low powered laptop. Systemd provides way to control the resource using Control interface. To see all the the interfaces exposed by systemd please refer systemd.resource-control manual page. The way to control the resource is using systemctl. Once container starts running we can run following command.
systemctl set-property container-name CPUShares=200 CPUQuota=30% MemoryLimit=500M
The manual page does say that these things can be put under [Slice] section of unit files. Now I don't have clear idea if this can be put under .nspawn files or not. For the sake of persisting the container I manually wrote the service file for container by copying systemd-nspawn@.service and adding [Slice] section. But if I don't know how to find out if this had any effect or not. If some one knows about this please share your suggestions to me and I will update this section with your provided information.
Conclussion All in all I like systemd-nspawn a lot. I use it to run container for development of apt-offline. I previously used lxc where all can be controlled using a single config file. But I feel systemd-nspawn is more tightly integrated with system than lxc. There is definitely more in systemd-nspawn than I've currently figured out. Only thing is its not as popular as other alternatives and definitely lacks good howto documentation.For now only way out is dig the manual pages, scratch your head, pull your hair out and figure out new possibilities in systemd-nspawn. ;-)

7 November 2015

Mart n Ferrari: Tales from the SRE trenches: Coda

This is the fifth -and last- part in a series of articles about SRE, based on the talk I gave in the Romanian Association for Better Software. Previous articles can be found here: part 1, part 2, part 3, and part 4.

Other lessons learned Apart from these guidelines for SRE I have described in previous posts, there were a few other practices I learned at google that I believe could benefit most operations teams out there (and developers too!).

Automate everything It's worth repeating, it is not sustainable to do things manually. When your manual work grows linearly with your usage, either your service is not growing enough, or you will need to hire more people than there are available in the market.

Involve Ops in systems design Nobody knows better than Ops the challenges of a real production environment, getting early feedback from your operations team can save you a lot of trouble down the road. They might give you insights about scalability, capacity planning, redundancy, and the very real limitations of hardware.

Give Ops time and space to innovate If there is so much emphasis on limiting the amount of operational load put on SRE, that freed time is not only used for creating monitoring dashboards or writing documentation. As I said in the beginning, SRE is an engineering team. And when engineers are not running around putting fires off all day long, they get creative. Many interesting innovations come from SREs even when there is no requirement for them.

Write design documents Design documents are not only useful for writing software. Fleshing out the details of any system on paper is a very useful way to find problems early in the process, communicate with your team exactly what are you building, and convince them why it is a great idea.

Review your code, version your changes This is something that is ubiquitous at Google, and not exclusive to SRE: everything is committed into source control: big systems, small scripts, configuration files. It might not seem worth it at the beginning, but having complete history of all your production environment is invaluable. And with source control, another rule comes into play: no change is committed without first being reviewed by a peer. It can be frustrating having to find a reviewer and wait for the approval for every small change, and some times reviews will be suboptimal, but once you get used to it, the benefits greatly out-weights any annoyances.

Before touching, measure Monitoring is not only for sending alerts, it is an essential tool of SRE. It allows you to gauge your availability, evaluate trends, forecast growth, perform forensic analysis. Also very importantly, it should give you the data needed to take decisions based on reality and not on guesses. Before optimising, find really where the bottleneck is; do not decide which database instance to use until you saw the utilisation of the past few months, etc. And speaking of that... We really need to have a chat about your current monitoring system. But that's for another day.
I hope I did not bore you too much with these walls of text! Some people told me directly they were enjoying these posts, so soon I will be writing more about related topics. In particular I would like to write about modern monitoring and Prometheus. I would love to hear your comments, questions, and criticisms. Comment

Andrew Cater: Debconf, ARM, Cambridge - 1250 7 November

Lunchtime - full talks over for the minute. Lightning talks may happen soon-ish.

Isolated knots of people chatting: Video Sprint folk chatting about how much can be done in time for Debconf 2016 in South Africa in the break out room.

Three or four people on laptops immediately behind me. Activity never stops

Thanks go to ARM for the venue in a good auditorium space, and also to the other sponsors who have sponsored our lunches or other expenses - and the staff here on the weekend / out of hours which I would guess is unusual.

Andrew Cater: Mini Debconf 2015 ARM Cambridge - ARM, 7 November

Belated thanks to the ARM folk who arranged car park spaces for me for Thursday and Friday. I'm mobility impaired so this made a huge difference.

Today is the more open day - not so much programming, more presentations and talks. Also being live streamed. http://video.debconf.org:8000/minidebconf.ogv

Schedule is slightly changed but the wiki is correct

The auditorium is pretty full even at 0925

Thanks as ever to front desk and the ARM security staff who have wekcined ** welcomed us. Here's to a good day

** typing on laptop balanced on lap

6 November 2015

Ian Campbell: Cambridge mini-Debconf

I am currently attending the mini-Debconf being held in space generously provided by ARM's offices in Cambridge, UK. Thanks to ARM and the other sponsors for making this possible. Yesterday I made a pass through the bug list for the Xen packages. According to the replies I have received from the BTS I looked at and acted on: Phew! Today I expect more of the same, starting with seeing where the new information in #799122 takes me.

Matthew Garrett: Why improving kernel security is important

The Washington Post published an article today which describes the ongoing tension between the security community and Linux kernel developers. This has been roundly denounced as FUD, with Rob Graham going so far as to claim that nobody ever attacks the kernel.

Unfortunately he's entirely and demonstrably wrong, it's not FUD and the state of security in the kernel is currently far short of where it should be.

An example. Recent versions of Android use SELinux to confine applications. Even if you have full control over an application running on Android, the SELinux rules make it very difficult to do anything especially user-hostile. Hacking Team, the GPL-violating Italian company who sells surveillance software to human rights abusers, found that this impeded their ability to drop their spyware onto targets' devices. So they took advantage of the fact that many Android devices shipped a kernel with a flawed copy_from_user() implementation that allowed them to copy arbitrary userspace data over arbitrary kernel code, thus allowing them to disable SELinux.

If we could trust userspace applications, we wouldn't need SELinux. But we assume that userspace code may be buggy, misconfigured or actively hostile, and we use technologies such as SELinux or AppArmor to restrict its behaviour. There's simply too much userspace code for us to guarantee that it's all correct, so we do our best to prevent it from doing harm anyway.

This is significantly less true in the kernel. The model up until now has largely been "Fix security bugs as we find them", an approach that fails on two levels:

1) Once we find them and fix them, there's still a window between the fixed version being available and it actually being deployed
2) The forces of good may not be the first ones to find them

This reactive approach is fine for a world where it's possible to push out software updates without having to perform extensive testing first, a world where the only people hunting for interesting kernel vulnerabilities are nice people. This isn't that world, and this approach isn't fine.

Just as features like SELinux allow us to reduce the harm that can occur if a new userspace vulnerability is found, we can add features to the kernel that make it more difficult (or impossible) for attackers to turn a kernel bug into an exploitable vulnerability. The number of people using Linux systems is increasing every day, and many of these users depend on the security of these systems in critical ways. It's vital that we do what we can to avoid their trust being misplaced.

Many useful mitigation features already exist in the Grsecurity patchset, but a combination of technical disagreements around certain features, personality conflicts and an apparent lack of enthusiasm on the side of upstream kernel developers has resulted in almost none of it landing in the kernels that most people use. Kees Cook has proposed a new project to start making a more concerted effort to migrate components of Grsecurity to upstream. If you rely on the kernel being a secure component, either because you ship a product based on it or because you use it yourself, you should probably be doing what you can to support this.

Microsoft received entirely justifiable criticism for the terrible state of security on their platform. They responded by introducing cutting-edge security features across the OS, including the kernel. Accusing anyone who says we need to do the same of spreading FUD is risking free software being sidelined in favour of proprietary software providing more real-world security. That doesn't seem like a good outcome.

comment count unavailable comments

Norbert Preining: Japanese TeX User Meeting Support for Japanese in TeX (Live) present state and future?

Tomorrow the Japanese TeX User Meeting 2015 will be held in Tokyo. Since I have come to Japan I have attended more or less every year this meeting, and in 2013 we could get the (international) TeX User Group Meeting to Tokyo and had a joint conference TUG 2013, which was a great success. (update 2015-11-09: slides added at the end) Japanese-TeX
During tomorrows meeting I will give a talk on Support for Japanese in TeX (Live) present state and future?:
The support for typesetting Japanese in TeX, in particular TeX Live, has been improved over the last few years considerably, so that at the current point we can ask ourselves what is next? This talk reviews a bit of the history of Japanese typesetting support in TeX Live, and discusses future development directions.
For those interested, here is a bit of an extended abstract. With a set of Japan specific TeX engines, the Japanese TeX Users always had a hard time setting up a proper TeX environment. Besides compiling the programs for each new version again, users had to install packages from a variety of sources, which often involved complicated routines. Fortunately, due to the excellent support on the TeX Q&A these difficulties were manageable. Compiling the Japanese TeX engines was first part of the pteTeX project, which adjusted teTeX to include also the required programs. Compilation was done by obtaining the source code of teTeX, plus patches that implement the Japanese TeX engines and support in driver programs. After the end of development of teTeX, TeX Live became the main TeX distribution system, and ptexlive was born to adjust the TeX Live sources to include the Japanese TeX engines. But fundamentally the same procedure remain: Get the source code, patch in additional functionality, compile and install. The last five years have seen a step-wise inclusion of various Japan-related programs in the main TeX Live source repository. Starting with pTeX (and pbibtex etc) and adjustments to dvipdfmx, followed by upTeX, and later on also (u)pmetapost, the main TeX Live source repository now contains all (with the exception of pxdvi) programs related to the Japanese TeX environment. At the same time, developers of various packages started to upload their packages to CTAN, which automatically made these packages available in TeX Live. Thus, the current status is quite satisfactory, since a normal TeX Live installation, without the need for any compilation, provides all necessary programs and packages. What remains a problem are inter-release updates of the binaries, when bugs in various engines are found. Future developments Besides the actual development of packages, and in particular luatexja, I see possible improvements in the following area: Summing up, I am quite happy to see that both the Japanese community, as well as the Korean community, have now taken the step to make their achievements available in an easy and straightforward way for everyone using TeX Live, and I am sure that the future will bring further improvements!
The slides accompanying the talk can be downloaded here: texconf15-texlive-japan.pdf. I also talked briefly on CJK font integration for Ghostscript, these slides can be downloaded here: texconf15-cjk-gs.pdf.

5 November 2015

Gunnar Wolf: WTF @omarfayad Should *all* computer activity be illegal now? #LeyFayad

Last week, Senator Omar Fayad presented one of the prime examples of a poorly redacted law that, if enacted, will make basically any way of computer use illegal. And yes, even if he states this is merely a draft, it has so many factual and conceptual errors that there is no way to trust sanity can be regained at any point. Oh, and before I continue with this rant: If the topic interests you, I suggest you to read the 10 key points about Ley Fayad, the worst Internet initiative in history, published by r3d.mx. [update] An English equivalent of the work at r3d, at revolution-news.com: #LeyFayad: The Worst Bill in Internet History The full text (in Spanish, of course) for the law initiative is available at the Senate webpage; the law will be called Ley Federal para Prevenir y Sancionar los Delitos Inform ticos (Federal law to prevent and punish informatic felonies<) A bad name to start with, as there are many laws already in that contested area. I started reading with the preamble (Exposici n de motivos), which already shows bad signs of imprecise redaction and is plagued with factual errors (i.e. asserting that the real danger stems from the Web migrating to the Web 2.0, from which stems that this migration and not any previous one. Or by stating that (quoting+translating a full paragraph):
Activities such as electronic commerce, digital periodism, publicity and the opinions, messages or elements written in social networks can lead to patrimonial, reputation, honor or professional activity losses for people.
He continues by stating that only 16% of the countries have some kind of cybersecurity strategy (and, of course, Mexico doesn't). That... Well, is very hard to believe, as Mexico has two separate policial groups devoted to cybersecurity, and laws regulating from electronic signatures, commerce, identity, privacy, use and abuse, and a long list. Of course, as most law proposals go, it quickly decays into a dry, boring document... And I must admit I didn't fully read it, but picked here and there. I won't copy in full the note I mentioned at the beginning at r3d.mx, but will continue with some strange points, such as:
Article 16
Every person that, without the corresponding authorization or exceeding the authorization confered, accesses, intercepts, interfers or uses an information system, will be punished by one to eight years of prision and fined by 800 to 1000 days of minimum wage
So, yes, borrowing your computer without getting explicit permission, or playing around with the options in kiosks, or tons of whatever we curious people do with systems we encounter are basis for jail. (And yes, fines in this country are expressed in "days of minimum wage", which goes at ~MX$70 per day, which is ~US$4). But it gets funkier quickly:
Article 17
Whoever fraudulently destroys, disables, damages or in any way alters the working of an informatic system or any of its components, will be punished by fice to fifteen years of jail and fined by up to a thousand minimum wage days The same punishment will be given to whoever, without authorization, destroys, damages, modifies, divulges, transfers or disables information contained in any Informatic System or any of its components. The punishment will be ten to twenty years in prision and a fine of up to a thousand days of minimum wage if the effects here mentioned are done by the creation, introduction or fraudulent transmission, by any means, of an informatic weapon or malicious code
This law is meant to protect against cyberfelonies, if such a thing exists. However, here we are putting at risk people even for accidental equipment destructions. I dropped your portable hard disk with my elbow off the table? Accuse me of acting fraudulently, and I'm up for a serious jail time. And yes, laws are meant to be interpreted... And I don't want to be at the receiving end of this one! In this last article, Fayad mentions informatic weapons, which are defined in the preamble as any informatic program, informatic system, or in general, any device or material created or designed with the purpose of committing an informatic crime. So the very next article makes me, as it should make all of my fellow students and researchers, very uneasy:
Article 18
Whoever uses informatic weapons or malicious code will be imprisioned by two to six years, and fined with 200 to 500 days of minimum wage.
Article 19
Whoever builds, distributes, commerces with informatic weapons or malicious codes will be punished by three to seven years of prision and 200 to 500 days of minimum wage.
If we need to analyze malware for our classes (or for paid work, or as a hobby), we clearly fall in article 18. If we write something that can be classified as malware (without even releasing it, as an academic excercise only!), we are covered by article 19. If I give my students code that's known to be malicious (which could be as inofensive as linking to a well-known Web comic), I'm also covered by article 19. I'll jump all the way to article 31 (reproduced only partially):
Article 31
Whoever, by any means, creates, captures, records, copies, alters, duplicates, clones or deletes the information contained in a credit or debit card (...) will be punished by 8 to 14 years of prision and 300 to 500 days of minimum wage. (...)
This clearly disincentivates any way of e-commerce. When I try to buy anything online, I have to capture+copy my (rightfully owned) credit card data. The services provider has to copy, process and then delete said information. Any e-transaction is punished by jail! Well... But thinking about this again, maybe I shouldn't be so worried about the malware distribution issue at my classes. There are clearer and more contundent articles. Say...
Article 35
Whoever convenes, organizes, is part of, or executes a cibernetic attack, will be punished by 20 to 30 years of prision and fined with 100 to 1000 days of minimum wage
Of course we have convened, organized, been part of and executed cibernetic attacks at the computer security lab at ESIME. Why would there be such a lab otherwise? Then, there are clear indications that the Senator didn't understand the topic his team was working on:
Article 37
Who manipulates the digital seals used by command of the public authority will be punished with 240 days of community work
Now... What is a digital seal? It's not a phisical one that does not allow opening the doors to a business found at fault, but something that just proves a document is legitimate and pristine. How can I manipulate them? Of course, if the seals are MD5-based, I can easily forge them (and SHA1-based, it seems they will be broken enough soon to be considered no longer trustable)... But that's about it! And there is more, lots more. I'm swamped with work, and have to get back to it. But chapters the following chapters have a lot of potential for finding holes. PS - And yes, the only use I do of Twitter is via the headlines in my blog ;-) [update] Ley Fayad is dead, yay! \o/ The senator withdrew the proposal.

4 November 2015

Mart n Ferrari: Tales from the SRE trenches: What can SRE do for you?

This is the fourth part in a series of articles about SRE, based on the talk I gave in the Romanian Association for Better Software. Previous articles can be found here: part 1, part 2, and part 3. As a side note, I am writing this from Heathrow Airport, about to board my plane to EZE. Buenos Aires, here I come!

So, what does SRE actually do? Enough about how to keep the holy wars under control and how to get work away from Ops. Let's talk about some of the things that SRE does. Obviously, SRE runs your service, performing all the traditional SysAdmin duties needed for your cat pictures to reach their consumers: provisioning, configuration, resource allocation, etc. They use specialised tools to monitor the service, and get alerted as soon as a problem is detected. They are also the ones waking up to fix your bugs in the middle of the night. But that is not the whole story: reliability is not only impacted by new launches. Suddenly usage grows, hardware fails, networks lose packets, solar flares flip your bits... When working at scale, things are going to break more often than not. You want these breakages to affect your availability as little as possible, there are three strategies you can apply to this end: minimise the impact of each failure, recover quickly, and avoid repetition. And of course, the best strategy of them all: preventing outages from happening at all.

Minimizing the impact of failures A single failure that takes the whole service down will affect severely your availability, so the first strategy as an SRE is to make sure your service is fragmented and distributed across what is called "failure domains". If the data center catches fire, you are going to have a bad day, but it is a lot better if only a fraction of your users depend on that one data center, while the others keep happily browsing cat pictures. SREs spend a lot of their time planning and deploying systems that span the globe to maximise redundancy while keeping latencies at reasonable levels.

Recovering quickly Many times, retrying after a failure is actually the best option. So another strategy is to automatically restart in milliseconds any piece of your system that fails. This way, less users are affected, while a human has time to investigate and fix the real problem. If a human needs to intervene, it is crucial that they get notified as fast as possible, and that they have quick access to all the information that is needed to solve the problem: detailed documentation of the production environment, meaningful and extensive monitoring, play-books1, etc. After all, at 2 AM you probably don't remember in which country the database shard lives, or what were the series of commands to redirect all the traffic to a different location. Implementing monitoring and automated alerts, writing documentation and play-books, and practising for disaster are other areas where SREs devote much effort.

Avoiding repetition Outages happen, pages ring, problems get fixed, but it should always be a chance for learning and improving. A page should require a human to think about a problem and find a solution. If the same problem keeps appearing, the human is not needed any more: the problem needs to be fixed at the root. Another key aspect of dealing with incidents is to write post-mortems2. Every incident should have one, and these should be tools for learning, not finger-pointing. Post-mortems can be an excellent tool, if people are honest about their mistakes, they are not used to blame other people, and issues are followed up by bug reports and discussions.

Preventing outages Of course nobody can prevent hard drives from failing, but there are certain classes of outages that can be forecasted with careful analysis. Usage spikes can bring a service down, but an SRE team will ensure that the systems are load-tested at higher-than-normal rates. They could also be prepared to quickly scale the service, provided the monitoring system will alert as soon as a certain threshold is reached. Monitoring is actually the key part in this: measuring relevant metrics for all the components of a system, and following their trends over time, SREs can completely avoid many outages: latencies growing out of acceptable bounds, disks filling up, progressive degradation of components, are all examples of typical problems automatically monitored (and alerted on) in an SRE team.
This is getting pretty long, so the next post will be the last one of these series, with some extra tips from SRE experience that I am sure can be applied in many places.

  1. A play-book is a document containing critical information needed when dealing with a specific alert: possible causes, hints for troubleshooting, links to more documentation. Some monitoring systems will automatically add a play-book URL to every alert sent, and you should be doing it too.
  2. Similarly to their real-life counterparts, a post-mortem is the process of examining a dead body (the outage), gutting it out and trying to understand what caused its demise.
Comment

Rapha&#235;l Hertzog: My Free Software Activities in October 2015

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 13.25 hours on Debian LTS. During this time I worked on the following things: I also started a conversation about what paid contributors could work on if they have some spare cycle as the current funding level might allow us to invest some time on work outside of just plain security updates. The Debian Administrator s Handbook I spent quite some time finalizing the Jessie book update, both for the content and for the layout of the printed book. Debian Handbook: cover of the jessie edition Misc Debian work GNOME 3.18. I uploaded a new gnome-shell-timer working with GNOME Shell 3.18 and I filed bugs #800660 and #802480 about an annoying gnome-keyring regression I did multiple test rounds with the Debian maintainers (Dmitry Shachnev, kudos to him!) and the upstream developers (see here and here). Apart from those regressions, I like GNOME 3.18! Python-modules team migration to Git. After the Git migration, and since the team policy now imposes usage of git-dpm on all members, I made some tries with it on the python-django package while pushing version 1.8.5 to experimental. And the least I can say is that I m not pleased with the result. I thus filed 3 bugs summarizing the problems I have with git-dpm: #801666 (no way to set the upstream branch names from within the repository), #801667 (no clean way to merge between packaging branches), #801668 (does not create upstream tag immediately on tarball import). That is on top of other randomly stupid bugs that were already reported like #801548 (does not work with perfectly valid pre-existing upstream tags). Django packaging. I filed bugs on all packages build-depending on python-django that fail to build with Django 1.8 and informed them that I would upload Django 1.8 to unstable in early November (it s done already). Then I fixed python-django-jsonfield myself since Distro Tracker relies on this package. Following this small mass-bug filing, I filed a wishlist bug on devscripts to improve the mass-bug helper script (see #801926). And since I used ratt to rebuild the packages, I filed a wishlist issue on this new tool as well. Tryton 3.6 upgrade. I upgraded my own Tryton installation to version 3.6 and filed bug #803066 because the SysV init script was not working properly. That also reminded me that the DD process of Matthias Behrle (the tryton package maintainer) was stalled due to a bug in the NM infrastructure so I pinged the NM team and we sorted out a way for me to advocate him and get his process going Distro Tracker. I continued my work to refactor the way we handle incoming mail processing (branch people/hertzog/mailprocessing). It s now mostly finished and I need to deploy it in a test environment before being able to roll it out on tracker.debian.org. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Vincent Sanders: I am not a number I am a free man

Once more the NetSurf developers tried to escape from a mysterious village by writing web browser code.

Michael Drake, Daniel Silverstone, Dave Higton and Vincent Sanders at NetSurf Developer workshop
The sixth developer workshop was an opportunity for us to gather together in person to contribute to NetSurf.

We were hosted by Codethink in their Manchester offices which provided a comfortable and pleasant space to work in.

Four developers managed to attend in person from around the UK: Michael Drake, Daniel Silverstone, Dave Higton and Vincent Sanders.

The main focus of the weekends activities was to work on improving our JavaScript implementation. At the previous workshop we had laid the groundwork for a shift to the Duktape JavaScript engine and since then put several hundred hours of time into completing this transition.

During this weekend Daniel built upon this previous work and managed to get DOM events working. This was a major missing piece of implementation which will mean NetSurf will be capable of interpreting JavaScript based web content in a more complete fashion. This work revealed several issues with our DOM library which were also resolved.

We were also able to merge several improvements provided by the Duktape upstream maintainer Sami Vaarala which addressed performance problems with regular expressions which were causing reports of "hangs" on slow processors.

The responsiveness of Sami and the Ducktape project has been a pleasant surprise making our switch to the library look like an increasingly worthwhile effort.

Overall some good solid progress was made on JavaScript support. Around half of the DOM interfaces in the specifications have now been implemented leaving around fifteen hundred methods and properties remaining. There is an aim to have this under the thousand mark before the new year which should result in a generally useful implementation of the basic interfaces.

Once the DOM interfaces have been addressed our focus will move onto the dynamic layout engine necessary to allow rendering of the changing content.

The 3.4 release is proposed to occur sometime early in the new year and depends on getting the JavaScript work to a suitable stable state.

Dave joined us for the first time, he was principally concerned with dealing with bugs and the bug tracker. It was agreeable to have a new face at the meeting and some enthusiasm for the RISC OS port which has been lacking an active maintainer for some time.

The turnout for this workshop was the same as the previous one and the issues raised then are still true. We still have a very small active core team who can commit only limited time which is making progress very slow and are lacking significant maintenance for several frontends.

Overall we managed to pack 16 hours of work into the weekend and addressed several significant problems.

3 November 2015

Norbert Preining: Two videos that make you happy

Sometimes one feels down and troubled, and instead of a helping hand, watching videos that cheer you up can get you a long way. Two of my favorite videos are one of the crazy Korean drummer Kwon Soon Keun (search on youtube), and the election campaign video of Iceland s Best party, founded by the musician and comedian J n Gnarr, which won the election of the Reykjav k City Council in 2010. Both pieces always make me laugh out loud, even after having seen them many times. Kwon Soon Keun
Besti flokkurinn
Let me know what you are watching/listening when you need a cheer-up!

Joey Hess: STM Region contents

concurrent-output released yesterday got a lot of fun features. It now does full curses-style minimization of the output, to redraw updated lines with optimal efficiency. And supports multiline regions/wrapping too long lines. And allows the user to embed ANSI colors in a region. 3 features that are in some tension and were fun to implement all together. But I have a more interesting feature to blog about... I've added the ability for the content of a Region to be determined by a (STM transaction). Here, for example, is a region that's a clock:
timeDisplay :: TVar UTCTime -> STM Text
timeDisplay tv = T.pack . show <$> readTVar tv
clockRegion :: IO ConsoleRegionHandle
clockRegion = do
    tv <- atomically . newTVar =<< getCurrentTime
    r <- openConsoleRegion Linear
    setConsoleRegion r (timeDisplay tv)
    async $ forever $ do
        threadDelay 1000000 -- 1 sec
        atomically . (writeTVar tv) =<< getCurrentTime
    return r
There's something magical about this. Whenever a new value is written into the TVar, concurrent-output automatically knows that this region needs to be updated. How does it know how to do that? Magic of STM. Basically, concurrent-output composes all the STM transactions of Regions, and asks STM to wait until there's something new to display. STM keeps track of whatever TVars might be looked at, and so can put the display thread to sleep until there's a change to display. Using STM I've gotten extensability for free, due to the nice ways that STM transactions compose. A few other obvious things to do with this: Compose 2 regions with padding so they display on the same line, left and right aligned. Trim a region's content to the display width. (Handily exported by concurrent-output in a TVar for this kind of thing.)
I'm tempted to write a console spreadsheet using this. Each visible cell of the spreadsheet would have its own region, that uses a STM transaction to display. Plain data Cells would just display their current value. Cells that contain a function would read the current values of other Cells, and use that to calculate what to display. Which means that a Cell containing a function would automatically update whenever any of the Cells that it depends on were updated! Do you think that a simple interactive spreadsheet built this way would be more than 100 lines of code?

Next.

Previous.