Search Results: "tar"

13 August 2016

Dariusz Dwornikowski: Automatic PostgreSQL config with Ansible

If for some reasons you can t use dedicated DBaaS for your PostgreSQL (like AWS RDS) then you need to run your database server on a cloud instance. In these kind of setup, when you scale up or down your instance size, you need to adjust PostgreSQL parameters according to the changing RAM size. There are several parameters in PostgreSQL that highly depend on RAM size. An example is shared_buffers for which a rule of thumb says that is should be set to 0.25*RAM. In DBaaS, when you scale the DB instance up or down, parameters are adjusted for you by the cloud provider, e.g. AWS RDS uses parameter groups for that reason, where particular parameters are defined depending on the size of the RAM of the RDS instance. So what can you when you do not have RDS or any other DBaaS? You can always keep several configuration files on your instance, each for a different memory size, you can rewrite you config every time you change the size of the instance or you can use Ansible role for that. Our Ansible role will be very simple, we will have two tasks. One will change the PostgreSQL config, the second one will just restart the database server: 
---
- name: Update PostgreSQL config
  template: src=postgresql.conf.j2 dest=/etc/postgresql/9.5/main/postgresql.conf
  register: pgconf
- name: Restart postgresql
  service: name=postgresql state=restarted
  when: pgconf.changed
Now we need the template, where are the calculations take place. RAM size will be taken from the Ansible s fact called ansible_memtotal_mb. Since it returns RAM size in MBs, we will stick to MBs. We will define the following parameters, you can adjust them to your needs: For max_connections we will define a default role variable of 100 but we will allow to specify it at a runtime. The relevant parts of the postgresql.conf.j2 are below: 
 max_connections =   max_connections        
 shared_buffers =   (((ansible_memtotal_mb/1024.0) round int)*0.25) int*1024  MB
 work_mem =   ((((ansible_memtotal_mb/1024.0) round int)*0.25)/max_connections*1024) round int  MB
 maintenance_work_mem =   ((ansible_memtotal_mb/1024.0) round int)*64  MB
 effective_cache_size =   (((ansible_memtotal_mb/1024.0) round int)*0.75) int*1024  MB
You can now run the role every time you change the instance size, and the config will be changed accordingly to the RAM size. You can extend the role and maybe add other constraints and change max_connections to you specific needs. An example playbook could look like: 
---
hosts: my_postgres
roles:
  - postgres-config 
vars:
  - max_connection: 300
And run it: 
$ ansible-playbook playbook.yml
The complete role can be found in my github repo.

Russell Coker: SSD and M.2

The Need for Speed One of my clients has an important server running ZFS. They need to have a filesystem that detects corruption, while regular RAID is good for the case where a disk gives read errors it doesn t cover the case where a disk returns bad data and claims it to be good (which I ve witnessed in BTRFS and ZFS systems). BTRFS is good for the case of a single disk or a RAID-1 array but I believe that the RAID-5 code for BTRFS is not sufficiently tested for business use. ZFS doesn t perform very well due to the checksums on data and metadata requiring multiple writes for a single change which also causes more fragmentation. This isn t a criticism of ZFS, it s just an engineering trade-off for the data integrity features. ZFS supports read-caching on a SSD (the L2ARC) and write-back caching (ZIL). To get the best benefit of L2ARC and ZIL you need fast SSD storage. So now with my client investigating 10 gigabit Ethernet I have to investigate SSD. For some time SSDs have been in the same price range as hard drives, starting at prices well below $100. Now there are some SSDs on sale for as little as $50. One issue with SATA for server use is that SATA 3.0 (which was released in 2009 and is most commonly used nowadays) is limited to 600MB/s. That isn t nearly adequate if you want to serve files over 10 gigabit Ethernet. SATA 3.2 was released in 2013 and supports 1969MB/s but I doubt that there s much hardware supporting that. See the SATA Wikipedia page for more information. Another problem with SATA is getting the devices physically installed. My client has a new Dell server that has plenty of spare PCIe slots but no spare SATA connectors or SATA power connectors. I could have removed the DVD drive (as I did for some tests before deploying the server) but that s ugly and only gives 1 device while you need 2 devices in a RAID-1 configuration for ZIL. M.2 M.2 is a new standard for expansion cards, it supports SATA and PCIe interfaces (and USB but that isn t useful at this time). The wikipedia page for M.2 is interesting to read for background knowledge but isn t helpful if you are about to buy hardware. The first M.2 card I bought had a SATA interface, then I was unable to find a local company that could sell a SATA M.2 host adapter. So I bought a M.2 to SATA adapter which made it work like a regular 2.5 SATA device. That s working well in one of my home PCs but isn t what I wanted. Apparently systems that have a M.2 socket on the motherboard will usually take either SATA or NVMe devices. The most important thing I learned is to buy the SSD storage device and the host adapter from the same place then you are entitled to a refund if they don t work together. The alternative to the SATA (AHCI) interface on an M.2 device is known as NVMe (Non-Volatile Memory Express), see the Wikipedia page for NVMe for details. NVMe not only gives a higher throughput but it gives more command queues and more commands per queue which should give significant performance benefits for a device with multiple banks of NVRAM. This is what you want for server use. Eventually I got a M.2 NVMe device and a PCIe card for it. A quick test showed sustained transfer speeds of around 1500MB/s which should permit saturating a 10 gigabit Ethernet link in some situations. One annoyance is that the M.2 devices have a different naming convention to regular hard drives. I have devices /dev/nvme0n1 and /dev/nvme1n1, apparently that is to support multiple storage devices on one NVMe interface. Partitions have device names like /dev/nvme0n1p1 and /dev/nvme0n1p2. Power Use I recently upgraded my Thinkpad T420 from a 320G hard drive to a 500G SSD which made it faster but also surprisingly quieter you never realise how noisy hard drives are until they go away. My laptop seemed to feel cooler, but that might be my imagination. The i5-2520M CPU in my Thinkpad has a TDP of 35W but uses a lot less than that as I almost never have 4 cores in use. The z7k320 320G hard drive is listed as having 0.8W low power idle and 1.8W for read-write, maybe Linux wasn t putting it in the low power idle mode. The Samsung 500G 850 EVO SSD is listed as taking 0.4W when idle and up to 3.5W when active (which would not be sustained for long on a laptop). If my CPU is taking an average of 10W then replacing the hard drive with a SSD might have reduced the power use of the non-screen part by 10%, but I doubt that I could notice such a small difference. I ve read some articles about power use on the net which can be summarised as SSDs can draw more power than laptop hard drives but if you do the same amount of work then the SSD will be idle most of the time and not use much power . I wonder if the SSD being slightly thicker than the HDD it replaced has affected the airflow inside my Thinkpad. From reading some of the reviews it seems that there are M.2 storage devices drawing over 7W! That s going to create some cooling issues on desktop PCs but should be OK in a server. For laptop use they will hopefully release M.2 devices designed for low power consumption. The Future M.2 is an ideal format for laptops due to being much smaller and lighter than 2.5 SSDs. Spinning media doesn t belong in a modern laptop and using a SATA SSD is an ugly hack when compared to M.2 support on the motherboard. Intel has released the X99 chipset with M.2 support (see the Wikipedia page for Intel X99) so it should be commonly available on desktops in the near future. For most desktop systems an M.2 device would provide all the storage that is needed (or 2*M.2 in a RAID-1 configuration for a workstation). That would give all the benefits of reduced noise and increased performance that regular SSDs provide, but with better performance and fewer cables inside the PC. For a corporate desktop PC I think the ideal design would have only M.2 internal storage and no support for 3.5 disks or a DVD drive. That would allow a design that is much smaller than a current SFF PC.
Related posts:
  1. Breaking SATA Connectors I ve just broken my second SATA connector. This isn t a...
  2. How I Partition Disks Having had a number of hard drives fail over the...
  3. Swap Space and SSD In 2007 I wrote a blog post about swap space...

David Moreno: Perl 5.12's each function

With Perl 5.12 released earlier this summer, the each function got a nice little addition that I d like to talk about: It now has the ability to work on arrays, not only key-value pair hashes, but not exactly as you d expect it (not like Ruby s each method). The traditional way to work with each, using a hash: 
my %h = (
    a => 1,
    b => 2,
    c => 3,
);
while(my($key, $value) = each %h)  
    say "index: $key => value: $value";
And the output. Of course, hashes being unordered lists, you won t get the nice little order of an array. 
index: c => value: 3
index: a => value: 1
index: b => value: 2
Now, when you use an array, each will return the next pair on the list consisting on the index of that element s position and the position itself. Since it returns the next pair, you can iterate through it on the same fashion as when using a hash: 
my @arr = ('a'..'z');
while(my($index, $value) = each @arr)  
    say "index: $index => value: $value";
This is particularly useful to access direct named variables, both the index and the element, while looping through an array. 
index: 0 => value: a
index: 1 => value: b
index: 2 => value: c
index: 3 => value: d
index: 4 => value: e
index: 5 => value: f
index: 6 => value: g
index: 7 => value: h
index: 8 => value: i
index: 9 => value: j
index: 10 => value: k
index: 11 => value: l
index: 12 => value: m
index: 13 => value: n
index: 14 => value: o
index: 15 => value: p
index: 16 => value: q
index: 17 => value: r
index: 18 => value: s
index: 19 => value: t
index: 20 => value: u
index: 21 => value: v
index: 22 => value: w
index: 23 => value: x
index: 24 => value: y
index: 25 => value: z

David Moreno: Disable Nginx logging

This is something that is specified clearly on the Nginx manual, but it s nice to have it as a quick reference. The access_log and error_log directives on Nginx are on separate modules (HTTP Log and Core modules respectively) and they don t behave the same way when all you want is to disable all logging on your server (in our case, we serve a gazillion static files and perform a lot of reverse proxying and we re not interested on tracking that). It s a common misconception that you can set error_log to off, because that s how you disable access_log (if you do that, the server will still log to the file $nginx_path/off). Instead, you have to set error_log to log to the always mighty black hole /dev/null using the highest level for logging (which triggers the fewest events), crit: 
http  
  server  
    # ...
    access_log off;
    error_log /dev/null crit;
    # ...
   
  #...
If you re the possessor of the blingest of bling-bling, you can disable all logging (not only for a server block), by putting error_log on the root of the configuration and access_log within your http block and make sure you don t override that on any of the inner blocks. And you re good to go.

David Moreno: Geo::PostalCode::NoDB 0.01

Geo::PostalCode is a great Perl module. It lets you find surrounding postal areas (zip codes) around a given an amount of miles (radius), calculate distance between them, among other nice features. Sadly, I couldn t get it to work with updated data and because the file its Berkely DB installer was producing was not being recognized by its parser, which bases off on DB_File. Since I was able to find working data for the source of zip codes, I ended up hacking the module and producing a version with no Berkeley DB support. So basically, and taken from the POD:
RATIONALE BEHIND NO BERKELEY DB
On a busy day at work, I couldn t get Geo::PostalCode to work with newer data (the data source TJMATHER points to is no longer available), so the tests shipped with his module pass, but trying to use real data no longer seems to work. DB_File marked the Geo::PostalCode::InstallDB output file as invalid type or format. If you don t run into that issue by not wanting to use this module, please drop me a note! I would love to learn how other people made it work.

So, in order to get my shit done, I decided to create this module. Loading the whole data into memory from the class constructor has been proven to be enough for massive usage (citation needed) on a Dancer application where this module is instantiated only once.
$ sudo cpanm Geo::PostalCode::NoDB now!

12 August 2016

Jonathan Dowland: Lush (and friends)

gig poster The gig poster
On July 31st a friend and I went to see Max mo Park and support at a mini-festival day in Times Square, Newcastle. The key attraction for me to this gig was the top support band, Lush who are back after a nearly 20 year hiatus.
record Nano Kino 7"
I first heard of Lush quite recently from the excellent BBC Documentary Girl in a Band: Tales from the Rock 'n' Roll Front Line. They were excellent: the set was quite heavy on material from their more dreampop/shoegaze albums which is to my taste.
Max mo 7"s Max mo 7"s
I also particularly enjoyed Warm Digits, motorik instrumental dance stuff that reminded me of Lemon Jelly mixed with Soulwax, who had two releases very reasonably priced on the merch stand; Nano Kino in the adjacent "Other Rooms", also channelling dreampop/shoegaze; and finally Max mo Park themselves. I was there for Lush really but I still really enjoyed the headliners. I've seen them several times but I've lost track of what they've been up to in recent years. Both their earliest material and several newer songs were well received by the home crowd and atmosphere in the enclosed Times Square was excellent.

Markus Koschany: My Free Software Activities in July 2016

Welcome to gambaru.de. Here is my monthly report that covers what I have been doing for Debian. Debian Android Debian Games Debian Java Debian LTS This was my sixth month as a paid contributor and I have been paid to work 14,7 hours on Debian LTS. In that time I did the following: Misc and QA

11 August 2016

Matthew Garrett: Microsoft's compromised Secure Boot implementation

There's been a bunch of coverage of this attack on Microsoft's Secure Boot implementation, a lot of which has been somewhat confused or misleading. Here's my understanding of the situation.

Windows RT devices were shipped without the ability to disable Secure Boot. Secure Boot is the root of trust for Microsoft's User Mode Code Integrity (UMCI) feature, which is what restricts Windows RT devices to running applications signed by Microsoft. This restriction is somewhat inconvenient for developers, so Microsoft added support in the bootloader to disable UMCI. If you were a member of the appropriate developer program, you could give your device's unique ID to Microsoft and receive a signed blob that disabled image validation. The bootloader would execute a (Microsoft-signed) utility that verified that the blob was appropriately signed and matched the device in question, and would then insert it into an EFI Boot Services variable[1]. On reboot, the boot loader reads the blob from that variable and integrates that policy, telling later stages to disable code integrity validation.

The problem here is that the signed blob includes the entire policy, and so any policy change requires an entirely new signed blob. The Windows 10 Anniversary Update added a new feature to the boot loader, allowing it to load supplementary policies. These must also be signed, but aren't tied to a device id - the idea is that they'll be ignored unless a device-specific policy has also been loaded. This way you can get a single device-specific signed blob that allows you to set an arbitrary policy later by using a combination of supplementary policies.

This is all fine in the Anniversary Edition. Unfortunately older versions of the boot loader will happily load a supplementary policy as if it were a full policy, ignoring the fact that it doesn't include a device ID. The loaded policy replaces the built-in policy, so in the absence of a base policy a supplementary policy as simple as "Enable this feature" will effectively remove all other restrictions.

Unfortunately for Microsoft, such a supplementary policy leaked. Installing it as a base policy on pre-Anniversary Edition boot loaders will then allow you to disable all integrity verification, including in the boot loader. Which means you can ask the boot loader to chain to any other executable, in turn allowing you to boot a compromised copy of any operating system you want (not just Windows).

This does require you to be able to install the policy, though. The PoC released includes a signed copy of SecureBootDebug.efi for ARM, which is sufficient to install the policy on ARM systems. There doesn't (yet) appear to be a public equivalent for x86, which means it's not (yet) practical for arbitrary attackers to subvert the Secure Boot process on x86. I've been doing my testing on a setup where I've manually installed the policy, which isn't practical in an automated way.

How can this be prevented? Installing the policy requires the ability to run code in the firmware environment, and by default the boot loader will only load signed images. The number of signed applications that will copy the policy to the Boot Services variable is presumably limited, so if the Windows boot loader supported blacklisting second-stage bootloaders Microsoft could simply blacklist all policy installers that permit installation of a supplementary policy as a primary policy. If that's not possible, they'll have to blacklist of the vulnerable boot loaders themselves. That would mean all pre-Anniversary Edition install media would stop working, including recovery and deployment images. That's, well, a problem. Things are much easier if the first case is true.

Thankfully, if you're not running Windows this doesn't have to be a issue. There are two commonly used Microsoft Secure Boot keys. The first is the one used to sign all third party code, including drivers in option ROMs and non-Windows operating systems. The second is used purely to sign Windows. If you delete the second from your system, Windows boot loaders (including all the vulnerable ones) will be rejected by your firmware, but non-Windows operating systems will still work fine.

From what we know so far, this isn't an absolute disaster. The ARM policy installer requires user intervention, so if the x86 one is similar it'd be difficult to use this as an automated attack vector[2]. If Microsoft are able to blacklist the policy installers without blacklisting the boot loader, it's also going to be minimally annoying. But if it's possible to install a policy without triggering any boot loader blacklists, this could end up being embarrassing.

Even outside the immediate harm, this is an interesting vulnerability. Presumably when the older boot loaders were written, Microsoft policy was that they would never sign policy files that didn't include a device ID. That policy changed when support for supplemental policies was added. without this policy change, the older boot loaders could still be considered secure. Adding new features can break old assumptions, and your design needs to take that into account.

[1] EFI variables come in two main forms - those accessible at runtime (Runtime Services variables) and those only accessible in the early boot environment (Boot Services variables). Boot Services variables can only be accessed before ExitBootServices() is called, and in Secure Boot environments all code executing before this point is (theoretically) signed. This means that Boot Services variables are nominally tamper-resistant.

[2] Shim has explicit support for allowing a physically present machine owner to disable signature validation - this is basically equivalent

comment count unavailable comments

Petter Reinholdtsen: Coz can help you find bottlenecks in multi-threaded software - nice free software

This summer, I read a great article "coz: This Is the Profiler You're Looking For" in USENIX ;login: about how to profile multi-threaded programs. It presented a system for profiling software by running experiences in the running program, testing how run time performance is affected by "speeding up" parts of the code to various degrees compared to a normal run. It does this by slowing down parallel threads while the "faster up" code is running and measure how this affect processing time. The processing time is measured using probes inserted into the code, either using progress counters (COZ_PROGRESS) or as latency meters (COZ_BEGIN/COZ_END). It can also measure unmodified code by measuring complete the program runtime and running the program several times instead. The project and presentation was so inspiring that I would like to get the system into Debian. I created a WNPP request for it and contacted upstream to try to make the system ready for Debian by sending patches. The build process need to be changed a bit to avoid running 'git clone' to get dependencies, and to include the JavaScript web page used to visualize the collected profiling information included in the source package. But I expect that should work out fairly soon. The way the system work is fairly simple. To run an coz experiment on a binary with debug symbols available, start the program like this: 
coz run --- program-to-run
This will create a text file profile.coz with the instrumentation information. To show what part of the code affect the performance most, use a web browser and either point it to http://plasma-umass.github.io/coz/ or use the copy from git (in the gh-pages branch). Check out this web site to have a look at several example profiling runs and get an idea what the end result from the profile runs look like. To make the profiling more useful you include <coz.h> and insert the COZ_PROGRESS or COZ_BEGIN and COZ_END at appropriate places in the code, rebuild and run the profiler. This allow coz to do more targeted experiments. A video published by ACM presenting the Coz profiler is available from Youtube. There is also a paper from the 25th Symposium on Operating Systems Principles available titled Coz: finding code that counts with causal profiling. The source code for Coz is available from github. It will only build with clang because it uses a C++ feature missing in GCC, but I've submitted a patch to solve it and hope it will be included in the upstream source soon. Please get in touch if you, like me, would like to see this piece of software in Debian. I would very much like some help with the packaging effort, as I lack the in depth knowledge on how to package C++ libraries.

10 August 2016

Tom Marble: webica

webica I've just pushed the first version of my new Clojure wrapper for Selenium called webica. The reason I need webica is that I want to do automated browser testing for ClojureScript based web applications. Certainly NodeJS, PhantomJS, Nashorn and the like are useful... but these can't quite emulate the full browser experience. We want to test our ClojureScript web apps in browsers -- ideally via our favorite automated continuous integration tools.

Selenium
My new approach with the webica library is to do full Java introspection in the spirit that amazonica does for the AWS API. In fact I wanted to take it a step further by actually generating Clojure source code via introspection that can be used by Codox to generate nice API docs (which you don't get with amazonica). That, alas, was a little trickier than expected due to pesky Quine-like problems :) . If you load the library on the REPL you can get a feeling for each namespace by calling the show-functions function. I realize this approach of aggressive introspection, playing fast and loose with types and application level dynamic dispatch are crazy antipatterns. In my defense I started out playing around to see "if I could do it". After seeing the result in the form of a shell script in Clojure -- imitating lmgtfy -- perhaps webica will actually be useful! I plan to talk about webica tonight at clojure.mn -- hope to see you there!

Julian Andres Klode: Porting APT to CMake

Ever since it s creation back in the dark ages, APT shipped with it s own build system consisting of autoconf and a bunch of makefiles. In 2009, I felt like replacing that with something more standard, and because nobody really liked autotools, decided to go with CMake. Well, the bazaar branch was never really merged back in 2009. Fast forward 7 years to 2016. A few months ago, we noticed that our build system had trouble with correct dependencies in parallel building. So, in search for a way out, I picked up my CMake branch from 2009 last Thursday and spent the whole weekend working on it, and today I am happy to announce that I merged it into master: 
123 files changed, 1674 insertions(+), 3205 deletions(-)
More than 1500 lines less build system code. Quite impressive, eh? This also includes about 200 lines of less code in debian/, as that switched from prehistoric debhelper stuff to modern dh (compat level 9, almost ready for 10). The annoying Tale of Targets vs Files Talking about CMake: I don t really love it. As you might know, CMake differentiates between targets and files. Targets can in some cases depend on files (generated by a command in the same directory), but overall files are not really targets. You also cannot have a target with the same name as a file you are generating in a custom command, you have to rename your target (make is OK with the generated stuff, but ninja complains about cycles because your custom target and your custom command have the same name). Byproducts for the (time) win One interesting thing about CMake and Ninja are byproducts. In our tree, we are building C++ files. We also have .pot templates depending on them, and .mo files depending on the templates (we have multiple domains, and merge the per-domain .pot with the all-domain .po file during the build to get a per-domain .mo). Now, if we just let them depend naively, changing a C++ file causes the .pot file to be regenerated which in turns causes us to build .mo files for every freaking language in the package. Even if nothing changed. Byproducts solve this problem. Instead of just building the .pot file, we also create a stamp file (AKA the witness) and write the .pot file (without a header) into a temporary name and only copy it to its final name if the content changed. The .pot file is declared as a byproduct of the command. The command doing the .pot->.mo step still depends on the .pot file (the byproduct), but as that only changes now if strings change, the .mo files only get rebuild if I change a translatable string. We still need to ensure that that the .pot file is actually built before we try to use it the solution here is to specify a custom target depending on the witness and then have the target containing the .mo build commands depend on that target. Now if you use make, you might now this trick already. In make, the byproducts remain undeclared, though, while in CMake we can now actually express them, and they are used by the Ninja generator and the Ninja build tool if you chose that over make (try it out, it s fast). Further Work Some command names are hardcoded, I should find_program() them. Also cross-building the package does not yet work successfully, but it only requires a tiny amount of patches in debhelper and/or cmake. I also tried building the package on a Fedora docker image (with dpkg installed, it s available in the Fedora sources). While I could eventually get the programs build and most of the integration test suite to pass, there are some minor issues to fix, mostly in the documentation building and GTest department: Fedora ships its docbook stylesheets in a different location, and ships GTest as a pre-compiled library, and not a source tree. I have not yet tested building on exotic platforms like macOS, or even a BSD. Please do and report back. In Debian, CMake is not up-to.date enough on the non-Linux platforms to build APT due to test suite failures, I hope those can be fixed/disabled soon (it appears to be a timing issue AFAICT). I hope that we eventually get some non-Debian backends for APT. I d love that.
Filed under: Debian, Uncategorized

9 August 2016

John Goerzen: Easily Improving Linux Security with Two-Factor Authentication

2-Factor Authentication (2FA) is a simple way to help improve the security of your systems. It restricts the scope of damage if a machine is compromised. If, for instance, you have a security token or authenticator app on your phone that is required for ssh to a remote machine, then even if every laptop you use to connect to the remote is totally owned, an attacker cannot establish a new ssh session on their own. There are a lot of tutorials out there on the Internet that get you about halfway there, so here is some more detail. Background In this article, I will be focusing on authentication in the style of Google Authenticator, which is a special case of OATH HOTP or TOTP. You can use the Google Authenticator app, FreeOTP, or a hardware token like Yubikey to generate tokens with this. They are all 100% compatible with Google Authenticator and libpam-google-authenticator. The basic idea is that there is a pre-shared secret key. At each login, a different and unique token is required, which is generated based on the pre-shared secret key and some other information. With TOTP, the other information is the current time, implying that both machines must be reasably well in-sync time-wise. With HOTP, the other information is a count of the number of times the pre-shared key has been used. Both typically have a window on the server side that can let times within a certain number of seconds, or a certain number of login accesses, work. The beauty of this system is that after the initial setup, no Internet access is required on either end to validate the key (though TOTP requires both ends to be reasonably in sync time-wise). The basics: user account setup and ssh authentication You can start with the basics by reading one of these articles: one, two, three. Debian/Ubuntu users will find both the pam module and the user account setup binary in libpam-google-authenticator. For many, you can stop there. You re done. But if you want to kick it up a notch, read on: Enhancement 1: Requiring 2FA even when ssh public key auth is used Let s consider a scenario in which your system is completely compromised. Unless your ssh keys are also stored in something like a Yubikey Neo, they could wind up being compromised as well if someone can read your files and sniff your keyboard, your ssh private keys are at risk. So we can configure ssh and PAM so that a OTP token is required even for this scenario. First off, in /etc/ssh/sshd_config, we want to change or add these lines: 
UsePAM yes
ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive
This forces all authentication to pass two verification methods in ssh: publickey and keyboard-interactive. All users will have to supply a public key and then also pass keyboard-interactive auth. Normally keyboard-interactive auth prompts for a password, but we can change /etc/pam.d/sshd on this. I added this line at the very top of /etc/pam.d/sshd: 
auth [success=done new_authtok_reqd=done ignore=ignore default=bad] pam_google_authenticator.so
This basically makes Google Authenticator both necessary and sufficient for keyboard-interactive in ssh. That is, whenever the system wants to use keyboard-interactive, rather than prompt for a password, it instead prompts for a token. Note that any user that has not set up google-authenticator already will be completely unable to ssh into their account. Enhancement 1, variant 2: Allowing automated processes to root On many of my systems, I have ~root/.ssh/authorized_keys set up to permit certain systems to run locked-down commands for things like backups. These are automated commands, and the above configuration will break them because I m not going to be typing in codes at 3AM. If you are very restrictive about what you put in root s authorized_keys, you can exempt the root user from the 2FA requirement in ssh by adding this to sshd_config: 
Match User root
  AuthenticationMethods publickey
This says that the only way to access the root account via ssh is to use the authorized_keys file, and no 2FA will be required in this scenario. Enhancement 1, variant 2: Allowing non-pubkey auth On some multiuser systems, some users may still want to use password auth rather than publickey auth. There are a few ways we can support that:
  1. Users without public keys will have to supply a OTP and a password, while users with public keys will have to supply public key, OTP, and a password
  2. Users without public keys will have to supply OTP or a password, while users with public keys will have to supply public key, OTP, or a password
  3. Users without public keys will have to supply OTP and a password, while users with public keys only need to supply the public key
The third option is covered in any number of third-party tutorials. To enable options 1 or 2, you ll need to put this in sshd_config: 
AuthenticationMethods publickey,keyboard-interactive keyboard-interactive
This means that to authenticate, you need to pass either publickey and then keyboard-interactive auth, or just keyboard-interactive auth. Then in /etc/pam.d/sshd, you put this: 
auth required pam_google_authenticator.so
As a sub-variant for option 1, you can add nullok to here to permit auth from people that do not have a Google Authenticator configuration. Or for option 2, change required to sufficient . You should not add nullok in combination with sufficient, because that could let people without a Google Authenticator config authenticate completely without a password at all. Enhancement 2: Configuring su A lot of other tutorials stop with ssh (and maybe gdm) but forget about the other ways we authenticate or change users on a system. su and sudo are the two most important ones. If your root password is compromised, you don t want anybody to be able to su to that account without having to supply a token. So you can set up google-authenticator for root. Then, edit /etc/pam.d/su and insert this line after the pam_rootok.so line: 
auth       required     pam_google_authenticator.so nullok
The reason you put this after pam_rootok.so is because you want to be able to su from root to any account without having to input a token. We add nullok to the end of this, because you may want to su to accounts that don t have tokens. Just make sure to configure tokens for the root account first. Enhancement 3: Configuring sudo This one is similar to su, but a little different. This lets you, say, secure the root password for sudo. Normally, you might sudo from your user account to root (if so configured). You might have sudo configured to require you to enter in your own password (rather than root s), or to just permit you to do whatever you want as root without a password. Our first step, as always, is to configure PAM. What we do here depends on your desired behavior: do you want to require someone to supply both a password and a token, or just a token, or require a token? If you want to require a token, put this at the top of /etc/pam.d/sudo: 
auth [success=done new_authtok_reqd=done ignore=ignore default=bad] pam_google_authenticator.so
If you want to require a token and a password, change the bracketed string to required , and if you want a token or a password, change it to sufficient . As before, if you want to permit people without a configured token to proceed, add nullok , but do not use that with sufficient or the bracketed example here. Now here comes the fun part. By default, if a user is required to supply a password to sudo, they are required to supply their own password. That does not help us here, because a user logged in to the system can read the ~/.google_authenticator file and easily then supply tokens for themselves. What you want to do is require them to supply root s password. Here s how I set that up in sudoers: 
Defaults:jgoerzen rootpw
jgoerzen ALL=(ALL) ALL
So now, with the combination of this and the PAM configuration above, I can sudo to the root user without knowing its password but only if I can supply root s token. Pretty slick, eh? Further reading In addition to the basic tutorials referenced above, consider: Edit: additional comments Here are a few other things to try: First, the libpam-google-authenticator module supports putting the Google Authenticator files in different locations and having them owned by a certain user. You could use this to, for instance, lock down all secret keys to be readable only by the root user. This would prevent users from adding, changing, or removing their own auth tokens, but would also let you do things such as reusing your personal token for the root account without a problem. Also, the pam-oath module does much of the same things as the libpam-google-authenticator module, but without some of the help for setup. It uses a single monolithic root-owned password file for all accounts. There is an oathtool that can be used to generate authentication codes from the command line.

Reproducible builds folks: Finishing the final variations for reprotest

Author: ceridwen I've been working on getting the last of the variations working. With no responses on the mailing list from anyone outside Debian and with limited time remaining, with Lunar I've decided to deemphasize it.
  1. Build path is done.
  2. Host and domain name use domainname and hostname. This site is old, but it indicates that domainname was available on most OSes and hostname was available everywhere as of 2004. Prebuilder uses a Linux-specific utility (unshare --uts) to run this variation in a chroot, but I'm not doing this for reprotest: if you want this variation, use qemu.
  3. User/group will not be portable, because they'll rely on useradd/groupadd and su. useradd and groupadd work on many but not all OSes, notably not including FreeBSD or MacOS X. su was universal in 2004.
  4. Time is not done but will probably be portable to some systems, because it will rely on date -s. Unfortunately, I haven't been able to find any information on how common date -s is across Unix-like OSes, as the -s option is not part of the POSIX standard.
  5. At the moment, I have no idea how to implement changes for /bin/sh and the login shell that will even work across different distributions of Linux, much less different OSes. There are a couple of key problems, starting with the need to find two different shells to use, because there's no way to find out what shells are installed. This blog post explains why /etc/shells doesn't work well for finding what shells are available: not everything in /etc/shells is necessarily a shell (Ubuntu has /usr/bin/screen) and not all available shells are in /etc/shells. Also, there's no good way to find out what shell is the system default because /bin/sh can be an arbitrary binary, not a symlink,and no good way to identify what it is if it is a binary. I can hard-code shell choices, but which shells? bash is obvious for Linux, but what's the best second choice?
On other topics:
  1. reprotest fails to build properly on jessie: Lunar said, and I agree, that fixing this is not a priority. I need someone with more knowledge of Debian Python packaging to help me. If I'm going to support old versions, I also need some kind of CI server, because I don't have the time or ability to maintain old versions of Debian and Python myself.
  2. libc-bin: ldconfig segfaults when run using "setarch uname26": I don't have a good solution for this, but I don't want to hard-code and maintain an architecture-specific disable. Would changing the argument to setarch work around the segfault? Is there a way to test for the presence of the bug that won't cause a segfault or similar crash?
  3. Please put adt-virt-* binaries back onto PATH: reprotest is not affected by this change because I forked the autopkgtest code rather than depending on it, so that reprotest can be installed through PyPi. At the moment, reprotest doesn't make its versions of the programs in virt/ available on $PATH. This is primarily because of the problems with distributing command-line scripts with setuptools. The approach I'm currently using, including the virt/ programs as non-code data with include reprotest/virt/* in MANIFEST.in, doesn't install them to make them available for other programs. Using one of the other approaches potentially could, but it's difficult to make the imports work with those approaches. (I haven't found a way to do it.) I think the best solution to this approach is to split autopkgtest into the Debian-specific components and the general-purpose virtualization components, but I don't have the time to do this myself or to negotiate with Martin Pitt, if he'd even be receptive. I'm also unsure at this point if it wouldn't be better for reprotest to switch from autopkgtest to using something like Ansible to run the virtualization, because Ansible has some solved some of the portability problems already and is not tied to Debian.
My goal is to finish the variations (finally), though as this has always proved more difficult than I expected in the past, I don't make any guarantees. Beyond that, I want to start working on finishing docstrings in the reprotest-specific (i.e., not inherited from autopkgtest) code, improving the documentation in general, and improving the tests.

Reproducible builds folks: Reproducible builds: week 67 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday July 31 and Saturday August 6 2016: Toolchain development and fixes Packages fixed and bugs filed The following 24 packages have become reproducible - in our current test setup - due to changes in their build-dependencies: alglib aspcud boomaga fcl flute haskell-hopenpgp indigo italc kst ktexteditor libgroove libjson-rpc-cpp libqes luminance-hdr openscenegraph palabos petri-foo pgagent sisl srm-ifce vera++ visp x42-plugins zbackup The following packages have become reproducible after being fixed: The following newly-uploaded packages appear to be reproducible now, for reasons we were not able to figure out. (Relevant changelogs did not mention reproducible builds.) Some uploads have addressed some reproducibility issues, but not all of them: Patches submitted that have not made their way to the archive yet: Package reviews and QA These are reviews of reproduciblity issues of Debian packages. 276 package reviews have been added, 172 have been updated and 44 have been removed in this week. 7 FTBFS bugs have been reported by Chris Lamb. Reproducibility tools Test infrastructure For testing the impact of allowing variations of the buildpath (which up until now we required to be identical for reproducible rebuilds), Reiner Herrmann contribed a patch which enabled build path variations on testing/i386. This is possible now since dpkg 1.18.10 enables the --fixdebugpath build flag feature by default, which should result in reproducible builds (for C code) even with varying paths. So far we haven't had many results due to disturbances in our build network in the last days, but it seems this would mean roughly between 5-15% additional unreproducible packages - compared to what we see now. We'll keep you updated on the numbers (and problems with compilers and common frameworks) as we find them. lynxis continued work to test LEDE and OpenWrt on two different hosts, to include date variation in the tests. Mattia and Holger worked on the (mass) deployment scripts, so that the - for space reasons - only jenkins.debian.net GIT clone resides in ~jenkins-adm/ and not anymore in Holger's homedir, so that soon Mattia (and possibly others!) will be able to fully maintain this setup, while Holger is doing siesta. Miscellaneous Chris, dkg, h01ger and Ximin attended a Core Infrastricture Initiative summit meeting in New York City, to discuss and promote this Reproducible Builds project. The CII was set up in the wake of the Heartbleed SSL vulnerability to support software projects that are critical to the functioning of the internet. This week's edition was written by Ximin Luo and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

Thorsten Alteholz: My Debian Activities in July 2016

FTP assistant This month I marked 248 packages for accept and rejected 60. I also sent 13 emails to maintainers asking questions. Again, this was a rather quiet month without much trouble. Debian LTS This was my twenty-fifth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. As the number of participants increases, this month my all in all workload has been only 14.70h. Strangely enough, most of the time I choosed packages, where at the end the vulnerable code of the corresponding CVE was not present in the Wheezy version. So I could mark several CVEs for bind, libgd2 and mupfd as not-affected, without doing an upload. Nevertheless I also did two uploads to fix another two CVEs: As there arrived some new CVEs for PHP5 I didn t do an upload this month. But don t purge your testing environments, a new version is comming soon :-). This month I also had another term of frontdesk work. Other stuff For the Alljoyn framework I took care of RC-bug #829148. I also uploaded a new version of rplay to fix #805959. In the Javascript world I could close #831006

Shirish Agarwal: Doha and the Supreme Court of DFSG Free

Hi, I am in two minds of what to write about Doha. My job has been vastly simplified by a friend when he shared with me https://www.youtube.com/watch?v=LdrAd-44LW0 . That video is more relevant and more closer to the truth than whatever I can share. As can be seen it is funny but more sad the way Qatarians are trying to figure out how things will be and as can be seen it seems to heading towards a real estate bubble . They would have to let go of the Sharia if they are thinking of wealthy westerners coming to stay put. I am just sad to know that many of my country-men are stuck there and although I hope the best for them, I dread it may turn out the way it has turned out for many people of Indians, and especially from Kerala in Saudi Arabia. I would touch about the Kerala situation probably in another blog post as this time is exclusively for legal aspects which were discussed in Debconf. A bit of backgrounder here, one part of my family is lawyers which means I have somewhat notion of law as practiced in our land. As probably everybody knows, India was ruled by the British for around 150 odd years. One of the things that they gave while leaving was/is the IPC (Indian Penal Code) and is practiced with the common law concept. The concept means precedence of any judgement goes quite some way in framing rulings and law of the land as time goes on besides the lobbying and the politics which happens in any democracy. Free software would not have been there without the GPL The General Public License. And the license is as much a legal document as it s something that the developers can work without becoming deranged, as it is one of the more simpler licenses to work with. My own understanding of the legal, ethical and moral issues around me were framed by two-three different TV shows, books (fiction and non-fiction alike) apart from what little news I heard in family. One was M*A*S*H* (with Alan Alda and his frailness, anarchism, humanism, civil rights), the Practise and Boston Legal which does lay bare the many grey areas that lawyers have to deal with ( The Practice also influenced a lot of civil rights understanding and First amendment, but as it is a TV show, how much of it is actually practiced for lawyers and how much moral dilemma they are can only be guessed at.) . In books it is artists like John Grisham, Michael Connelly as well as Perry Mason Agatha Christie. In non-fiction look at the treasures under bombayhighcourt e-books corner and series of Hamlyn Lectures. I would have to warn that all of the above are major time-sinks but rewarding in their own way. Also haven t read all of them as time and interests are constrained but do know they are good for understanding bit of our history. I do crave for a meetup kind of scenario when non-lawyers can read and discuss about facets of law . All that understanding was vastly amplified by Groklaw.net which made non-lawyers at the very least be able to decipher and understand what is going on in the free software world. After PJ (Pamela Jones) closed it in 2013 due to total surveillance by the Free World (i.e. the United States of America, NSA) we have been thirsty. We do get occasionally somewhat mildly interesting articles in lwn.net or arstechnica.net but nowhere the sheer brilliance of groklaw. So, it was a sheer stroke of luck that I met Mr. Bradley M. Kuhn who works with Karen Sandler on Software Conservancy. While I wanted to be there for his presentation, it was just one of those days which doesn t go as planned. However, as we met socially and over e-mail there were two basic questions which I asked him which also imbibes why we need to fight for software freedom in the court of law. Below is a re-wording of what he shared . Q1. why do people think that GPL still needs to be challenged in the court of law while there are gpl-violations which has been more or less successfully defended in the court of law ? Bradley Kuhn the GPL violations is basically a violation of one or more clauses of the GPL license and not the GPL license as a whole and my effort during my lifetime would be to make/have such precedents that the GPL is held as a valid license in the court of law. Q2. Let s say IF GPL is held to be valid in the court of law, would FSF benefit monetarily, at least to my mind it might be so, as more people and comapnies could be convinced to use strong copyleft licenses such as GPLv3 or AGPLv3 . Bradley Kuhn It may or may not. It is possible that even after winning, that people and especially companies may go for weak copyleft licenses if it suits them. The only benefit would probably would be to those people who are already using GPLv3 as the law could be used to protect them as well. Although we would want and welcome companies who would use strong copyleft license such as the GPL, the future is in future and hence uncertain. Both possibilities co-exist. While Bradley didn t say it, I would add further here it probably would mean also moving from being a more offensive mode (which GPL-violations is based upon where a violation occurs and somebody either from the victim s side or a by-stander notices the violation, brings it to the notice of the victim and the GPL-volations team.) to perhaps it being defended by the DMCA people themselves, once GPL is held as a valid license in the eyes of law. Although should you use the DMCA or not is a matter of choice, personal belief system as well as your legal recourses. I have to share that the FSF and the GPL-violations team are probably very discerning when they take up the fight as most of the work done by them is pro-bono (i.e. they don t make a single penny/paisa from the work done therein.) and hence in view of scarce resources, it makes sense to go only for the biggest violators in the hopes that you can either make them agree to compensate and agree to the terms of license of any software/hardware combination or sue them and take a bigger share of the reward/compensation awarded by the Court to help the defendant and maybe some of the proceeds donated by the defendant and people like you and me to make sure that Conservancy and the GPL-violations team is still around to help the next time something similar happens.
Bradley Kuhn presenting at #Debconf 16

Bradley Kuhn presenting at #Debconf 16

Now, as far as his presentation is concerned, whose video can be seen at http://meetings-archive.debian.net/pub/debian-meetings/2016/debconf16/The_Supreme_Court_of_DFSGFree.webm , I thought it was tame. While he talked about gaming the system in some sense, he was sharing that the system debian-legal works (most-of-the-time). The list actually works because many far more brilliant people than me take time to understand the intricacies of various licenses and how they should be interpreted through the excellently written Debian Free Software Guidelines and whether the license under discussion contravenes the DFSG or is part of it. I do agree with his point though that the ftp-master/s and the team may not be the right person to judge the license in adherence to the DFSG, or her/is not giving a reason for rejecting a package to not entering into the package archive. I actually asked the same question on debian-legal and while I had guessed, it seems there is enough review of the licenses per-se as answer from Paul Wise shows. Charles Pessley also shared an idea he has documented which probably didn t get much traction as involves more work on DD s without any benefit to show for it. All in all I hope it sheds some light on why there is need to be more aware of law in software freedom. Two Organizations which work on software freedom from legal standpoint are SFLC (Delhi) headed by the charming Mr. Eben Moglen and ALF (Bangalore). I do hope more people, especially developers take a bit more interest in some of the resources mentioned above.
Filed under: Miscellenous Tagged: #Alternative Law Forum, #bombayhighcourt e-library, #Common Law, #Debconf16, #Fiction, #Hewlyn lectures, #India, #Jurispudence, #legal fiction, #real estate bubble, #SFLC.in, #Software Freedom, #timesink, Doha, Law

8 August 2016

Michael Stapelberg: Debian Code Search: improving client-side latency

A while ago, it occurred to me that querying Debian Code Search seemed slow, which surprised me because I previously spent quite some effort on making it faster, see Debian Code Search Instant and Taming the latency tail for the most recent substantial architecture overhaul and related optimizations. Upon taking a closer look, I realized that while performing the search query on the server side was pretty fast, the perceived slowness was due to the client side being slow. By being slow , I mean that it took a long time until something was drawn on the screen (high latency) and that what was happening on the screen was janky (stuttering, not smooth). Part of that slowness was due to historical reasons: the client-side architecture was optimized for the use-case where users open Debian Code Search s index page and then submit a search query, but I was using Chrome s address bar to send a search query (type codesearch , then hit the TAB key). Further, we only added a non-JavaScript version after we launched the JavaScript version. Hence, the redirects and progressive enhancements we implemented are more of a kludge than a well thought out design. After this bit of original investigation, I opened GitHub issue #69 to track the work on making Debian Code Search faster. In that issue, I captured how Chrome s network inspector visualizes the work necessary to render the page: Chrome network inspector: before A couple of quick wins There are a couple of little fixes and improvements on which I m not going to spend too much time on, but which I list for completeness anyway just in case they come in handy for a similar project of yours: Bigger changes The URL pattern has changed. Previously, we had 2 areas of the website, one for JavaScript-compatible clients and one for the rest. When you hit the wrong one, you were redirected. In some areas, we couldn t tell which area is the correct one for you, so you would always incur a redirect: one example for this is the search bar. With the new URL pattern, we deliver both versions under the same URL: the elements only used by the JavaScript code are hidden using CSS by default, then made visible by JavaScript code. The elements only used by the non-JavaScript code are wrapped in a <noscript> tag. All CSS which is required for the initial page rendering is now inlined in the responses, allowing the browser to immediately render a response without requiring any additional round trips. All non-essential CSS has been moved into a separate CSS file which is loaded asynchronously. This is done using a pattern like <link rel="preload" href="foo.css" as="style" onload="this.rel='stylesheet'">, see also filamentgroup/loadCSS. We switched from WebSockets to the EventSource API because the former is not compatible with HTTP/2, whereas the latter is. This removes a round trip and some custom code for WebSocket reconnecting, because EventSource does that for you. The progress bar animation used to animate the background-position property. It turns out that browsers can only animate the position, scale, rotation and opacity properties smoothly, because such animations can be off-loaded to the GPU. Hence, we have re-implemented the progress bar animation using the position property. The biggest win for improving client-side latency from the Chrome address bar was introducing Service Workers (see commit 7f31aef402cb782056e290a797f224171f4af270). Our Service Worker caches static assets and a placeholder results page. The placeholder page is presented immediately when you start a search (e.g. from the address bar), making the first response immediate, i.e. rendered within 100ms. Having assets and the result page out of the way, the first round trip is used for actually doing the search, removing all unnecessary overhead. With all of these improvements in place, rendering latency goes down from half a second to well under 100 ms, and this is what the Chrome network inspector looks like: Chrome network inspector: after

7 August 2016

Dirk Eddelbuettel: littler 0.3.1

max-heap image The second release of littler as a CRAN package is now available, following in the now more than ten-year history as a package started by Jeff in the summer of 2006, and joined by me a few weeks later. littler is the first command-line interface for R and predates Rscript. It is still faster, and in my very biased eyes better as it allows for piping as well shebang scripting via #!, uses command-line arguments more consistently and still starts faster. It prefers to live on Linux and Unix, has its difficulties on the OS X due yet-another-braindeadedness there (who ever thought case-insensitive filesystems where a good idea?) and simply does not exist on Windows (yet -- the build system could be extended -- see RInside for an existence proof, and volunteers welcome!). This release brings us fixes and enhancements from three other contributors, a couple new example scripts, more robust builds, extended documentation and more -- see below for details. The NEWS file entry is below.
Changes in littler version 0.3.1 (2016-08-06)
  • Changes in examples
    • install2.r now passes on extra options past -- to R CMD INSTALL (PR #37 by Steven Pav)
    • Added rcc.r to run rcmdcheck::rcmdcheck()
    • Added (still simple) render.r to render (R)markdown
    • Several examples now support the -x or --usage flag to show extended help.
  • Changes in build system
    • The AM_LDFLAGS variable is now set and used too (PR #38 by Mattias Ellert)
    • Three more directories, used when an explicit installation directory is set, are excluded (also #38 by Mattias)
    • Travis CI is now driven via run.sh from our fork, and deploys all packages as .deb binaries using our PPA where needed
  • Changes in package
    • SystemRequirements now mentions the need for libR, i.e. an R built with a shared library so that we can embed R.
    • The docopt and rcmdcheck packages are now suggested, and added to the Travis installation.
    • A new helper function r() is now provided and exported so that the package can be imported (closes #40).
    • URL and BugReports links were added to DESCRIPTION.
  • Changes in documentation
    • The help output for installGithub.r was corrected (PR #39 by Brandon Bertelsen)
Full details for the littler release are provided as usual at the ChangeLog page. The code is available via the GitHub repo, from tarballs off my littler page and the local directory here -- and now of course all from its CRAN page and via install.packages("littler"). Binary packages are available directly in Debian as well as soon via Ubuntu binaries at CRAN thanks to the tireless Michael Rutter. will probably have new Comments and suggestions are welcome at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

6 August 2016

Mirco Bauer: Ethereum GPU Mining on Linux How-To

TL;DR Install/use Debian 8 or Ubuntu 16.0.4 then execute: 
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:ethereum/ethereum
sudo sed 's/jessie/vivid/' -i /etc/apt/sources.list.d/ethereum-ethereum-*.list
sudo apt-get update
sudo apt-get install ethereum ethminer
geth account new
# copy long character sequence within  , that is your <YOUR_WALLET_ADDRESS>
# if you lose the passphrase, you lose your coins!
sudo apt-get install linux-headers-amd64 build-essential
chmod +x NVIDIA-Linux-x86_64-367.35.run
sudo NVIDIA-Linux-x86_64-367.35.run
ethminer -G -F http://yolo.ethclassic.faith:9999/0x<YOUR_WALLET_ADDRESS> --farm-recheck 200
echo done
My Attention Span is > 60 seconds Ethereum is a crypto currency similar to Bitcoin as it is based on the blockchain technology. Ethereum is not yet another Bitcoin clone though, since it has an additional feature called Smart Contracts that makes it unique and very promising. I am not going into details how Ethereum works, you can get that into great detail on the Internet. This post is about Ethereum mining. Mining is how crypto coins are created. You need to spent computing time to get coins out. At the beginning CPU mining was sufficient, but as the Ethereum network difficulty has increased you need to use GPUs as they can calculate at a much higher hashrate than a general purpose CPU can do. About 2 months ago I bought a new gaming rig, with a Nvidia GTX 1070 so I can experience virtual-reality gaming with a HTC Vive at a great framerate. As it turns out modern graphics cards are very good at hashing so I gave it a spin. Initially I did this mining setup with Windows 10, as that is the operating system on my gaming rig. If you want to do Ethereum mining using your GPU, then you really want to use Linux. On Windows the GTX 1070 produced a hashrate of 6 MH/s (megahashes per second) while the same hardware does 25 MH/s on Linux. The hashrate multiplied by 4 by using Linux instead of Windows. Sounds good? Keep reading and follow this guide. You have to pick a Linux distro to use for mining. As I am a Debian developer, all my systems run Debian, which is what I am also using for this guide. The same procedure can be done for Ubuntu as it is similar enough. For other distros you have to substitute the steps yourself. So I assume you already have Debian 8 or Ubuntu 16.04 installed on your system.

Install Ethereum Software First we need the geth tool which is the main Ethereum "client". Ethereum is really a peer-to-peer network, that means each node is a server and client at the same time. A node that contains the complete blockchain history in a database is called a full node. For this guide you don't need to run a full node, as mining pools do this for you. We still need geth to create the private key of your Ethereum wallet. Somewhere we have to receive the coins we are mining ;) Add the Ethereum APT repository using these commands: 
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:ethereum/ethereum
sudo apt-get update
On Debian 8 (on Ubuntu you can skip this) you need to replace the repository name with this command: 
sudo sed 's/jessie/vivid/' -i /etc/apt/sources.list.d/ethereum-ethereum-*.list
sudo apt-get update
Install ethereum, ethminer and geth: 
sudo apt-get install ethereum ethminer geth

Create Ethereum Wallet A wallet is where coins are "stored". They are not really stored in the wallet because the wallet is just a private key that nobody has. The balance of that wallet is visible to everyone using the blockchain database. And this is what full nodes do, they contain and distribute the database to all other peers. So this this command to create your first private key for your wallet: 
geth account new
Be aware, that this passphrase protects the private key of your wallet. Anyone who has access to that file and knows your passphrase will have full control over your coins. And also do not forget the passphrase, as if you do, you lost all your coins! The output of "geth account new" shows a long character/number sequence quoted in . This is your wallet address and you should write that number down, as if someone wants to send you money, then it is to that address. We will use that for the mining pool later.

Install (proprietary) nvidia driver For OpenCL to work with nvidia graphics cards, like my GTX 1070, you need to install this proprietary driver from nvidia. If you have an older card maybe the opensource drivers will work for you. For the nvidia pascal cards numbers 10xx you will need this driver package. After you have agreed the terms, download the NVIDIA-Linux-x86_64-367.35.run file. But before we can use that installer we need to install some dependencies that installer needs as it will have to compile a Linux kernel module for you. Install the dependencies using this command: 
sudo apt-get install linux-headers-amd64 build-essential
Now we can make the installer executable and run it like this: 
chmod +x NVIDIA-Linux-x86_64-367.35.run
sudo NVIDIA-Linux-x86_64-367.35.run
If that step completed without error, then we should be able to run the mining benchmark! 
ethminer -M -G
The -M means "run benchmark" and the -G is for GPU mining. The first time you run it it will create a DAG file and that will takes a while. For me it took about 12 minutes on my GTX 1070. After that is should show a inner mean hashrate. If it says H/s that is hashes per second and KH is kilo (H/1000) and MH is megahashes per second (KH/1000). I had numbers around 25-30 MH/s, but for real mining you will see an average that is a balanced number and not a min/max range.

Pick Ethereum Network Now it gets serious, you need to decide 2 things. First which Ethereum network you want to mine for and the second is using which pool. Ethereum has 2 networks, one is called Ethereum One or Core, while the other is called Ethereum Classic. Ethereum has made a hardfork to undo the consequences of a software bug in the DAO. The DAO is a smart contract for a decentralized organization. Because of that bug, a blackhat could use that bug to obtain money from that DAO. The Ethereum developers made a poll and decided that the consequences will be undone. Not everyone agreed and the old network stayed alive and is now called Ethereum Classic short ETC. The hardfork kept its short name ETH. This is important to understand for mining, because the hashing difficulty has a huge difference between ETH and ETC. As of writing, the hashrate of ETC is at 20% compared to ETH. Thus you need less computing time to get ETC coins and more time to get ETH coins. Differently said, ETC mining is currently more profitable.

Pick a Pool Hmmmm, I want a swimming pool, thanks! Just kidding... You can mine without a pool, that is called solo mining, but you will get less reward. A mining pool are multiple computers that work on the same block to find a solution quicker than others. The pool has an aggregated hashrate that is higher than other solo miners. Each found block by anyone in this pool will be rewarded to everyone in the pool. The reward of 5 ether currently per block gets split in the same ratio of hashrate each member provides (minus the pool fee). So while you get less for a found block, you still have a steady lower income rate instead of higher with less chance of finding one (in time). Simply said: you have to find a new block faster than the others to receive the reward. If you want to mine Ethereum Classic (ETC) use one of the pools listed here (at the bottom of the page). If you want to mine Ethereum One / Core (ETH) use one of the pools listed here.

Run ethminer The instruction page of the pool website usually says how to start the miner program, but here is an example of the pool that I use (because pony!): 
ethminer -G -F http://yolo.ethclassic.faith:9999/0x<YOUR_WALLET_ADDRESS> --farm-recheck 200

Profit If this guide was helpful for you, you can tip me at ethereum:0x9ec1220d2f2fadd3f0c96e3007daa827bc83fbd6 or simply run the ethminer using my wallet address for a day or two: 
ethminer -G -F http://yolo.ethclassic.faith:9999/0x9ec1220d2f2fadd3f0c96e3007daa827bc83fbd6 --farm-recheck 200
Happy mining!

5 August 2016

Steve Kemp: Using the compiler to help you debug segfaults

Recently somebody reported that my console-based mail-client was segfaulting when opening an IMAP folder, and then when they tried with a local Maildir-hierarchy the same fault was observed. I couldn't reproduce the problem at all, as neither my development host (read "my personal desktop"), nor my mail-host had been crashing at all, both being in use to read my email for several months. Debugging crashes with no backtrace, or real hint of where to start, is a challenge. Even when downloading the same Maildir samples I couldn't see a problem. It was only when I decided to see if I could add some more diagnostics to my code that I came across a solution. My intention was to make it easier to receive a backtrace, by adding more compiler options: 
  -fsanitize=address -fno-omit-frame-pointer
I added those options and my mail-client immediately started to segfault on my own machine(s), almost as soon as it started. Ultimately I found three pieces of code where I was allocating C++ objects and passing them to the Lua stack, a pretty fundamental part of the code, which were buggy. Once I'd tracked down the areas of code that were broken and fixed them the user was happy, and I was happy too. Its interesting that I've been running for over a year with these bogus things in place, which "just happened" to not crash for me or anybody else. In the future I'll be adding these options to more of my C-based projects, as there seems to be virtually no downside. In related news my console editor has now achieved almost everything I want it to, having gained: The only outstanding feature, which is a biggy, is support for Undo which I need to add. Happily no segfaults here, so far..

Next.

Previous.