Search Results: "directhex"

12 September 2023

Jo Shields: Building a NAS

The status quo Back in 2015, I bought an off-the-shelf NAS, a QNAP TS-453mini, to act as my file store and Plex server. I had previously owned a Synology box, and whilst I liked the Synology OS and experience, the hardware was underwhelming. I loaded up the successor QNAP with four 5TB drives in RAID10, and moved all my files over (after some initial DoA drive issues were handled).
QNAP TS-453mini product photoQNAP TS-453mini product photo
That thing has been in service for about 8 years now, and it s been a mixed bag. It was definitely more powerful than the predecessor system, but it was clear that QNAP s OS was not up to the same standard as Synology s perhaps best exemplified by HappyGet 2 , the QNAP webapp for downloading videos from streaming services like YouTube, whose icon is a straight rip-off of StarCraft 2. On its own, meaningless but a bad omen for overall software quality
The logo for QNAP HappyGet 2 and Blizzard's Starcraft 2 side by sideThe logo for QNAP HappyGet 2 and Blizzard s StarCraft 2 side by side
Additionally, the embedded Celeron processor in the NAS turned out to be an issue for some cases. It turns out, when playing back videos with subtitles, most Plex clients do not support subtitles properly instead they rely on the Plex server doing JIT transcoding to bake the subtitles directly into the video stream. I discovered this with some Blu-Ray rips of Game of Thrones some episodes would play back fine on my smart TV, but episodes with subtitled Dothraki speech would play at only 2 or 3 frames per second. The final straw was a ransomware attack, which went through all my data and locked every file below a 60MiB threshold. Practically all my music gone. A substantial collection of downloaded files, all gone. Some of these files had been carried around since my college days digital rarities, or at least digital detritus I felt a real sense of loss at having to replace. This episode was caused by a ransomware targeting specific vulnerabilities in the QNAP OS, not an error on my part. So, I decided to start planning a replacement with:
  • A non-garbage OS, whilst still being a NAS-appliance type offering (not an off-the-shelf Linux server distro)
  • Full remote management capabilities
  • A small form factor comparable to off-the-shelf NAS
  • A powerful modern CPU capable of transcoding high resolution video
  • All flash storage, no spinning rust
At the time, no consumer NAS offered everything (The Asustor FS6712X exists now, but didn t when this project started), so I opted to go for a full DIY rather than an appliance not the first time I ve jumped between appliances and DIY for home storage.

Selecting the core of the system There aren t many companies which will sell you a small motherboard with IPMI. Supermicro is a bust, so is Tyan. But ASRock Rack, the server division of third-tier motherboard vendor ASRock, delivers. Most of their boards aren t actually compliant Mini-ITX size, they re a proprietary Deep Mini-ITX with the regular screw holes, but 40mm of extra length (and a commensurately small list of compatible cases). But, thankfully, they do have a tiny selection of boards without the extra size, and I stumbled onto the X570D4I-2T, a board with an AMD AM4 socket and the mature X570 chipset. This board can use any AMD Ryzen chip (before the latest-gen Ryzen 7000 series); has built in dual 10 gigabit ethernet; IPMI; four (laptop-sized) RAM slots with full ECC support; one M.2 slot for NVMe SSD storage; a PCIe 16x slot (generally for graphics cards, but we live in a world of possibilities); and up to 8 SATA drives OR a couple more NVMe SSDs. It s astonishingly well featured, just a shame it costs about $450 compared to a good consumer-grade Mini ITX AM4 board costing less than half that. I was so impressed with the offering, in fact, that I crowed about it on Mastodon and ended up securing ASRock another sale, with someone else looking into a very similar project to mine around the same timespan. The next question was the CPU. An important feature of a system expected to run 24/7 is low power, and AM4 chips can consume as much as 130W under load, out of the box. At the other end, some models can require as little as 35W under load the OEM-only GE suffix chips, which are readily found for import on eBay. In their PRO variant, they also support ECC (all non-G Ryzen chips support ECC, but only Pro G chips do). The top of the range 8 core Ryzen 7 PRO 5750GE is prohibitively expensive, but the slightly weaker 6 core Ryzen 5 PRO 5650GE was affordable, and one arrived quickly from Hong Kong. Supplemented with a couple of cheap 16 GiB SODIMM sticks of DDR4 PC-3200 direct from Micron for under $50 a piece, that left only cooling as an unsolved problem to get a bootable test system. The official support list for the X570D4I-2T only includes two rackmount coolers, both expensive and hard to source. The reason for such a small list is the non standard cooling layout of the board instead of an AM4 hole pattern with the standard plastic AM4 retaining clips, it has an Intel 115x hole pattern with a non-standard backplate (Intel 115x boards have no backplate, the stock Intel 115x cooler attaches to the holes with push pins). As such every single cooler compatibility list excludes this motherboard. However, the backplate is only secured with a mild glue with minimal pressure and a plastic prying tool it can be removed, giving compatibility with any 115x cooler (which is basically any CPU cooler for more than a decade). I picked an oversized low profile Thermalright AXP120-X67 hoping that its 120mm fan would cool the nearby MOSFETs and X570 chipset too.
Thermalright AXP120-X67, AMD Ryzen 5 PRO 5650GE, ASRock Rack X570D4I-2T, all assembled and running on a flat surface

Testing up to this point Using a spare ATX power supply, I had enough of a system built to explore the IPMI and UEFI instances, and run MemTest86 to validate my progress. The memory test ran without a hitch and confirmed the ECC was working, although it also showed that the memory was only running at 2933 MT/s instead of the rated 3200 MT/s (a limit imposed by the motherboard, as higher speeds are considered overclocking). The IPMI interface isn t the best I ve ever used by a long shot, but it s minimum viable and allowed me to configure the basics and boot from media entirely via a Web browser.
Memtest86 showing test progress, taken from IPMI remote control window
One sad discovery, however, which I ve never seen documented before, on PCIe bifurcation. With PCI Express, you have a number of lanes which are allocated in groups by the motherboard and CPU manufacturer. For Ryzen prior to Ryzen 7000, that s 16 lanes in one slot for the graphics card; 4 lanes in one M.2 connector for an SSD; then 4 lanes connecting the CPU to the chipset, which can offer whatever it likes for peripherals or extra lanes (bottlenecked by that shared 4x link to the CPU, if it comes down to it). It s possible, with motherboard and CPU support, to split PCIe groups up for example an 8x slot could be split into two 4x slots (eg allowing two NVMe drives in an adapter card NVME drives these days all use 4x). However with a Cezanne Ryzen with integrated graphics, the 16x graphics card slot cannot be split into four 4x slots (ie used for for NVMe drives) the most bifurcation it allows is 8x4x4x, which is useless in a NAS.
Screenshot of PCIe 16x slot bifurcation options in UEFI settings, taken from IPMI remote control window
As such, I had to abandon any ideas of an all-NVMe NAS I was considering: the 16x slot split into four 4x, combined with two 4x connectors fed by the X570 chipset, to a total of 6 NVMe drives. 7.6TB U.2 enterprise disks are remarkably affordable (cheaper than consumer SATA 8TB drives), but alas, I was locked out by my 5650GE. Thankfully I found out before spending hundreds on a U.2 hot swap bay. The NVMe setup would be nearly 10x as fast as SATA SSDs, but at least the SATA SSD route would still outperform any spinning rust choice on the market (including the fastest 10K RPM SAS drives)

Containing the core The next step was to pick a case and power supply. A lot of NAS cases require an SFX (rather than ATX) size supply, so I ordered a modular SX500 unit from Silverstone. Even if I ended up with a case requiring ATX, it s easy to turn an SFX power supply into ATX, and the worst result is you have less space taken up in your case, hardly the worst problem to have. That said, on to picking a case. There s only one brand with any cachet making ITX NAS cases, Silverstone. They have three choices in an appropriate size: CS01-HS, CS280, and DS380. The problem is, these cases are all badly designed garbage. Take the CS280 as an example, the case with the most space for a CPU cooler. Here s how close together the hotswap bay (right) and power supply (left) are:
Internal image of Silverstone CS280 NAS build. Image stolen from ServeTheHome
With actual cables connected, the cable clearance problem is even worse:
Internal image of Silverstone CS280 NAS build. Image stolen from ServeTheHome
Remember, this is the best of the three cases for internal layout, the one with the least restriction on CPU cooler height. And it s garbage! Total hot garbage! I decided therefore to completely skip the NAS case market, and instead purchase a 5.25 -to-2.5 hot swap bay adapter from Icy Dock, and put it in an ITX gamer case with a 5.25 bay. This is no longer a served market 5.25 bays are extinct since nobody uses CD/DVD drives anymore. The ones on the market are really new old stock from 2014-2017: The Fractal Design Core 500, Cooler Master Elite 130, and Silverstone SUGO 14. Of the three, the Fractal is the best rated so I opted to get that one however it seems the global supply of new old stock fully dried up in the two weeks between me making a decision and placing an order leaving only the Silverstone case. Icy Dock have a selection of 8-bay 2.5 SATA 5.25 hot swap chassis choices in their ToughArmor MB998 series. I opted for the ToughArmor MB998IP-B, to reduce cable clutter it requires only two SFF-8611-to-SF-8643 cables from the motherboard to serve all eight bays, which should make airflow less of a mess. The X570D4I-2T doesn t have any SATA ports on board, instead it has two SFF-8611 OCuLink ports, each supporting 4 PCI Express lanes OR 4 SATA connectors via a breakout cable. I had hoped to get the ToughArmor MB118VP-B and run six U.2 drives, but as I said, the PCIe bifurcation issue with Ryzen G chips meant I wouldn t be able to run all six bays successfully.
NAS build in Silverstone SUGO 14, mid build, panels removed
Silverstone SUGO 14 from the front, with hot swap bay installed

Actual storage for the storage server My concept for the system always involved a fast boot/cache drive in the motherboard s M.2 slot, non-redundant (just backups of the config if the worst were to happen) and separate storage drives somewhere between 3.8 and 8 TB each (somewhere from $200-$350). As a boot drive, I selected the Intel Optane SSD P1600X 58G, available for under $35 and rated for 228 years between failures (or 11,000 complete drive rewrite cycles). So, on to the big expensive choice: storage drives. I narrowed it down to two contenders: new-old-stock Intel D3-S4510 3.84TB enterprise drives, at about $200, or Samsung 870 QVO 8TB consumer drives, at about $375. I did spend a long time agonizing over the specification differences, the ZFS usage reports, the expected lifetime endurance figures, but in reality, it came down to price $1600 of expensive drives vs $3200 of even more expensive drives. That s 27TB of usable capacity in RAID-Z1, or 23TB in RAID-Z2. For comparison, I m using about 5TB of the old NAS, so that s a LOT of overhead for expansion.
Storage SSD loaded into hot swap sled

Booting up Bringing it all together is the OS. I wanted an appliance NAS OS rather than self-administering a Linux distribution, and after looking into the surrounding ecosystems, decided on TrueNAS Scale (the beta of the 2023 release, based on Debian 12).
TrueNAS Dashboard screenshot in browser window
I set up RAID-Z1, and with zero tuning (other than enabling auto-TRIM), got the following performance numbers:
IOPSBandwidth
4k random writes19.3k75.6 MiB/s
4k random reads36.1k141 MiB/s
Sequential writes 2300 MiB/s
Sequential reads 3800 MiB/s
Results using fio parameters suggested by Huawei
And for comparison, the maximum theoretical numbers quoted by Intel for a single drive:
IOPSBandwidth
4k random writes16k?
4k random reads90k?
Sequential writes 280 MiB/s
Sequential reads 560 MiB/s
Numbers quoted by Intel SSD successors Solidigm.
Finally, the numbers reported on the old NAS with four 7200 RPM hard disks in RAID 10:
IOPSBandwidth
4k random writes4301.7 MiB/s
4k random reads800632 MiB/s
Sequential writes 311 MiB/s
Sequential reads 566 MiB/s
Performance seems pretty OK. There s always going to be an overhead to RAID. I ll settle for the 45x improvement on random writes vs. its predecessor, and 4.5x improvement on random reads. The sequential write numbers are gonna be impacted by the size of the ZFS cache (50% of RAM, so 16 GiB), but the rest should be a reasonable indication of true performance. It took me a little while to fully understand the TrueNAS permissions model, but I finally got Plex configured to access data from the same place as my SMB shares, which have anonymous read-only access or authenticated write access for myself and my wife, working fine via both Linux and Windows. And that s it! I built a NAS. I intend to add some fans and more RAM, but that s the build. Total spent: about $3000, which sounds like an unreasonable amount, but it s actually less than a comparable Synology DiskStation DS1823xs+ which has 4 cores instead of 6, first-generation AMD Zen instead of Zen 3, 8 GiB RAM instead of 32 GiB, no hardware-accelerated video transcoding, etc. And it would have been a whole lot less fun!
The final system, powered up
(Also posted on PCPartPicker)

23 August 2023

Jo Shields: Retirement

Apparently it s nearly four years since I last posted to my blog. Which is, to a degree, the point here. My time, and priorities, have changed over the years. And this lead me to the decision that my available time and priorities in 2023 aren t compatible with being a Debian or Ubuntu developer, and realistically, haven t been for years. As of earlier this month, I quit as a Debian Developer and Ubuntu MOTU. I think a lot of my blogging energy got absorbed by social media over the last decade, but with the collapse of Twitter and Reddit due to mismanagement, I m trying to allocate more time for blog-based things instead. I may write up some of the things I ve achieved at work (.NET 8 is now snapped for release Soon ). I might even blog about work-adjacent controversial topics, like my changed feelings about the entire concept of distribution packages. But there s time for that later. Maybe. I ll keep tagging vaguely FOSS related topics with the Debian and Ubuntu tags, which cause them to be aggregated in the Planet Debian/Ubuntu feeds (RSS, remember that from the before times?!) until an admin on those sites gets annoyed at the off-topic posting of an emeritus dev and deletes them. But that s where we are. Rather than ignore my distro obligations, I ve admitted that I just don t have the energy any more. Let someone less perpetually exhausted than me take over. And if they don t, maybe that s OK too.

24 March 2017

Jo Shields: Mono repository changes, beginning Mono vNext

Up to now, Linux packages on mono-project.com have come in two flavours RPM built for CentOS 7 (and RHEL 7), and .deb built for Debian 7. Universal packages that work on the named distributions, and anything newer. Except that s not entirely true. Firstly, there have been compatibility repositories users need to add, to deal with ABI changes in libtiff, libjpeg, and Apache, since Debian 7. Then there s the packages for ARM64 and PPC64el neither of those architectures is available in Debian 7, so they re published in the 7 repo but actually built on 8. A large reason for this is difficulty in our package publishing pipeline apt only allows one version-architecture mix in the repository at once, so I can t have, say, 4.8.0.520-0xamarin1 built on AMD64 on both Debian 7 and Ubuntu 16.04. We ve been working hard on a new package build/publish pipeline, which can properly support multiple distributions, based on Jenkins Pipeline. This new packaging system also resolves longstanding issues such as can t really build anything except Mono and Architecture: All packages still get built on Jo s laptop, with no public build logs So, here s the old build matrix:
Distribution Architectures
Debian 7 ARM hard float, ARM soft float, ARM64 (actually Debian 8), AMD64, i386, PPC64el (actually Debian 8)
CentOS 7 AMD64
And here s the new one:
Distribution Architectures
Debian 7 ARM hard float (v7), ARM soft float, AMD64, i386
Debian 8 ARM hard float (v7), ARM soft float, ARM64, AMD64, i386, PPC64el
Raspbian 8 ARM hard float (v6)
Ubuntu 14.04 ARM hard float (v7), ARM64, AMD64, i386, PPC64el
Ubuntu 16.04 ARM hard float (v7), ARM64, AMD64, i386, PPC64el
CentOS 6 AMD64, i386
CentOS 7 AMD64
The compatibility repositories will no longer be needed on recent Ubuntu or Debian just use the right repository for your system. If your distribution isn t listed sorry, but we need to draw a line somewhere on support, and the distributions listed here are based on heavy analysis of our web server logs and bug requests. You ll want to change your package manager repositories to reflect your system more accurately, once Mono vNext is published. We re debating some kind of automated handling of this, but I m loathe to touch users sources.list without their knowledge. CentOS builds are going to be late I ve been doing all my prototyping against the Debian builds, as I have better command of the tooling. Hopefully no worse than a week or two. edit I guess Ubuntu 12.04 is coming back too, despite being EOL, for TravisCI support.

4 December 2016

Jo Shields: A quick introduction to Flatpak

Releasing ISV applications on Linux is often hard. The ABI of all the libraries you need changes seemingly weekly. Hence you have the option of bundling the world, or building a thousand releases to cover a thousand distribution versions. As a case in point, when MonoDevelop started bundling a C Git library instead of using a C# git implementation, it gained dependencies on all sorts of fairly weak ABI libraries whose exact ABI mix was not consistent across any given pair of distro releases. This broke our policy of releasing works on anything .deb and .rpm packages. As a result, I pretty much gave up on packaging MonoDevelop upstream with version 5.10. Around the 6.1 release window, I decided to take re-evaluate question. I took a closer look at some of the fancy-pants new distribution methods that get a lot of coverage in the Linux press: Snap, AppImage, and Flatpak. I started with AppImage. It s very good and appealing for its specialist areas (no external requirements for end users), but it s kinda useless at solving some of our big areas (the ABI-vs-bundling problem, updating in general). Next, I looked at Flatpak (once xdg-app). I liked the concept a whole lot. There s a simple 3-tier dependency hierarchy: Applications, Runtimes, and Extensions. An application depends on exactly one runtime. Runtimes are root-level images with no dependencies of their own. Extensions are optional add-ons for applications. Anything not provided in your target runtime, you bundle. And an integrated updates mechanism allows for multiple branches and multiple releases parallel-installed (e.g. alpha & stable, easily switched). There s also security-related sandboxing features, but my main concerns on a first examination were with the dependency and distribution questions. That said, some users might be happier running Microsoft software on their Linux desktop if that software is locked up inside a sandbox, so I ve decided to embrace that functionality rather than seek to avoid it. I basically stopped looking at this point (sorry Snap!). Flatpak provided me with all the functionality I wanted, with an extremely helpful and responsive upstream. I got to work on trying to package up MonoDevelop. Flatpak (optionally!) uses a JSON manifest for building stuff. Because Mono is still largely stuck in a Gtk+2 world, I opted for the simplest runtime, org.freedesktop.Runtime, and bundled stuff like Gtk+ into the application itself. Some gentle patching here & there resulted in this repository. Every time I came up with an exciting new edge case, upstream would suggest a workaround within hours or failing that, added new features to Flatpak just to support my needs (e.g. allowing /dev/kvm to optionally pass through the sandbox). The end result is, as of the upcoming 0.8.0 release of Flatpak, from a clean install of the flatpak package to having a working MonoDevelop is a single command: flatpak install --user --from https://download.mono-project.com/repo/monodevelop.flatpakref For the current 0.6.x versions of Flatpak, the user also needs to flatpak remote-add --user --from gnome https://sdk.gnome.org/gnome.flatpakrepo first this step will be automated in 0.8.0. This will download org.freedesktop.Runtime, then com.xamarin.MonoDevelop; export icons n stuff into your user environment so you can just click to start. There s some lingering experience issues due the sandbox which are on my radar. Run on external console doesn t work, for example, or open containing folder . There are people working on that (a missing DBus# feature to allow breaking out of the sandbox). But overall, I m pretty happy. I won t be entirely satisfied until I have something approximating feature equivalence to the old .debs. I don t think that will ever quite be there, since there s just no rational way to allow arbitrary /usr stuff into the sandbox, but it should provide a decent basis for a QA-able, supportable Linux MonoDevelop. And we can use this work as a starting point for any further fancy features on Linux. Gtk# app development in Flatpak MonoDevelop Editing MonoDevelop in MonoDevelop. *Inception noise*

2 June 2015

Jo Shields: mono-project.com Linux packages, June 2015 edition

The latest stable release of Mono has happened, the first bugfix update to our 4.0 branch. Here are the release highlights, and some other goodies. Stable Packages This release covers Mono 4.0.1, and MonoDevelop 5.9. As promised last time, this includes builds for RPM-based x64 systems (CentOS 7 minimum), Debian-based x64, i386, ARMv5 Soft Float, and ARMv7 Hard Float systems (Debian 7/Ubuntu 12.04 minimum). Version numbering From now on, we re going to be clearer with our version numbering scheme. Historically, we ve shipped, say, 4.0.0 to the public internally, there have been a lot of builds on this target branch, all of which get an internal revision number. 4.0.0 as-shipped was in fact 4.0.0.143 internally that was the first 4.0.0 branch release approved of for stable release. This release is the first service release on the 4.0.0 branch, numbered 4.0.1.44 it ll be officially referred to as 4.0.1 in some places, but isn t the same as 4.0.1.0, which already released on Linux/Windows a while back, to include an emergency bugfix for those platforms. That was sorta a screwup really. Using the 4-part version removes the ambiguity, rather than having 44 different 4.0.1 s in existence. And we ll aim to be clearer in future about what is alpha, what is beta, and what is final (and what is a random emergency snapshot). Alpha Linux packages Want to see things earlier? We ve now got the structure in place to provide Linux packages (and source releases) to mirror what we do on Mac. When we upload a prospective package to our Mac customers, we will automatically trigger builds for Linux too. See http://www.mono-project.com/download/alpha/ Beta Linux packages See above. s/alpha/beta/. Weekly git Master snapshots We already have packages in place for every git commit, which parallel-install Mono into /opt. This is different. Weekly (or, right now, when I manually run the requisite Jenkins job), the latest Mac build of Mono git master from our internal CI system will be copied to a public location just for you, a source tarball generated, and packages built. See here for info on making use of that.
directhex@marceline:~$ mono --version
Mono JIT compiler version 4.3.0 (Nightly 4.3.0.21/88d2b9d Thu May 28 10:54:32 UTC 2015)

21 January 2015

Jo Shields: mono-project.com Linux packages, January 2015 edition

The latest version of Mono has released (actually, it happened a week ago, but it took me a while to get all sorts of exciting new features bug-checked and shipshape). Stable packages This release covers Mono 3.12, and MonoDevelop 5.7. These are built for all the same targets as last time, with a few caveats (MonoDevelop does not include F# or ASP.NET MVC 4 support). ARM packages will be added in a few weeks time, when I get the new ARM build farm working at Xamarin s Boston office. Ahead-of-time support This probably seems silly since upstream Mono has included it for years, but Mono on Debian has never shipped with AOT d mscorlib.dll or mcs.exe, for awkward package-management reasons. Mono 3.12 fixes this, and will AOT these assemblies optimized for your computer on installation. If you can suggest any other assemblies to add to the list, we now support a simple manifest structure so any assembly can be arbitrarily AOT d on installation. Goodbye Mozroots! I am very pleased to announce that as of this release, Mono users on Linux no longer need to run mozroots to get SSL working. A new command, cert-sync , has been added to this release, which synchronizes the Mono SSL certificate store against your OS certificate store and this tool has been integrated into the packaging system for all mono-project.com packages, so it is automatically used. Just make sure the ca-certificates-mono package is installed on Debian/Ubuntu (it s always bundled on RPM-based) to take advantage! It should be installed on fresh installs by default. If you want to invoke the tool manually (e.g. you installed via make install, not packages) use
cert-sync /path/to/ca-bundle.crt
On Debian systems, that s
cert-sync /etc/ssl/certs/ca-certificates.crt
and on Red Hat derivatives it s
cert-sync /etc/pki/tls/certs/ca-bundle.crt
Your distribution might use a different path, if it s not derived from one of those. Windows installer back from the dead Thanks to help from Alex Koeplinger, I ve brought the Windows installer back from the dead. The last release on the website was for 3.2.3 (it s actually not this version at all it s complicated ), so now the Windows installer has parity with the Linux and OSX versions. The Windows installer (should!) bundles everything the Mac version does F#, PCL facades, IronWhatever, etc, along with Boehm and SGen builds of the Mono runtime done with Visual Studio 2013. An EXPERIMENTAL OH MY GOD DON T USE THIS IN PRODUCTION 64-bit installer is in the works, when I have the time to try and make a 64-build of Gtk#.

15 November 2014

Jo Shields: mono-project.com Linux packages an update

It s been pointed out to me that many people aren t aware of the current status of Linux packages on mono-project.com, so I m here s a summary: Stable packages Mono 3.10.0, MonoDevelop 5.5.0.227, NuGet 2.8.1 and F# 3.1.1.26 packages are available. Plus related bits. MonoDevelop on Linux does not currently include the F# addin (there are a lot of pieces to get in place for this to work). These are built for x86-64 CentOS 7, and should be compatible with RHEL 7, openSUSE 12.3, and derivatives. I haven t set up a SUSE 1-click install file yet, but I ll do it next week if someone reminds me. They are also built for Debian 7 on i386, x86-64, and IBM zSeries processors. The same packages ought to work on Ubuntu 12.04 and above, and any derivatives of Debian or Ubuntu. Due to ABI changes, you need to add a second compatibility extension repository for Ubuntu 12.04 or 12.10 to get anything to work, and a different compatibility extension repository for Debian derivatives with Apache 2.4 if you want the mod-mono ASP.NET Apache module (Debian 8+, Ubuntu 13.10+, and derivatives, will need this).
MonoDevelop 5.5 on Ubuntu 14.04

MonoDevelop 5.5 on Ubuntu 14.04

In general, see the install guide to get these going. Docker You may have seen Microsoft recently posting a guide to using ASP.NET 5 on Docker. Close inspection would show that this Docker image is based on our shiny new Xamarin Mono docker image, which is based on Debian 7.The full details are on Docker Hub, but the short version is docker pull mono:latest gets you an image with the very latest Mono.
directhex@desire:~$ docker pull mono:latest
Pulling repository mono
9da8fc8d2ff5: Download complete 
511136ea3c5a: Download complete 
f10807909bc5: Download complete 
f6fab3b798be: Download complete 
3c43ebb7883b: Download complete 
7a1f8e485667: Download complete 
a342319da8ea: Download complete 
3774d7ea06a6: Download complete 
directhex@desire:~$ docker run -i -t mono:latest mono --version 
Mono JIT compiler version 3.10.0 (tarball Wed Nov  5 12:50:04 UTC 2014)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	LLVM:          supported, not enabled.
	GC:            sgen
The Dockerfiles are on GitHub.

4 October 2014

Jo Shields: The unstoppable march of mobile technology

It s been more than 2 years since my last post about my smartphone. In the time after that post I upgraded my much loved Windows Phone 7 device to Windows Phone 8 (which I got rid of within months, for sucking), briefly used Firefox OS, then eventually used a Nexus 4 for at least a year. After years of terrible service provision and pricing, I decided I would not stay with my network Orange a moment longer and in getting a new contract, I would get a new phone too. So on Friday, I signed up to a new 15 per month contract with Three, including 200 minutes, unlimited data, and 25GB of data roaming in the USA and other countries (a saving of 200,000 per month versus Orange). Giffgaff is similarly competitive for data, but not roaming. No other network in the UK is competitive. For the phone, I had a shortlist of three: Apple iPhone 6, Sony Xperia Z3 Compact, and Samsung Galaxy Alpha. These are all small phones by 2014 standards, with a screen about the same size as the Nexus 4. I didn t consider any Windows Phone devices because they still haven t shipped a functional music player app on Windows Phone 8. Other more fringe OSes weren t considered, as I insist on trying out a real device in person before purchase, and no other comparable devices are testable on the high street. iPhone 6 This was the weakest offering, for me. 120 more than the Samsung, and almost 200 more than the Sony, a much lower hardware specification, physically larger, less attractive, and worst of all mandatory use of iTunes for Windows for music syncing.
iPhone6_PF_SpGry_iPhone6_PB_SpGry_iPhone6_PSL_SpGry_Homescreen-PRINT

Apple iPhone 6, press shot from apple.com, all rights reserved

The only real selling point for me would be for access to iPhone apps. And, I guess, decreased chance of mockery by co-workers. Galaxy Alpha Now on to the real choices. I ve long felt that Samsung s phones are ugly plasticy tat the Galaxy S5 is popular, well-marketed, but looks and feels cheap compared to HTC s unibody aluminium One. They ve also committed the cardinal sin of gimping the specifications of their mini (normal-sized) phones, compared to the normal (gargantuan) versions. The newly released S5 Mini is about the same spec as early 2012 s S3, the S4 Mini was mostly an S2 internally, and so on. However, whilst HTC have continued along these lines, Samsung have finally released a proper phone under 5 , in the Alpha.
Samsung Galaxy Alpha press shot from samsungmobile.com, all rights reserved

Samsung Galaxy Alpha press shot from samsungmobile.com, all rights reserved

The Alpha combines a 4.7 AMOLED screen, a plastic back, metal edges, 8-core big.LITTLE processor, and 2GB RAM. It is a PRETTY device the screen really dazzles (as is the nature of OLED). It feels like a mix of design cues from an iPhone and Samsung s own, keeping the angular feel of iPhone 4->5S rather than the curved edges on the iPhone 6. The Galaxy Alpha was one of the two devices I seriously considered. Xperia Z3 Compact The other Android device I considered was the Compact version of Sony s new Xperia Z3. Unlike other Android vendors, Sony decided that mini shouldn t mean low end when they released the Z1 compact earlier this year. The Z3 follows suit, where the same CPU and storage are found on both the big and little versions.
Sony Xperia Z3 Compact press shot from Sony Xperia Picasa album. CC BY-NC-SA 3.0

Sony Xperia Z3 Compact press shot from Sony Xperia Picasa album. CC BY-NC-SA 3.0

The Z3C has a similar construction to the Nexus 4, with glass front and back, and plastic rim. The specification is similar to the Galaxy Alpha (with a quadcore 2.5GHz Qualcomm processor about 15% faster than the big.LITTLE Exynos in the Galaxy Alpha). It differs in a few places LCD rather than AMOLED (bad); a non-removable (bad) 2600 mAh battery (good) compared to the removable 1860 mAh in the Samsung; waterproofing (good); A less hateful Android shell (Xperia on Android vs Samsung Touchwiz). For those considering a Nexus-4-replacement class device (yes, rjek, that means you), both the Samsung and the Sony are worth a look. They both have good points and bad points. In the end, both need to be tested to form a proper opinion. But for me, the chunky battery and tasteful green were enough to swing it for the Sony. So let s see where I stand in a few months time. Every phone I ve owned, I ve ended up hating it for one reason or another. My usual measure for whether a phone is good or not is how long it takes me to hit the I can t use this limit. The Nokia N900 took me about 30 minutes, the Lumia 800 lasted months. How will the Z3 Compact do? Time will tell.

1 September 2014

Jo Shields: Xamarin Apt and Yum repos now open for testing

Howdy y all Two of the main things I ve been working on since I started at Xamarin are making it easier for people to try out the latest bleeding-edge Mono, and making it easier for people on older distributions to upgrade Mono without upgrading their entire OS. Public Jenkins packages Every time anyone commits to Mono git master or MonoDevelop git master, our public Jenkins will try and turn those into packages, and add them to repositories. There s a garbage collection policy currently the 20 most recent builds are always kept, then the first build of the month for everything older than 20 builds. Because we re talking potentially broken packages here, I wrote a simple environment mangling script called mono-snapshot. When you install a Jenkins package, mono-snapshot will also be installed and configured. This allows you to have multiple Mono versions installed at once, for easy bug bisecting.
directhex@marceline:~$ mono --version
Mono JIT compiler version 3.6.0 (tarball Wed Aug 20 13:05:36 UTC 2014)
directhex@marceline:~$ . mono-snapshot mono
[mono-20140828234844]directhex@marceline:~$ mono --version
Mono JIT compiler version 3.8.1 (tarball Fri Aug 29 07:11:20 UTC 2014)
The instructions for setting up the Jenkins packages are on the new Mono web site, specifically here. The packages are built on CentOS 7 x64, Debian 7 x64, and Debian 7 i386 they should work on most newer distributions or derivatives. Stable release packages This has taken a bit longer to get working. The aim is to offer packages in our Apt/Yum repositories for every Mono release, in a timely fashion, more or less around the same time as the Mac installers are released. Info for setting this up is, again, on the new website. Like the Jenkins packages, they are designed as far as I am able to cleanly integrate with different versions of major popular distributions though there are a few instances of ABI breakage in there which I have opted to fix using one evil method rather than another evil method. Please note that these are still at preview or beta quality, and shouldn t be considered usable in major production environments until I get a bit more user feedback. The RPM packages especially are super new, and I haven t tested them exhaustively at this point I d welcome feedback. I hope to remove the testing!!! warning labels from these packages soon, but that relies on user feedback to my xamarin.com account preferably (jo.shields@)

19 July 2014

Jo Shields: Transition tracker

Friday was my last day at Collabora, the awesome Open Source consultancy in Cambridge. I d been there more than three years, and it was time for a change. As luck would have it, that change came in the form of a job offer 3 months ago from my long-time friend in Open Source, Miguel de Icaza. Monday morning, I fly out to Xamarin s main office in Boston, for just over a week of induction and face time with my new co workers, as I take on the title of Release Engineer. My job is to make sure Mono on Linux is a first-class citizen, rather than the best-effort it s been since Xamarin was formed from the ashes of the Attachmate/Novell deal. I m thrilled to work full-time on what I do already as community work including making Mono great on Debian/Ubuntu and hope to form new links with the packer communities in other major distributions. And I m delighted that Xamarin has chosen to put its money where its mouth is and fund continued Open Source development surrounding Mono. If you re in the Boston area next week or the week after, ping me via the usual methods! IMG_20140719_203043

21 March 2014

Jo Shields: Stephenson s Rocket the new name for Ye Olde SteamOSe

I ve made a new release of my curiously popular SteamOS derivative, and given it a new name: Stephenson s Rocket. Stephenson's Rocket You can download the new release from here. Release highlights:

7 February 2014

Jo Shields: Dear Fake Debian Developers, shoo.

Another post about the Valve/Collabora free games thing. This time, the bad bit people trying to scam free games from us. Before I start, I want to make one thing clear there are people who have requested keys who don t meet the criteria, but are honest and legitimate in their requests. This blogspam is not about them at all. If you re in that category, you re not being complained about. So. Some numbers. At time of writing, I ve assigned keys to 279 Debian Developers or Debian Maintainers almost 25% of the total eligible pool of about 1200. I ve denied 22 requests. Of these 10 were polite requests from people who didn t meet the conditions stated (e.g. Ubuntu developers rather than Debian). These folks weren t at all a problem for us, and I explained politely that they didn t meet the terms we had agreed at the time with Valve. No problem at all with those folks. Next, we have the chancers, 7 of them, who pretended to be eligible when they clearly weren t. For example, two people sent me signed requests pointing to their entry on the Debian New Maintainers page when challenged over the key not being in the keyring. The NM page showed that they had registered as non-uploading Debian Contributors a couple of hours previously. A few just claimed I am a DD, here is my signature when they weren t DDs at all. Those requests were also binned.
Papers, Please screenshot - denied entry application

DENIED

And then we move onto the final category. These people tried identity theft, but did a terrible job of it. There were 5 people in this category:
From: Xxxxxxxx Xxxxxx <xxxxxxxx.xxxxxx@ieee.org>
Subject: free subscription to Debian Developer
8217 A205 5E57 043B 2883 054E 7F55 BB12 A40F 862E
This is not a signature, it s a fingerprint. Amusingly, it s not the fingerprint for the person who sent my mail, but that of Neil McGovern a co-worker at Collabora. Neil assured me he knew how to sign a mail properly, so I shitcanned that entry.
From: "Xxxxx, Xxxxxxxxx" <x.xxxxx@bbw-bremen.de>
Subject: Incoming!
Hey dude,
I want to have the redemption code you are offering for the Valve Games
mQGiBEVhrscRBAD4M5+qxhZUD67PIz0JeoJ0vB0hsLE6QPV144PLjLZOzHbl4H3N
...snip...
Lz8An1TEmmq7fltTpQ+Y1oWhnE8WhVeQAKCzh3MBoNd4AIGHcVDzv0N0k+bKZQ=3D=3D
=3Du/4R
Wat? Learn to GPG!
From: Xxxxxx-Xxxx Le Xxxxxxx Xxxx <xx.xxxxxxxxx@gmail.com>
Subject: pass steam
Hey me voila
Merci beaucoup
valve
2069 1DFC C2C9 8C47 9529 84EE 0001 8C22 381A 7594
Like the first one, a fingerprint. This one is for S bastien Villemot. Don t scammers know how to GPG sign?
From: "Xxxxxxxxx Xxxxxxx" <xxxxxxxx@web.de>
Subject: thanks /DD/Steam gifts us finally something back
0x6864730DF095E5E4
Yet again, a fingerprint. This one is for Marco Nenciarini. I found this request particularly offensive due to the subject line the haughty tone from an identity thief struck me as astonishingly impertinent. Still, when will scammers learn to GPG?
From: Sven Hoexter <svenhoexter@gmail.com>
Subject: Valve produced games
I'm would like to get the valve produced games
My keyring: 0xA6DC24D9DA2493D1 Sven Hoexter <hoexter> sig:6
Easily the best scam effort, since this is the only one which both a) registered an email address under the name of a DD, and b) used a fingerprint which actually corresponds to that human. Sadly for the scammer, I m a suspicious kind of person, so my instinct was to verify the claim via IRC.
31-01-2014 16:52:48 > directhex: Hoaxter, have you started using gmail without updating your GPG key? (note: significantly more likely is someone trying to steal your identity a little to steal valve keys from collabora)
31-01-2014 16:54:51 < Hoaxter!~sh@duckpond6.stormbind.net: directhex: I do not use any Google services and did not change my key
So yeah. Nice try, scammer. I m not listing, in all of this, the mails which Neil received from people who didn t actually read his mail to d-d-a. I m also not listing a story which I ve only heard second ha actually no, this one is too good not to share. Someone went onto db.debian.org, did a search for every DD in France, and emailed every Jabber JID (since they look like email addresses) asking them to forward unwanted keys. All in all, the number of evildoers is quite low, relative to the number of legitimate claims 12 baddies to 279 legitimate keys issued. But still, this is why the whole key issuing thing has been taking me so long and why I have the convoluted signature-based validation system in place. Enjoy your keys, all 279 of you (or more by the time some of you read this). The offer has no explicit expiry on it Valve will keep issuing keys as long as there is reason to, and Collabora will continue to administer their allocation as long as they remain our clients. It s a joint gift to the community thousands of dollars worth of games from Valve, and a significant amount of my time to administer them from Collabora.

27 January 2014

Jo Shields: Dear Debian Developers, lrn2gpg

For some strange reason, I ve been receiving a lot of GPG-signed mail from Debian Developers and Maintainers lately. In response to each of these mails, I need to send a GPG-encrypted reply. The rate at which I m able to send replies has been significantly hampered by the poor state in which many DD/DM s maintain their GPG keys. Here are a few common mistakes, so you can consider correcting them. Ensure you have a UID for the email address(es) you use When I send an encrypted mail, I need to be sure that the recipient is legit. This means any decent mail client should refuse to send an encrypted message to foo@bar.com unless that email address is known somehow to GPG. In many cases, someone with a valid key for foo@bar.com would send their signed mail from foobar@gmail.com without that being a valid UID. In some cases, foo@bar.com isn t even a valid email address anymore (i.e. the bar.com mail server says no such mailbox). You should have a UID for each address you use. Signatures are per-UID You may well have a valid UID for foo@debian.org, foo@bar.com, and foobar@gmail.com but the PGP trust model doesn t automatically trust every UID as much as its peers. Each individual UID needs to be trusted (i.e. signed/uploaded) by others. What if you added billg@microsoft.com as a UID should that automatically be trusted? Clearly not. Just because you have foo@debian.org doesn t mean it s trusted for encryption without some signatures. Make sure you actually have an encrypting subkey GPG sucks, and as a result, it reports Skipping unusable pubkey when the issue is a lack of valid encrypting subkeys. If you have revoked all encrypting subkeys, or allowed them to expire, then I cannot send you encrypted mail. Exact naming matters Bob Bobbertson <foo@bar.com> and B b B bb rts n <foo@bar.com> are different people. Check your mail client s From: setting, to ensure it matches your UID. If not, fix one of them. Check your webmail plugin isn t shit Some people use third party plugins to integrate GPG into their webmail client (e.g. Hotmail or GMail). Make sure this actually works. Don t use Enigmail Enigmail is a popular plugin to integrate GPG into Mozilla Thunderbird. It doesn t work, in most cases. Almost every single BADSIG in my inbox is due to Enigmail. Thunderbird will insert spurious line wraps and escape characters into your mail after signing, resulting in invalid signatures. It s mostly okay if you never quote mail, and restrict messages to about 70 characters. I know plenty of Debian Developers don t care about GPG other than for package signing but please, for the sanity of the rest of us, take an occasional moment to care a little. I should note that the worst offenders for keys which don t just work were Developers with 1024D keys the best behaved were Maintainers of all stripes.

9 January 2014

Jo Shields: Here Ye, Here Ye

Valve Software s Steam is the number one digital game distribution service, with more than 65 million registered accounts. Steam runs on Windows, Mac OS, and Linux x86/amd64 computers, and provides access to several thousand games, at varying price points an enormous growth from less than a dozen games for Windows only about a decade ago. Valve s latest endeavour has been to bring their storefront into the living room, with a three-pronged attack: a new game controller, a programme of licensed x86 consoles to plug into the family TV, and an OS to run on the Steam Machines to tie it all together. December saw the first public release of their SteamOS which, as it turns out, is basically just a preconfigured desktop Linux. Specifically, it s a Debian Wheezy derivative, comprising a subset of 502 source packages from Wheezy; 8 of Valve s own source packages; and 51 source packages which have been either patched compared to Wheezy, backported from post-Wheezy, or both. For example, the compiler used by default is gcc-4.7 (rather than Wheezy s 4.6) and the libc version postdates Wheezy too. Valve s official instructions and installer release concerned quite a few people who had planned to try SteamOS on an older PC, by mandating a large (500GB) hard disk and a PC with UEFI firmware. Very quickly a number of instructions started appearing from people trying to fix what people felt were real issues specifically provision for BIOS-based computers, and installation from optical media. After being assured that redistribution of derivatives of SteamOS were entirely authorized (and, in fact, encouraged) I decided to produce my own variant, calling upon my own experience with debian-installer modification from past and present jobs, as well as calling upon the skills and experience of the UK Debian community as needed. The end result is Ye Olde SteamOSe. SteamOS on VMware Workstation 10 This weekend saw the third release of Ye Olde SteamOSe, a derivative designed to greatly widen the pool of computers capable of running Valve s OS. And unlike the first two releases, the public response this time has been crazy. Like, totally crazy. Combined score across several subreddits totals about 1700. 7 pages of Google search results. Hundreds of tweets. Mentions in dozens of blogs around the world. Coverage on the Linux Action Show. Unilaterally added to Softpedia s list of distributions. Almost 2000 views of a video installation walkthrough I posted on YouTube. Crazy. Sadly the idea to try and track visitors to the page didn t occur to me until long after the initial rush subsided, but 1000 visitors on Tuesday for a news story which landed on Sunday is still pretty hot in my book. It s also interesting to observe the demographics of site visitors the #1 referrer on Tuesday was Dutch PC site Tweakers.net, and StumbleUpon outranks reddit for referrals. About 66% of visitors interested in installing my Debian Wheezy derivative derivative came using Windows (20% using Linux), which suggests there s a lot of potential Linux users out amongst the Windows gamer masses. So what s the purpose of this self-congratulatory blog post? Just dick-waving? Well, there s an element of that (I m only human), but I think it might be nice to alert the audience on Planet Debian/Ubuntu, many of whom are not big gamers, to the next big thing in embedded Linux except this time a real GNU/X.org/sysVinit distribution, not some NIH thing like Android. So now you know!

15 November 2013

Jo Shields: The directhex! In an Adventure with MIPS

We ve been trying to build Mono on MIPS in Debian for a long time. Just under a decade, in fact. Mono 0.29.99.20040114-3 was the first attempt, back when the Mono source was 10 meg, not today s ~80. It never worked though. Not once. At the end of 2004, 10 upstream versions later, we gave up, and turned off mipsel as a target architecture in the package. When I became a Debian Developer in 2009, one of the first things I did was try again with Mono on various previously unsupported architectures, by running a build on the Debian porterboxes. Supposedly MIPS support was there, as evidenced by all the MIPS-related files in the source tree. But I d try it, and it d fail, and upstream would say works for me . I repeat the process every major release or so, to see what s changed. The breakthrough Spurred by the removal of two bitrotted architectures (IA64 and SPARC), I decided to try again, this time investigating more deeply into the reason for the failures, with the help of upstream developer Alex R nne Petersen. A couple of hours of IRC-driven gdb and test program disassembly later, a seemingly innocent comment flagged something in my brain: 09-07-2013 15:10:11 < alexrp!~zor@baldr.rfw.name: TIL that mul and mult are not the same thing on mips Why is this notable? Well, MIPS processors lack a whole bunch of instructions which are commonly used in assembler. MUL is one of them it s valid MIPS assembler, which is expanded to MULT/MFLO when compiling. Call it a macro, or a mnemonic, or shorthand the preferred term is pseudoinstructions . So what s the issue? 09-07-2013 15:18:27 > directhex: mono isn't trying to use a mul instruction, right? i mean, that instruction doesn't exist as far as the cpu is concerned, it's a macro the compiler does things with See where this is headed? 09-07-2013 15:23:59 > directhex: mini/mini-mips.c:#define USE_MUL 1 /* use mul instead of mult/mflo for multiply */ Argh. See, the thing about MIPS pseudoinstructions is they may be real instructions on a given CPU implementation. Strictly speaking MUL isn t a standard instruction, but a given CPU might have it anyway, to make multiplication a little faster (by using only one instruction, not two, for multiplication). In this case, the Debian MIPS infrastructure is based around ICT Loongson-2E processors which don t have that extension but the upstream Mono developers were building and testing on an extended CPU, never seeing the issue themselves. Flipping that define to 0 (and amending the instruction length setting in another file) fixed the build. Mono was running on MIPS for me for the first time ever. Digging through the history in git showed just how annoying this implementation quirk was. USE_MUL was added in late 2008 replacing a previously used #if 1 . The mult/mflo version of the code existed in the Mono source since the first time the full MIPS port was committed in 2006, but we never saw it. The breakage So, with that patched to work, I added mipsel to the Experimental build of Mono which still failed. The runtime would build fine, but the class library build would fail at random times, with random meaningless stack traces. Unrepeatable. Some kind of race condition. The build would eventually succeed if I hammered make a few times, but that s no good for the Debian build daemons. Back to square one except I had an epiphany yesterday. I have heard more than once that Loongson processors are missing a few instructions. What if one of those was being hit, intermittently? I started doing a search in places that might need to work around that kind of issue, and found this. A patch to binutils in 2009, replacing one no-op instruction with another, when /usr/bin/as is fed the -mfix-loongson2f-nop flag. Turns out NOP is another pseudoinstruction on MIPS. Well, more of an alias. The opcode 0 000000 is Shift Left Logical with 0 registers and 0 data, which is a no-op. But on all but the latest generation of Loongson-2F chips, that opcode can, under heavy load, fail causing inconsistent state in the CPU registers. The flag to as replaces sll 0,0,0 with or $at,$at,0 , which is also a no-op instruction, but doesn t trigger the failure on Loongson-2F chips (and 2E chips, although that s not stated in the documentation). As long as ALL your programs get fed through as , you don t have a problem, since it uses the replacement opcode but what if you use a JITter to generate your own opcodes? Oh fuck, it couldn t be diff --git a/mono/arch/mips/mips-codegen.h b/mono/arch/mips/mips-codegen.h
index dc4df7d..1dbd1c6 100644
--- a/mono/arch/mips/mips-codegen.h
+++ b/mono/arch/mips/mips-codegen.h
@@ -334,7 +334,7 @@ enum
/* misc and coprocessor ops */
#define mips_move(c,dest,src) mips_addu(c,dest,src,mips_zero)
#define mips_dmove(c,dest,src) mips_daddu(c,dest,src,mips_zero)
-#define mips_nop(c) mips_sll(c,0,0,0)
+#define mips_nop(c) mips_or(c,mips_at,mips_at,0)
#define mips_break(c,code) mips_emit32(c, ((code)<<6) 13)
#define mips_mfhi(c,dest) mips_format_r(c,0,0,0,dest,0,16)
#define mips_mflo(c,dest) mips_format_r(c,0,0,0,dest,0,18)
Oh yes it could! Mono was using sll 0,0,0 (the recommended no-op instruction from the MIPS instruction reference manual), causing failures in my tests, because Debian s build and test infrastructure just happens to use defective silicon. And, again, upstream were unable to reproduce a problem because they use better silicon than we do. So what now? Well, last night I uploaded mono_3.2.3+dfsg-3, which includes the above patch to force the replacement no-op instruction. It test built fine on the porterbox, and it should (when the damn experimental buildd gets around to it), just work. Finally. After just under a decade, Mono packages will be available on MIPS in Debian. And after all this time, all we had to change was 4 lines to work around 7 year old Chinese knock-off processors. The edit So, things are finally built. It turns out that despite everything, the replacement NOP opcode is not enough. If you re-read the post to the binutils list, pay close attention to: In theory this is still not enough to fully eliminate possible hangs, but the possiblity is extremely low now and hard to be hit in real code. It s a filthy lie. It s easy to hit the issue in real code: just do a from-source build of the whole Mono class library. With the replacement instruction it builds .NET 2.0, 3.0, 3.5, 4.0, and most of 4.5, before dying in the same way as before an improvement on failing early in the 2.0 build, but not enough. Thankfully, 2 out of the 5 Debian mipsel build servers are not Loongson 2 they re 11 year old Broadcom SWARM developer boards. Not fast but also not broken. Luck smiled on me, and caused my build to go to one of these Broadcom machines. As a result (experimental_mipsel-dchroot)directhex@eder:~$ mono --version head -1
Mono JIT compiler version 3.2.3 (Debian 3.2.3+dfsg-3)
It s been a long time coming.

7 May 2013

Jo Shields: Windows 8: Blood from a Stone

Ordinarily, I m a big believer that it is important to keep up to date with what every piece of software which competes with yours is doing, to remain educated on the latest concepts. Sometimes, there are concepts that get added which are definitely worth ripping off. We ve ripped off plenty of the better design choices from Windows or Mac OS, over the years, for use in the Free Desktop. So, what about Windows 8, the hip new OS on everyone s lips? Well, here s the thing I ve been using it on and off for a few months now for running legacy apps, and I can t for the life of me find anything worth stealing. Let s take the key change Windows 8 has apps built with a new design paradigm which definitely isn t called Metro. Metro apps don t really have windows in the traditional sense they re more modeled on full-screen apps from smartphones or tablets than on Windows 1.0 -> 7. Which is fine, really, if you re running Windows 8 on a tablet or touchscreen device. But what if you re not? What about the normal PC user? As Microsoft themselves ask:

How do you multitask with #windows8? The answer to that is, well, you sorta don t. Metro apps can exist in three states fullscreen, almost fullscreen, or vertical stripe. You re allowed to have two apps at most at the same time one mostly full screen, and one vertical stripe. So what happens if you try to *use* that? Let s take a fairly common thing I do watch a video and play Minesweeper. In this example, the video player is the current replacement for Windows Media Player, and ships by default. The Minesweeper game isn t installed by default, but is the only Minesweeper game in the Windows 8 app store which is gratis and by Microsoft Game Studios. Here s option A:

Unusable Windows 8 option 1 And for contrast, here s option B:

Unusable Windows 8 option 2 Which of these does a better job of letting me play Minesweeper and watch a video at the same time? Oh, here s option C, dumping Microsoft s own software, and using a third-party video player and third party Minesweeper implementation:

Windows 7 flavour It s magical almost as if picking my own window sizes makes the experience better. So, as you can see above, the old OS is still hiding there, in the form of a Windows 8 app called Desktop . Oh, sorry, didn t I say? Metro apps, and non-Metro apps, are segregated. You can run both (the Desktop app can also be almost-fullscreen or a vertical strip), but they get their own lists of apps when multitasking. Compare the list on the left with the list at the bottom:

Needs moar task lists And it s even more fun for apps like Internet Explorer, which can be started in both modes (and you often need both modes). Oh, and notice how the Ribbon interface from Office 2007 has invaded Explorer, filling the view with large buttons to do things you never want to do under normal circumstances. So, that s a short primer on why Windows 8 is terrible. Is there really nothing here worth stealing? Actually, yes, there is! After much research, I have discovered Windows 8 s shining jewel:

win8multitasking-004 The new Task Manager is lovely. I want it on my Linux systems. But that s it.

8 November 2012

Jo Shields: Evil, or why Douglas Crockford is harmful to Free Software

Yesterday I received a new serious bug report against Mono in Debian. For those not in the know, serious severity is release critical, and can trigger removal of a package from the distribution in order to make a shipping release (e.g. if Debian is in deep freeze, as is the case right now). Bug 692614 relates to a single source file, ./mcs/class/System.Web.Extensions/System.Web.Script.Serialization/JsonDeserializer.cs, which is part of the System.Web.Extensions assembly. This file was written in 2008 by an upstream Mono developer, and is based on some code from json.org specifically JSON_parser.c and JSON_checker.c json.org code carries a non-standard license. Specifically, it s MIT/X11 with an added clause: The Software shall be used for Good, not Evil. I ve had some discussions with Mono upstream, and they believe that they have resolved the matter to their satisfaction their solution will make its way into Debian soon. But in the meantime, let s talk about the clause. The Free Software Foundation s Freedom 0 reads:
A program is free software if the program s users have the four essential freedoms:
  • The freedom to run the program, for any purpose (freedom 0).
The Debian Free Software Guidelines and Open Source Initiative s Open Source Definition clause 6 reads:
No Discrimination Against Fields of Endeavor The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.
It s fairly clear that the JSON.org license clause goes against both of these, making any piece of software using that license neither Open Source nor Free Software. It is not distributable by any organization which mandates Freedom for its users not in Debian, not in Fedora, not on Google Code. Anybody who cares about their users will reject a clause like this, because it has awful chilling effects. Think it s funny? It really isn t. Who can use the code without being at risk of a lawyer knocking on their doors? Can the Catholic Church? The ACLU? Republicans? Democrats? Military contractors? Genetic engineers? Big pharma? Petrochemicals firms? Communists? Fascists? Mono developers? WikiLeaks? Without a clear definition of good and evil , people need to seriously consider whether they are safe to use the code because if the developer and users interpretations of the terms differ, there could be hell to pay. Would you want to ship a hardware device let s say you re making a smart TV and have some developer of a library send his lawyers around to tell you sorry, TV rots kids brains, it s clearly evil to get your entire distribution channel shut down by injunction? It s not an oversight, kids. Upstream are well aware of the pain this childishness causes anyone who takes software licenses seriously. They giggle about it at conferences, like tweenagers who think they ve one-upped the adults in the room. This non-Free license is intended to mock people who take licensing seriously. In fact, one could easily categorise efforts to pollute the Free Software ecosystem with fake, non-Free software as evil, which would mean all JSON.org code fails to comply with its own license, due to shipping with its license (Inception, anyone?). Before the jokers in the room claim that this kind of problem is deserved by Mono, let s take a look at all the software in Debian which Douglas Crockford endangers with his childishness. PHP? OwnCloud? jQuery? You think Debian serves its users well by pulling jQuery from the next release in order to serve the ego of a man behaving like an eleven year old? It is also interesting to note that the author of this do no evil clause works at PayPal. Read that twice, go and repair your irony meters, then come back. In conclusion: thank you so VERY much Doug Crockford for making the world of Free Software measurably worse.

29 June 2012

Jo Shields: How to build a bungalow begin with the chimney

Whilst nobody was looking, Mono became the most important framework in the world of games. And I mean that with only a smidge more than my usual levels of hyperbole. It really is fantastically important, as far as game development goes. I blogged a long time ago about its use inside EA s multi-million-selling The Sims 3. And the massive performance gains in Second Life when they switched from their in-house scripting language to Mono are well documented. But I m not talking about those, I m talking about the now, the recent stuff. First up is Unity 3D. Not to be confused with the Ubuntu user interface, Unity 3D is a proprietary game development tool which began life for Mac, and has now managed to become the single most popular game engine amongst smartphone developers. It targets a bunch of new platforms not just Android and Windows and iOS, but in the new 4.0 release, Linux too for the first time. And every Unity 3D game is a Mono game the core of Unity is a fork of Mono, which enables it to have the performance and ease of development which it does. A reasonably large percentage of the cross-platform games on Kickstarter, such as Wasteland 2 and Super Retro Squad, are cross-platform because Unity 3D makes it easy and each of those games is powered by Mono. However, I have better things to do today than gush about a proprietary tool. No, far more important are the Free ways to develop games. So today I m going to talk about MonoGame. MonoGame is an implementation of Microsoft XNA a high level framework for developing in .NET languages, originally designed to allow indie developers access to the Xbox 360. XNA covers many of the bases covered by SDL sound, graphics, input, and so on, but can go onto platforms where SDL isn t an option, such as the aforementioned Xbox 360. And with MonoGame, any game written for XNA is a game which can not only run on Microsoft s approved platforms Windows, Xbox 360, Windows Phone but also on other platforms such as Mac OS X, iPhone, iPad, Android, and Linux. Cross-platform is good. More games are good. And MonoGame enables every Xbox 360 indie developer to throw a Linux port out of the door with a relative minimum of fuss. There are 2,521 games on the Xbox Live Indie Games store (games by individuals or small groups, with no major publisher) and each of those can become a game for Debian and Ubuntu with very little work. Not to mention the games with actual publishers (some portion of the 492 games on XBLA), not to mention the mobile games written for Windows Phone, Android and iOS. All of these developers have an easy route to Linux releases, as long as the effort remains low. And with today s publishing in the Debian archive of MonoGame 2.5.1, the effort is as low as can be. Anyone who purchased the recent Humble Indie Bundle 5, and played Bastion, has seen MonoGame in action Bastion began life as an XNA game for Xbox 360, and the Linux port was made possible only by MonoGame. Bastion s 100+ industry awards should indicate the possibilities open to a creative mind with decent developer tools. MonoGame is Free Software, built on top of other Free Software, with an active upstream development team genuinely responsive to the needs of Linux distributions. It s well worth a look not just for porting, but for new projects too. And I m delighted that it will be available to all users of Debian 7.0, as well as Ubuntu 12.10.

19 June 2012

Jo Shields: Enormity

(This blog post is a more firm version of a series of tweets and forum posts I made a few weeks ago. This should also be considered a refresh of this post by Mirco Bauer a few years ago) It has been said that Mono is bloated, and that people should use lighter frameworks that don t pull in hundreds of meg . How much basis in reality is there for those comments? Well, let s do a little thought exercise. How much space is the minimum space needed for several languages to work? For this experiment, we will be looking at the required installation size on disk of several language frameworks, each time installing the bare minimum for a command-line hello world app in that language to run. For example, the Ruby result is the minimum install required to run:
puts 'Hello world'
and the Mono result is the minimum install required to run the following code, which has been compiled on a different machine using dmcs :
public class Hello1
 
   public static void Main()
    
      System.Console.WriteLine("Hello, World!");
    
 
Obviously, for dynamic languages like Ruby and Python, there is no compiler step needed. For languages like C# and Java, it is fair to compile these on a different machine as end-users of packages running these frameworks receive binaries, not source they do not need a development environment. So, on to the test environment. This is a bare bootstrapped AMD64 Debian Unstable system, as of today (i.e. debootstrap sid /tmp/badger ). Its install size is 275MB. So how much space? Ruby 1.9 Test code is as follows:
puts 'Hello world'
Install requirements are as follows:
root@dream:/# aptitude install ruby1.9.1
The following NEW packages will be installed:
  libffi5 a  libruby1.9.1 a  libyaml-0-2 a  ruby1.9.1 
0 packages upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 4703 kB of archives. After unpacking 13.1 MB will be used.
OpenJDK 7 Test code is as follows:
class HelloWorldApp  
    public static void main(String[] args)  
        System.out.println("Hello world");
     
 
Install requirements are as follows:
root@dream:/# aptitude install openjdk-7-jre-headless
The following NEW packages will be installed:
  ca-certificates a  ca-certificates-java a  dbus a  fontconfig-config a  
  icedtea-7-jre-cacao a  icedtea-7-jre-jamvm a  java-common a  
  krb5-locales a  libavahi-client3 a  libavahi-common-data a  
  libavahi-common3 a  libcap2 a  libcups2 a  libdbus-1-3 a  libexpat1 a  
  libffi5 a  libfontconfig1 a  libfreetype6 a  libglib2.0-0 a  
  libglib2.0-data a  libgssapi-krb5-2 a  libjpeg8 a  libk5crypto3 a  
  libkeyutils1 a  libkrb5-3 a  libkrb5support0 a  liblcms2-2 a  libnspr4 a  
  libnss3 libnss3-1d a  libpcre3 a  libpcsclite1 a  libsystemd-login0 a  
  libxml2 a  openjdk-7-jre-headless openjdk-7-jre-lib a  openssl a  
  sgml-base a  shared-mime-info a  ttf-dejavu-core a  tzdata-java a  ucf a  
  xml-core a  
0 packages upgraded, 43 newly installed, 0 to remove and 0 not upgraded.
Need to get 51.5 MB of archives. After unpacking 137 MB will be used.
Python 3.2 Test code is as follows:
print ("Hello world")
Install requirements are as follows:
root@dream:/# aptitude install python3.2-minimal
The following NEW packages will be installed:
  file a  libexpat1 a  libffi5 a  libmagic1 a  mime-support a  python3.2 a  python3.2-minimal 
0 packages upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 4853 kB of archives. After unpacking 17.7 MB will be used.
Mono 2.10 (C#) Test code is as follows:
public class Hello
 
   public static void Main()
    
      System.Console.WriteLine("Hello world");
    
 
Install requirements are as follows:
root@dream:/# aptitude install mono-runtime
The following NEW packages will be installed:
  binfmt-support a  cli-common a  libmono-corlib4.0-cil a  libmono-i18n-west4.0-cil a  libmono-i18n4.0-cil a  
  libmono-security4.0-cil a  libmono-system-configuration4.0-cil a  libmono-system-security4.0-cil a  
  libmono-system-xml4.0-cil a  libmono-system4.0-cil a  mono-4.0-gac a  mono-gac a  mono-runtime 
0 packages upgraded, 13 newly installed, 0 to remove and 0 not upgraded.
Need to get 4383 kB of archives. After unpacking 11.5 MB will be used.
Of course, every time I try to make the argument that Mono is much leaner than what people propose as an alternative, the anti-Mono brigade go on the defensive and insist that nobody in their right minds uses Python or Ruby either everything ever should be written in Qt. So, hypothetically Qt 4.8 (C++) Test code is as follows:
#include <QTextStream>
int main(int argc, char **argv)
 
   QTextStream s(stdout);
   s << "Hello, world!\n";
   return 0;
 
Install requirements are as follows:
root@dream:/# aptitude install libqtcore4
The following NEW packages will be installed:
  libffi5 a  libglib2.0-0 a  libglib2.0-data a  libpcre3 a  libqtcore4 
  libxml2 a  sgml-base a  shared-mime-info a  xml-core a  
0 packages upgraded, 9 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.3 MB of archives. After unpacking 27.8 MB will be used.
Presented without comment for your consideration. Edit: There are complaints that these numbers are misleading, because they show default package manager behaviour, rather than what happens when passing extra flags or config file settings to Apt. When disabling installation of Recommends: packages, the numbers change as follows:

10 May 2012

Jo Shields: Sleeping with the enemy: my life with Windows Phone

In my last blog post about smartphones, I urged the universe at large to help maintain a variety of ecosystems, to help foster competition and originality amongst vendors and the same day I hit publish, WebOS was killed. Apparently the universe hates me. Since then, a few things have changed. My main phone since the day of its release was the HP Pre 3, running WebOS and whilst I still have a soft spot for the OS, the Pre 3 was simply too buggy for me to use full time. The main issue is that I use my phone as an MP3 player in the car but the Pre 3 would pause playback at the end of a track every half dozen tracks or so making it impossible to drive the 85 miles to work without needing to root around in the armrest and poke a touchscreen. Not something I really want to do whilst moving and ultimately too big a papercut to deal with. So, come the new year, I moved on to my next device, a Nokia N9 running MeeGo Harmattan. Ultimately, this was an even bigger failure for me than the Pre 3 was, and I lasted maybe two weeks with it before giving up and going back to the HP. Beyond massive usability errors in the software (especially the braindead unkillable pop-up demanding Internet access, even when none is available), the worst for me was how it handled the MP3 player task. My usual way of working is to have the phone hooked up to the stereo with a 3.5mm jack, and the car switches to headset Bluetooth profile to handle calls this is pretty common on cars too old to support A2DP profile (Stereo music-capable headphones). WebOS and Android are fine with this but not the N9. The N9, instead, will output all audio through the last connected audio device, regardless of how much that might not be helpful. Get in car, start music playing, plug in cable, start engine and it plays audio for about three seconds before the Bluetooth connects, and it switches to outputting music via the Headset bluetooth profile (not something that my car can do). Unplug and replug the cable, and music works but incoming calls are silent until I disconnect the 3.5mm jack, as it outputs the headset audio through the headphone socket. I just couldn t deal with this big a step back from WebOS as far as my workflow goes, and gave up. So, where next? Well, a funny thing happened a co-worker with generally very good instincts regarding consumer electronics usability told me that his housemate had just bought a Nokia Lumia 800 Windows phone (the WP7-based cousin to the N9) and loved it. Enough that said co-worker was considering getting one himself. This was a very strange thing to hear, especially from an iPhone owner, about a Microsoft product. I d been generally interested in WP7 on an academic level for a while, but to hear that degree of praise of the actual product was interesting. Also interesting, and roughly simultaneous, was seeing Sajid Anwar s reverse engineering of the proprietary Zune file transfer protocol go from theory into an actual set of libmtp patches. So if the capability to use Banshee to transfer music on is here or near and it can t be as braindead as Harmattan when it comes to headphone/bluetooth behaviour, then why not jump ship and squeeze a handset out of Orange? About a week after my co-worker replaced his iPhone with a Lumia 800, I bought one too. So where to begin? Well, I ll begin at the start: WP7 is a joy to use. It really just is. It s the first mobile OS to try something radically different in the UI department for years. Everyone else these days (especially Android) builds iPhone rip-offs to varying degrees, and even the iPhone interface has a lot in common with the old old OLD interfaces found on the dumb Nokia phones of the 1990s. WP7 has an interface which provides just the right level of passively visible information and interactivity, and manages to do it with an elegance that no Android home screen filled with widgets will ever manage. The uncluttered screens are easy to read, and the Metro usability paradigms are trivial to pick up and learn. Without a doubt I d recommend WP7 to friends and family from a usability perspective, and the Microsoft engineers and designers responsible for cooking up the WP7 interface are worthy of praise. And I m not the only one saying this Apple co-founder Steve Woz Wozniak recently came out with a similar line. That s the good. There s also some bad, make no mistake. I m going to cover all the reasons WP7 sucks over several paragraphs. But overall, a smartphone is a device which I expect to suck the question is how bad the suck is, and whether it gets in the way of me using the device for what I need at the time. Moreso than MeeGo, moreso than Android, and even WebOS (and I m still a big WebOS fan), WP7 has more good points than bad points. But there s still some room for improvement, and some room for caution and since I know there are a few Microsoft folks following me on Twitter, I m going to go over my prescription for continued platform success. Oh, one more thing before I start: I know WP7 isn t Free Software. As an end user, I really don t care about that. I just want something that works something I didn t get from WebOS and Harmattan, both of which are primarily Free Software stacks. I m not saying there s a causal relationship there, or that a mobile OS can t be both Free Software and good just that as an end user, my favourite platform right now is non-Free. Take from that whatever you like. It s also vitally important, as Free Software folks, never to lose sight of what the other players in the market are up to. If you can t objectively assess why people are using a proprietary option by using it & recognising its good points (i.e. what to steal & what to improve) then you can t hope to win over users. So. WP7 s downsides in detail. In-place updates. Seriously guys, even Apple can manage this now. Why can t Windows Phone? I understand that making backups is smart and all updates come with a mandatory backup but I really shouldn t be tied to a PC to update a post-PC device. Also, those backups are useless, since they cannot be restored onto replacement devices in the case of failure or theft, so fix that too. Update all the things. An iPhone sold in June 2009 still has access to the latest iOS releases. Android phones are notorious for shipping with an outdated version of the OS, then getting at most one major update over the phone s lifetime (usually the device is abandoned by its manufacturer within months of release). Which camp does Microsoft want to align with, there? Every Windows phone 7 device released should receive Windows Phone 8, even with some features disabled. Anything less is punishing every existing customer, in the hope that you ll attract new ones not a winning strategy for a fringe platform whose biggest evangelists are its users. Fix IMAP. IMAP isn t hard. Yet WP7 never seems to work properly with a subset of my mail, never showing the message body & just saying Downloading forever. Fix it. Bing sucks. Bing s search results are terrible. Either do something to make them bearable, or allow me to pick which search engine I get when I hit the search button. A Google live tile isn t the same thing. Make killing apps easier. I know you stole the WebOS card view for multitasking (hold the back button) please also steal the WebOS ability to close apps. I don t want to have to go into an app and bash back repeatedly until it quits. This is particularly annoying for Internet Explorer. Make reinstalling apps easier. If I want to install every app I previously had installed on a new device, without restoring a backup, this should be easy. There are third party apps which try to plug this gap. Find a way to support copyleft. I d like to port a few C# apps to WP7, but because they re LGPL, I can t. The code s copyright holders would have no issue with their code being on WP7, as long as end users have a mechanism to replace the libraries, so why not find a way to allow this? e.g. when compiling an app, let me mark a library as user-replaceable , then allow for some mechanism where an end user can replace those assemblies with their own version. Let me use multiple Google calendars. WP7 only lets me add/see appointments on my default Google calendar. I want to add/see things on my wife s calendar, which is shared with me. WebOS can do this. MTP-Z is the devil. I do not need or want encrypted end to end communication between my PC and my camera device, to transfer a photo off. I do not need or want encrypted end to end communication between my PC and my MP3 player to transfer a photo on. Let s be honest, the only reason for MTP-Z is to enforce DRM on Zune-rented music tracks and honestly, there s no good reason to require MTP-Z for *all* communications if all you want to do is protect one folder or file extension. Now, since MTP-Z theoretically forces me to use Zune for many tasks better handled by other apps, now I get to write multiple criticisms of Zune s desktop app and as long as MTP-Z is enforced, every Zune failing is a Windows Phone failing too. Zune: Support Windows codec infrastructure, and transcode where needed. Windows Media Player can play Ogg Vorbis files. No, not out of the box, but if one installs the required codecs. Zune should support the same files as WMP if you want to ensure people don t try to copy files to a portable device which are not supported on that device, then you should have an API in place to allow for pluggable seamless transcoding of files as required Banshee allows me to do this (e.g. to copy files I have as .flac to devices which do not support it). Zune: Search my tracks, not the web. Zune s searching is terrible it doesn t do as-you-type searches, and when I hit enter, matches from my collection are given a tiny little space compared to matches from the Zune music store. Let me easily pick the track I feel like listening to, don t make it a chore Zune: Let s solve metadata together. I absolutely love how nicely the Zune app on desktop and on phone shifts as appropriate to the currently playing artist (e.g. changing the lock screen to an image of the artist in question). However, Zune doesn t make it obvious how to set an album s metadata to support this, and it s particularly frustrating when it s a minor difference of spelling causing a track not to get the nice treatment e.g. UNKLE versus U.N.K.L.E. . Either start making heavy use of audio fingerprinting services like MusicBrainz to fill in metadata, or allow me to search for fully supported artists when filling in track metadata Zune: Random playlists are useless on devices. I like smart playlists. In Banshee, I have one to pick 12GB of random tracks, which I can sync to my phone. I can t do this with Zune. If I try to just sync all my random music to my phone, it errors out due to lack of space. If I have a random playlist, the random selection changes multiple times during a sync resetting the sync, wiping out half the tracks that were transferred on, and starting again. As a result, the sync goes on for literally hours, never ending up with more than a gig or so of tracks on the phone. Random playlists should be freezable, so I can transfer them to my device in peace, then get a new random selection when I want. So, that s my list of miserable failure and it s still a less painful list than any other mobile OS I ve used. Perhaps one day Android will approach being usable, perhaps Blackberry s BBX will actually appeal to human beings rather than corporate IT managers, and perhaps Mozilla s delightfully named Boot to Gecko will get some traction. Who knows. All I know is, My Lumia 800 is the best phone I think I ve ever owned, and it s important for anyone working in the mobile space to understand why.

Next.