Keith and I have discovered a change in the behavior of the protection
circuits integrated in the LiPo batteries we sell for use with
Altus Metrum products that poses a risk for
our customers. This post is meant to document what we now know, communicate
changes we're planning as a result, and explain what we think flyers of our
existing electronics and batteries may want to do to maximize their chances
of successful flights.
Background
Choosing batteries and designing pyro circuits for high power rocketry avionics
involves a variety of trade-offs. Reliability is the highest concern, both
because nobody wants to lose an airframe due to a failed pyro event, and
also because airframes recovering anomalously have safety implications. But
we also care about other factors including cost and weight, and usually
want to minimize the complexity of both the electronics and the overall
installation.
The objective of a pyro circuit is to dump enough energy rapidly into an
electric match to cause it to catch fire. We need batteries both to power
the electronics that decide when to fire the charge, and to provide the
energy that actually ignites the match.
The two most common battery types seen in the rocketry hobby are alkaline
cells, often the nominal 9V rectangular variety, and rechargeable batteries
based on Lithium Polymer (LiPo) chemistry. LiPo cells are 3.7 volts per
cell nominal voltage, are very light, and have a high energy density.
LiPo capacity is measured in units of current times time, so an 850mAh
cell should be able to deliver 850 milliamps for an hour. The battery
industry also uses something called a "C rate" to describe how fast the
battery can be usefully discharged, wich is a multiplier relative to the
battery capacity. So a battery with 850mAh capacity and a "2C" rating can
deliver current at a sustained rate of 1700mA until discharged, while the
same capacity at a "5C" rating can deliver 4250mA.
By comparison, most 9V alkaline batteries are actually composed of 6
individual 1.5V cells enclosed in a wrapper. It's hard to get hard numbers
for capacity and discharge rate, since in an alkaline cell the two are not
independent, and the discharge rate is related to the volume of each
individual cell. The
data sheet for an Energizer 522
shows just over 600mAh at a 25mA discharge rate, dropping to about 300mAh
at a 500mA discharge rate.
Importantly for use in pyro circuits, LiPo cells have a
very low source
impedance, which means they can source immense amounts of current. It's not
unusual for cells in the 1000mAh range to have ratings in excess of
30C! Because this rapid discharge ability can pose a risk of fire, it's common
for LiPo cells to come with a "protection board" integrated into the battery
assembly that is designed to limit the current to some rate such as 2C
continuous duty.
In large airframes, or projects that involve staging, air-starts, or other
complex pyro event sequences, the most reliable approach will always be
to use separate batteries for the control electronics and the source of
pyro firing power. In the limit, having separate pyro batteries for each
pyro charge with the control electronics only providing the switching to
connect the batteries to the charges could even make sense. But for most
airframes, this is overkill, and the increases in mass, volume, and wiring
complexity just don't make sense.
The challenge, then, is how to design electronics that will robustly initiate
pyro events without negatively affecting the rest of the electronics when
operating from a single shared battery.
Altus Metrum Pyro Circuits
The very first prototypes of
TeleMetrum
were designed to use a single-cell LiPo battery, and had an on-board 100mA
charging circuit. Because we needed 5 volts to power the accelerometer, we
had a small switching regulator that up-converted the LiPo voltage, and we
used some of that regulator's output to charge a 1000uF capacitor. The pyro
circuit used Fairchild FDN335N N-channel MOSFET switches in a low-side
switching configuration to dump the energy stored in the capacitor through
an attached ematch. Those FETs had an on resistance of under a tenth of an
ohm in our operating conditions. The circuit worked very reliably, but the
1000uF electrolytic cap was huge and we struggled with the mechanics of such
a large part hanging off the board...
It turns out that 3.7 volts is way more than enough to fire a typical
low-current e-match or equivalents like the
Quest Q2G2
igniters. In fact,
bench testing with a good digital oscilloscope showed that a typical e-match
with resistance of 1.3-1.8 ohms will fire in approximately 13 microseconds
when hit with the nominal 3.7 volts from a LiPo.
So, starting with our v0.2 boards, we switched to using the LiPo battery
voltage directly to fire the pyro charges, eliminating the clunky electrolytic
capacitor entirely. We also switched to the Fairchild FDS9926A dual N-channel
MOSFET whose specs are better in all regards for our application. The on
resistance is down around 40 milli-ohms in our circuit, such that the current
ratings are much higher (FET current limits are primarily driven by how much
heat is generated due to current flowing through the channel's on resistance).
Because using the LiPo voltage directly means we're effectively temporarily
putting a very low resistance across the battery during the pyro events, the
input voltage to the voltage regulator gets pulled down. To ensure the
processor could "ride through" these events, we added a 100uF surface mount
bulk capacitor on the 3.3 volt regulated voltage rail, which bench testing
demonstrated was more than sufficient to maintain processor operation through
pyro events. And that is essentially the same pyro circuit on all the boards,
both
TeleMetrum and
TeleMini, that we have shipped to date.
What's Changed
The LiPo batteries we source and sell with our electronics come with a
protection board and cable terminated in a 2-pin, 2mm "JST" connector. The
specs on the protection board have always been "2C continuous", but we
observed the
ability to source much higher currents for short periods such as the 13
micro-seconds or so required to fire an e-match. Thus these protection
circuits seemed just fine .. we could fire e-matches with a burst of current
but were protected against short circuits in the wiring or our boards by
the 2C continuous limit.
Unfortunately, the most recent batch of batteries we sourced seem to have a
much "twitchier" protection circuit. We can draw more than 2C for short
bursts, but not as much as with prior boards, and not for as long an interval.
With a 1 ohm power resistor on the pyro terminals of one of our boards, we
get about 9 milliseconds of power before the protection circuit cuts in and
shuts the battery down. The power stays down until
all load is removed,
which at least means the board is turned off and back on again, and in some
cases could even mean the battery is unplugged and re-plugged since we draw
trace current to keep the GPS memory alive even when the power is turned off,
and at least some of the new batteries see that as enough to keep the power
turned off after an over-current event.
For many e-matches, this isn't an issue, since 9 milliseconds is
way longer
than the 13 microseconds needed to fire the charge. Unfortunately, we've
discovered that many of the e-matches bought and used in the rocketry hobby
are actually made for use in the fireworks industry, where it is desireable
to retain continuity
after firing so that series connections of e-matches all can fire even if
some fire faster than others! This means that while the resistance goes
up some after firing,
sometimes the drain on the battery remains sufficient
to cause the protection circuit to kick in even after the pyro charge has
fired.
What We're Doing About It
If we remove the protection circuit from the LiPo (or jumper around it), all
existing Altus Metrum products will operate successfully with pyro charges
thave have an effective resistance of as low as 1 ohm. That's lower than
any e-match or Q2G2 we've ever seen, so in effect what this means is that
if you have an existing Altus Metrum flight computer, and you remove the LiPo
protection circuit, you're good to go. This does not really make things any
more "dangerous", since our battery chargers are all current limited and our
discharge patterns will never cause heating of the battery. Frankly, in a
rocket, the most likely way to cause a problem with a LiPo is by smashing or
puncturing the actual battery during bench work or during a crash... and
those cause the same problems with or without a protection board present.
In the future, we will ship batteries that have either a much higher C rating
on the protection circuit, or have no protection circuit at all.
The number of problems reported by actual customers that we think should be
attributed to LiPo protection circuit boards is
very low, and we
suspect most of our customers who are happily flying their boards can continue
to do so. Ground testing where you fire pyro charges (or at least e-matches)
using RF to issue the commands (not USB, since the LiPo charger is running
any time USB is connected!) will confirm whether there's a problem. If the
board resets (does startup beeps) after a pyro event, or shuts down completely
(no LED activity), then you have a problem. If the matches light and the
board keeps running, you're good to go.
However,
any Altus Metrum customer with LiPo batteries sourced from us or
our distributors who is worried about this problem (even if you haven't seen
problems in ground testing or previous flights), and who doesn't want to try
soldering on the battery circuit board yourself, is welcome to contact us
about removing the protection circuit for you. We won't charge anything
other than shipping.
To take advantage of this offer, just send email to
fixmybattery@altusmetrum.org telling us how many of which capacity batteries
you have that you'd like updated, and we'll respond with an RMA number and
shipping details.
Going Even Further
As previously indicated, with the LiPo protection circuits removed, all of
our current products will work reliably with at least 1 ohm across the pyro
terminals. That should cover all real-world flying conditions just fine,
but we're not satisfied yet.
We've designed a new pyro control circuit that transfers the maximum possible
energy to the load regardless of battery voltage without ever allowing the
voltage to the processor to droop at all. We're testing this new circuit in
various prototypes now, and if it pans out it will probably show up first in
MegaMetrum and then trickle down to
TeleMetrum and
TeleMini as those products
are updated later this year. The new pyro circuit tolerates 0 ohms (dead
shorts) on the pyro terminals for as long as the battery can provide
current, which is as good as it gets. We think the circuit is clever enough
that we'll probably write more about it once we're finished validating it.
DebConf4
This tshirt is 16 years old and from DebConf4. Again, I should probably wash it at 60 celcius for once...
Finally, DebConf4 and more importantly FISL, which was really big (5000 people?) and after that,
the 
To be clear: for Stretch we recommend that reproducible builds are done in the same build path as the "original" build.
Finally, and just for our future references, when we enabled build path variation on Saturday, August 20th 2016, the numbers for unstable were:
What happened in the 
Long over due post
It s been a long time I haven t written here. And lots of things happened in the OpenStack planet. As a full time employee with the mission to package OpenStack in Debian, it feels like it is kind of my duty to tell everyone about what s going on.
Liberty is out, uploaded to Debian
Since my last post, OpenStack Liberty, the 12th release of OpenStack, was released. In late August, Debian was the first platform which included Liberty, as I proudly outran both RDO and Canonical. So I was the first to make the announcement that Liberty passed most of the Tempest tests with the beta 3 release of Liberty (the Beta 3 is always kind of the first pre-release, as this is when feature freeze happens). Though I never made the announcement that Liberty final was uploaded to Debian, it was done just a single day after the official release.
Before the release, all of Liberty was living in Debian Experimental. Following the upload of the final packages in Experimental, I uploaded all of it to Sid. This represented 102 packages, so it took me about 3 days to do it all.
Tokyo summit
I had the pleasure to be in Tokyo for the Mitaka summit. I was very pleased with the cross-project sessions during the first day. Lots of these sessions were very interesting for me. In fact, I wish I could have attended them all, but of course, I can t split myself in 3 to follow all of the 3 tracks.
Then there was the 2 sessions about Debian packaging on upstream OpenStack infra. The goal is to setup the OpenStack upstream infrastructure to allow packaging using Gerrit, and gating each git commit using the usual tools: building the package and checking there s no FTBFS, running checks like lintian, piuparts and such. I knew already the overview of what was needed to make it happen. What I didn t know was the implementation details, which I hoped we could figure out during the 1:30 slot. Unfortunately, this didn t happen as I expected, and we discussed more general things than I wished. I was told that just reading the docs from the infra team was enough, but in reality, it was not. What currently needs to happen is building a Debian based image, using disk-image-builder, which would include the usual tools to build packages: git-buildpackage, sbuild, and so on. I m still stuck at this stage, which would be trivial if I knew a bit more about how upstream infra works, since I already know how to setup all of that on a local machine.
I ve been told by Monty Tailor that he would help. Though he s always a very busy man, and to date, he still didn t find enough time to give me a hand. Nobody replied to my request for help in the openstack-dev list either. Hopefully, with a bit of insistence, someone will help.
Keystone migration to Testing (aka: Debian Stretch) blocked by python-repoze.who
Absolutely all of OpenStack Liberty, as of today, has migrated to Stretch. All? No. Keystone is blocked by a chain of dependency. Keystone depends on python-pysaml2, itself blocked by python-repoze.who. The later, I upgraded it to version 2.2. Though python-repoze.what depends on version <= 1.9, which is blocking the migration. Since python-repoze.who-plugins, python-repoze.what and python-repoze.what-plugins aren t used by any package anymore, I asked for them to be removed from Debian (see #805407). Until this request is processed by the FTP masters, Keystone, which is the most important piece of OpenStack (it does the authentication) will be blocked for migration to Stretch.
New OpenStack server packages available
On my presentation at Debconf 15, I quickly introduced new services which were released upstream. I have since packaged them all:
This year, on top of the many excellent contributed talks, BoFs, and other
events always part of DebConf (some of which have already been announced) we
are excited to have confirmed the following keynote speakers.
During the Open Weekend (Saturday, August 15th and Sunday, August 16th), we
will have keynotes delivered by:
The radio silence here on my blog has been not from lack of activity, but the inverse. Linux.conf.au chewed up the few remaining spare cycles I have had recently (after family and work), but not from organising the conference (been there, got the T-Shirt and the bag). So, let s do a run through of what has happened
The process has been ongoing for 
There are actually lots of people I admire in Debian. I tried to name a few
without thinking, but I wasn't thinking so I lost count as soon as I ran out of
fingers. I know however that many enjoy to stay out of the spotlight and keep
their fun focused on a few specific things. I am one of those myself.
Oh dear,