Keith and I have discovered a change in the behavior of the protection
circuits integrated in the LiPo batteries we sell for use with
Altus Metrum products that poses a risk for
our customers. This post is meant to document what we now know, communicate
changes we're planning as a result, and explain what we think flyers of our
existing electronics and batteries may want to do to maximize their chances
of successful flights.
Background
Choosing batteries and designing pyro circuits for high power rocketry avionics
involves a variety of trade-offs. Reliability is the highest concern, both
because nobody wants to lose an airframe due to a failed pyro event, and
also because airframes recovering anomalously have safety implications. But
we also care about other factors including cost and weight, and usually
want to minimize the complexity of both the electronics and the overall
installation.
The objective of a pyro circuit is to dump enough energy rapidly into an
electric match to cause it to catch fire. We need batteries both to power
the electronics that decide when to fire the charge, and to provide the
energy that actually ignites the match.
The two most common battery types seen in the rocketry hobby are alkaline
cells, often the nominal 9V rectangular variety, and rechargeable batteries
based on Lithium Polymer (LiPo) chemistry. LiPo cells are 3.7 volts per
cell nominal voltage, are very light, and have a high energy density.
LiPo capacity is measured in units of current times time, so an 850mAh
cell should be able to deliver 850 milliamps for an hour. The battery
industry also uses something called a "C rate" to describe how fast the
battery can be usefully discharged, wich is a multiplier relative to the
battery capacity. So a battery with 850mAh capacity and a "2C" rating can
deliver current at a sustained rate of 1700mA until discharged, while the
same capacity at a "5C" rating can deliver 4250mA.
By comparison, most 9V alkaline batteries are actually composed of 6
individual 1.5V cells enclosed in a wrapper. It's hard to get hard numbers
for capacity and discharge rate, since in an alkaline cell the two are not
independent, and the discharge rate is related to the volume of each
individual cell. The
data sheet for an Energizer 522
shows just over 600mAh at a 25mA discharge rate, dropping to about 300mAh
at a 500mA discharge rate.
Importantly for use in pyro circuits, LiPo cells have a
very low source
impedance, which means they can source immense amounts of current. It's not
unusual for cells in the 1000mAh range to have ratings in excess of
30C! Because this rapid discharge ability can pose a risk of fire, it's common
for LiPo cells to come with a "protection board" integrated into the battery
assembly that is designed to limit the current to some rate such as 2C
continuous duty.
In large airframes, or projects that involve staging, air-starts, or other
complex pyro event sequences, the most reliable approach will always be
to use separate batteries for the control electronics and the source of
pyro firing power. In the limit, having separate pyro batteries for each
pyro charge with the control electronics only providing the switching to
connect the batteries to the charges could even make sense. But for most
airframes, this is overkill, and the increases in mass, volume, and wiring
complexity just don't make sense.
The challenge, then, is how to design electronics that will robustly initiate
pyro events without negatively affecting the rest of the electronics when
operating from a single shared battery.
Altus Metrum Pyro Circuits
The very first prototypes of
TeleMetrum
were designed to use a single-cell LiPo battery, and had an on-board 100mA
charging circuit. Because we needed 5 volts to power the accelerometer, we
had a small switching regulator that up-converted the LiPo voltage, and we
used some of that regulator's output to charge a 1000uF capacitor. The pyro
circuit used Fairchild FDN335N N-channel MOSFET switches in a low-side
switching configuration to dump the energy stored in the capacitor through
an attached ematch. Those FETs had an on resistance of under a tenth of an
ohm in our operating conditions. The circuit worked very reliably, but the
1000uF electrolytic cap was huge and we struggled with the mechanics of such
a large part hanging off the board...
It turns out that 3.7 volts is way more than enough to fire a typical
low-current e-match or equivalents like the
Quest Q2G2
igniters. In fact,
bench testing with a good digital oscilloscope showed that a typical e-match
with resistance of 1.3-1.8 ohms will fire in approximately 13 microseconds
when hit with the nominal 3.7 volts from a LiPo.
So, starting with our v0.2 boards, we switched to using the LiPo battery
voltage directly to fire the pyro charges, eliminating the clunky electrolytic
capacitor entirely. We also switched to the Fairchild FDS9926A dual N-channel
MOSFET whose specs are better in all regards for our application. The on
resistance is down around 40 milli-ohms in our circuit, such that the current
ratings are much higher (FET current limits are primarily driven by how much
heat is generated due to current flowing through the channel's on resistance).
Because using the LiPo voltage directly means we're effectively temporarily
putting a very low resistance across the battery during the pyro events, the
input voltage to the voltage regulator gets pulled down. To ensure the
processor could "ride through" these events, we added a 100uF surface mount
bulk capacitor on the 3.3 volt regulated voltage rail, which bench testing
demonstrated was more than sufficient to maintain processor operation through
pyro events. And that is essentially the same pyro circuit on all the boards,
both
TeleMetrum and
TeleMini, that we have shipped to date.
What's Changed
The LiPo batteries we source and sell with our electronics come with a
protection board and cable terminated in a 2-pin, 2mm "JST" connector. The
specs on the protection board have always been "2C continuous", but we
observed the
ability to source much higher currents for short periods such as the 13
micro-seconds or so required to fire an e-match. Thus these protection
circuits seemed just fine .. we could fire e-matches with a burst of current
but were protected against short circuits in the wiring or our boards by
the 2C continuous limit.
Unfortunately, the most recent batch of batteries we sourced seem to have a
much "twitchier" protection circuit. We can draw more than 2C for short
bursts, but not as much as with prior boards, and not for as long an interval.
With a 1 ohm power resistor on the pyro terminals of one of our boards, we
get about 9 milliseconds of power before the protection circuit cuts in and
shuts the battery down. The power stays down until
all load is removed,
which at least means the board is turned off and back on again, and in some
cases could even mean the battery is unplugged and re-plugged since we draw
trace current to keep the GPS memory alive even when the power is turned off,
and at least some of the new batteries see that as enough to keep the power
turned off after an over-current event.
For many e-matches, this isn't an issue, since 9 milliseconds is
way longer
than the 13 microseconds needed to fire the charge. Unfortunately, we've
discovered that many of the e-matches bought and used in the rocketry hobby
are actually made for use in the fireworks industry, where it is desireable
to retain continuity
after firing so that series connections of e-matches all can fire even if
some fire faster than others! This means that while the resistance goes
up some after firing,
sometimes the drain on the battery remains sufficient
to cause the protection circuit to kick in even after the pyro charge has
fired.
What We're Doing About It
If we remove the protection circuit from the LiPo (or jumper around it), all
existing Altus Metrum products will operate successfully with pyro charges
thave have an effective resistance of as low as 1 ohm. That's lower than
any e-match or Q2G2 we've ever seen, so in effect what this means is that
if you have an existing Altus Metrum flight computer, and you remove the LiPo
protection circuit, you're good to go. This does not really make things any
more "dangerous", since our battery chargers are all current limited and our
discharge patterns will never cause heating of the battery. Frankly, in a
rocket, the most likely way to cause a problem with a LiPo is by smashing or
puncturing the actual battery during bench work or during a crash... and
those cause the same problems with or without a protection board present.
In the future, we will ship batteries that have either a much higher C rating
on the protection circuit, or have no protection circuit at all.
The number of problems reported by actual customers that we think should be
attributed to LiPo protection circuit boards is
very low, and we
suspect most of our customers who are happily flying their boards can continue
to do so. Ground testing where you fire pyro charges (or at least e-matches)
using RF to issue the commands (not USB, since the LiPo charger is running
any time USB is connected!) will confirm whether there's a problem. If the
board resets (does startup beeps) after a pyro event, or shuts down completely
(no LED activity), then you have a problem. If the matches light and the
board keeps running, you're good to go.
However,
any Altus Metrum customer with LiPo batteries sourced from us or
our distributors who is worried about this problem (even if you haven't seen
problems in ground testing or previous flights), and who doesn't want to try
soldering on the battery circuit board yourself, is welcome to contact us
about removing the protection circuit for you. We won't charge anything
other than shipping.
To take advantage of this offer, just send email to
fixmybattery@altusmetrum.org telling us how many of which capacity batteries
you have that you'd like updated, and we'll respond with an RMA number and
shipping details.
Going Even Further
As previously indicated, with the LiPo protection circuits removed, all of
our current products will work reliably with at least 1 ohm across the pyro
terminals. That should cover all real-world flying conditions just fine,
but we're not satisfied yet.
We've designed a new pyro control circuit that transfers the maximum possible
energy to the load regardless of battery voltage without ever allowing the
voltage to the processor to droop at all. We're testing this new circuit in
various prototypes now, and if it pans out it will probably show up first in
MegaMetrum and then trickle down to
TeleMetrum and
TeleMini as those products
are updated later this year. The new pyro circuit tolerates 0 ohms (dead
shorts) on the pyro terminals for as long as the battery can provide
current, which is as good as it gets. We think the circuit is clever enough
that we'll probably write more about it once we're finished validating it.
Last week was the
OpenStack Design Summit
in Portland, OR where we, developers, discussed and designed the new
OpenStack release (Havana) coming up.
The summit has been wonderful. It was my first OpenStack design summit --
even more as a PTL -- and bumping into various people I've never met so far
and worked with online only was a real pleasure!
<figure>
<figcaption>Me and Nick ready to talk about Ceilometer new features.</figcaption>
</figure>
Nick Barcet from
eNovance, our dear previous Ceilometer PTL, and
myself, talked about Ceilometer and presented the work that bas been done
for Grizzly, with some previews of what we'll like to see done for its
Havana release. You can take a look at
the slides
if you're curious.
Design sessions
Ceilometer had his design sessions during the last days of the summit. We
noted a lot of things and commented during the sessions in our
Etherpads instances.
The first session was a description of Ceilometer core architecture for
interested people, and was a wonderful success considering that the room was
packed. Our Doug Hellmann did a wonderful job
introducing people to Ceilometer and answering question.
<figure>
<figcaption>Doug explaining Ceilometer architecture.</figcaption>
</figure>
The next session was about getting feedbacks from our users. We had a lot of
surprise to discover wonderful real use-cases and deployments, like the CERN
using Ceilometer and generating 2 GB of data per day!
The following sessions ran on Thursday and were much more about new features
discussion. A lot ot already existing blueprints were discussed and quickly
validated during the first morning session. Then,
Sandy Walsh introduced the architecture they
use inside StackTach, so we can
start thinking about getting things from it into Ceilometer.
API improvements were discussed without surprises and with a good consensus
on what needs to be done. The four following sessions that occupied a lot of
the days were related to alarming. All were lead by Eoghan Glynn, from
RedHat, who did an amazing job presenting the possible
architectures with theirs pros and cons. Actually, all we had to do was to
nod to his designs and acknowledge the plan on how to build this.
That last two sessions were about discussing advanced models for billing
where we got some interesting feedback from Daniel Dyer from HP, and then
were a quick follow-up of the StackTach presentation from the morning
session.
Havana roadmap
The
list of blueprints targetting Havana is available
and should be finished by next week. If you want to propose blueprints,
you're free to do so and inform us about it so we can validate it. The same
applies if you wish to implement one of them!
API extension
I do think the API version 2 is going to be heavily extended during this
release cycle. We need more feature, like the
group-by
functionnality.
Healthnmon
In parallel of the design sessions, discussions took place in the
unconference room with the Healthnmon developers to figure out a plan in
order to merge some of their efforts into Ceilometer. They should provide a
component to help Ceilometer supports more hypervisors than it currently
does.
Alarming
Alarming is definitely going to be the next big project for Ceilometer.
Today, Eoghan and I started building blueprints on alarming,
centralized in a general blueprint.
We know this is going to happen for real and very soon, thanks to the
engagements of eNovance and
RedHat who are commiting resources to this amazing
project!
The Debian project distributes software packages via its archive, propagated
through a network of mirrors. This archive is organised in three areas,
main, that contains the Debian system, and contrib and non-free, that
contain accessory packages that are not Free. The packages in non-free
contain files that have a proprietary license. The source packages in
contrib contain only Free files, but produce binary packages that are
either useless without the access to remote services, or contain non-Free
files, or cause the installation of non-Free materials.
The contrib area is an endless source of discussions, as it is not always
easy to draw the line between what is usable and what is useless without
remote services. I wonder if it would be better to replace the contrib
area by a new priority level, remote , lower than extra. Packages that
are entirely Free (source and binary) and that do not cause the installation
of proprietary software would be fit for main. For the packages that can
not function without non-Free software, given that the work to be done to
free them is the same regardless whether the proprietary code is included in
the source package (in non-free) or not (in contrib), I think that in
both cases the package should be in non-free as it is not Free in
practice.
This would allow for a simple criterion: does the selection of a binary
package trigger the installation of non-Free files? If yes, the package and
its source belong to non-free. If not, they belong to main. Then, if
they need to access a remote service that would not be possible to set up on
a network where only Debian is available, their priority would be remote.
As the first and third quantile are close together we can assume that the majority of the work is done by a few, especially since the second quantile is 5. This is supported by the histogram below, where the x axis is the number of recorded messages and y is the number of developers.
A colleague of mine at Facultad de Ingenier a pointed me to a note published in the Faculty's gazette about 
The first leg was skipped because of the low cloud cover. Weather reports in Switzerland, such as 
I know it's been ages since I last blogged anything at all. To some degree I had a down phase, but I hope to get out of it. It's nice to see that there are people out there who give me a prod every now and then, and don't let me drown. Thanks, most of you probably know that I mean exactly you, and in case you are uncertain, you probably are meant if you contact me every now and then.
Anyway, if you remember that I blogged about 
Imagine you had an excellent successful Kickstarter campaign, and during
it a lot of people asked for an Android port to be made of the software.
Which is written in Haskell. No problem, you'd think -- the user interface
can be written as a local webapp, which will be nicely platform agnostic
and so make it easy to port. Also, it's easy to promise a lot of stuff
during a Kickstarter campaign. Keeps the graph going up. What could go
wrong?
So, rather later you realize there is no Haskell compiler for Android. At
all. But surely there will be eventually. And so you go off and build the
webapp. Since Yesod seems to be the pinnacle of type-safe Haskell web
frameworks, you use it. Hmm, there's this Template Haskell stuff that it
uses a lot, but it only makes compiles a little slow, and the result is
cool, so why not.
Then, about half-way through the project, it seems time to get around
to this Android port. And, amazingly, a
I've only helped
"politically" here and there with this over the years, and I'm
happy to see it live. Your thanks for this should go to the blog
editors---Francesca Ciceri and Ana Guerrero Lopez---and to DSA for
making it real. A proper delegation for the editors is pending and
I'm confident the next DPL will pick it up.
DRI3K First Steps
Here s an update on 
The differences between how one fuels one's car are quite pronounced,
between California and Australia.
Firstly, if you're paying with plastic, it's a given that you can pay at the
pump. I could count on my hands the number of times I've had to walk into a
gas station to pay for gas in the US. Having a small child, I was not
looking forward to having to either leave her in the car so I could pay for
my petrol, or having to deal with all of the rigmarole of getting her out of
her car seat, just so she can accompany me inside the petrol station to make
a brief transaction and then have to get her back into her car seat again.
Not to mention how it drags out the whole process. Yesterday I had to wait
for a pump while everyone leaves their car, queues inside to pay a single
cashier, and then returns to their car and drives away. It'd be an
interesting 
Regular expressions are a wonderful tool, but as many other powerful tools they tend to be misused. If I had a penny for everytime I recommended against using regular expressions for parsing HTML fragments . yeah you know what I mean.
This time made me think a bit about my own use of regular expressions. Do I ever fall in the trap of misusing regular expressions myself? Looking at my current code base I couldn t find any glaring misuses, if I could only get a list of all the regular expressions in my current project.
Regular expressions to the rescue? No, that would probably be quite an horrible adventure. Luckily I have tools to parse Perl: