We’re On Like Donkey Kong

Firmware Stability

The fixes for working around silicon issues that Nordic provided initially showed some good results in that they removed the stability issues during extreme, rapid temperature change (freeze a Wimoto Climate to ~-15c, then bring it to ~+40c quickly). However, when we started to power profile the devices earlier this week, we found that the reason things were stable is that the main CPU clock was being used nearly 100% of the time. I’ll attempt to explain and illustrate why this is a “very bad thing”, but it’s a relatively hard to articulate (or at least I’m not sure I’m able to do it very well 🙂 ).

Bluetooth Low Energy (Bluetooth Smart) achieves much of its potential energy savings through pretty much spending most of its life sleeping — the best way to save energy is not to expend it, after all. The radio, by far the most power hungry component, wakes up in a orchestrated fashion to advertise the device or to exchange data with another device in a coordinated fashion. Even when you’re “connected” to a Bluetooth Low Energy device, the radio is only on periodically and there’s all sort of complicated orchestration to pull off keep things in sync between the two radios. One additional way to save lots of power whilst sleeping is to switch from the clock that drives the CPU (which in our case is 16Mhz and referred to as “HFCLK”; high-frequency clock) to one that just drives a counter that will wake up the CPU or other peripheral blocks on the silicon when necessary. This is called an LFCLK (low-frequency clock) in our world and can either be an RC oscillator included on the silicon, or an external “watch” (32.768KHz) crystal.

Due to some issues on the silicon revision of the Nordic nRF51822 we use inside our system-on-a-chip, two major things weren’t happening properly:

(1) The LFCLK included on the silicon wasn’t getting recalibrated periodically. The on-chip clock source is a resistor-capacitor network (an “RC oscillator” as they’re called). One of the interesting things about an RC oscillator is their frequency is partly a function of their temperature and temperature changes need to be compensated for with periodic recalibration. In fact, you can technically use an RC oscillator as a temperature sensor to exploit this phenomenon. Unfortunately, these recalibration routines were failing to work properly in our latest firmware due to the particular mix of silicon plus SDK plus SoftDevice (Bluetooth Smart radio stack).

(2) The nRF51822 is designed for aggressive power management and you can literally switch various “blocks” of the chip off to save power. These can be peripherals such as the UART, I2C/TWI lines, various onboard power supplies, the radio baseband itself, and even the crystal that drives the main ARM Cortex CPU (“HFCLK). Unfortunately, when we implemented fixes to get the LFCLK to be stabilized, a lot of our work to shut off various blocks on the chip fell to pieces — causing high current consumption, which would have caused battery-life woes.

If you’re spending most of your time asleep and using tiny amounts of current (~3-4uA) and only waking up periodically (every few hundred milliseconds) for short “bursts” where you use relatively high amounts of current (a couple of milliseconds using 1mA to ~20mA),  you can keep going for a long time even on a relatively puny battery (a CR2450 coin cell, which has a capacity of about 620mAh, in our case). If for whatever reason you aren’t spending most of your time (99%+) sleeping, you suddenly start to see battery life disappearing rapidly as there is a huge (a factor) difference between sleep current and the various active states of current (~1-20mA) — especially if you happen to not be able to shut down the CPU clock source.

For our devices, we also have several housekeeping tasks that will wake up the CPU — everything from reading hardware sensors, to the data logger routines, calibration, alarm management, realtime clock management. We do a *lot* of things onboard compared to your average BLE tracker or similar device — you can get a glimpse into from the complexity of our profiles in our developers docs. These functions are all fairly aggressively power optimized too.

We’ve spent most of the week going back and forth with our Nordic application engineer and have now what we believe to be the right “constellation” of fixes. We also had to move from temperature-based recalibration of the internal LFCLK to a simple, periodic recalibration — which consumes more energy, but the temperature-based calibration just wasn’t working properly even after applying various “fixes”.

The following ‘scope trace might help illustrate things a little better for those interested (at least one person has asked for a technical explanation so far). This is a “healthy” power profile where we spend most of the time at ~3.5uA of power consumption and then periodically shoot up to ~1mA when the CPU is active and ~15-20mA when the radio is active. We basically want to see a flat line for the most part. A couple of days ago that flat line was pretty much where the peaks for active CPU were, which affected energy performance by about a factor of 500.

climate_current

So, the long and the short of things:

– We believe we’re back in “good shape” as far as radio stability is concerned
– We believe we’re in reasonably good shape as far as power profiling is concerned, although it’s taken some effort to get there.
– To avoid any potential “battery-gate” incident, even though we don’t think that’ll happen, we’re including an additional CR2450 battery in the secret compartment inside each Wimoto inner carton, which also contains the Velcro pads for Climate and Sentry sensors to be attached to objects. See below for pictures illustrating where they’re hidden.

IMG_3050 IMG_3051 IMG_3052

Shipping

We’re hoping to ship several hundred Climate’s (first wave of what we can ship) next Friday. It will take some time to get to all the products, all the backers, and all the perks; so please, please, please be [a little more] patient and we’ll continue to try and set expectations on what’s going out.

20 comments

  1. Good, crossing fingers that there will be no other issues, that the battery will keep up and that all promised functions are well working with my iPhone.
    Then crossing fingers as well to receive the product in a reasonable frame time 😉
    Will you provide some ETA to us? Last time you said my ETA should have been beginning of October last year.
    Cheers and thanks for all those regular updates!

  2. I hope this is the time,I’m really tired of waiting and waiting…. I really hope that the reward will be finally receive fine tunning flawless pieces…Will you also ship Thermo and Growmotes beggining next Friday??

  3. Dale C. Carlson · ·

    Great news. Congratulations. I look forward to seeing it work.
    Dale and Cookie Carlson

  4. lannierose · ·

    Thanks for the technical explanation. It is amazing how much complex technology is packed into such a seemingly trivial device, and for such a low cost. You guys (and gals?) are really paying your dues!

  5. Thank you for sharing the details with us, I really appreciate your communicativeness in the last weeks. That’s how you keep your backers involved and motivated. Seriously. I’m happy to read some technical details.
    At least we now have a entertaining serialized detective story. 😉
    Keep up the good work folks, you’re almost there! 🙂

    1. Thanks, Ben. We’re trying to ship a good product and rewriting our firmware is ultimately a good thing for our customers — better field-upgradeability, concurrent operation with smart devices and our/your gateway devices, etc.

      1. I totally agree, I would also prefer shipping with the new firmware instead of the old one. I don’t mind waiting a few more weeks. After all, that won’t make a significant difference. 🙂

        As an engineer I always prefer to have a good product with a delay rather than having a buggy product on time. It’s in the nature of working on the edge of technology that things are going wrong. And that you guys are pioneers is proven by the fact that there is no competitor in the market that has a comparable product available today, despite the significant customer demand (as shown by the impatient backers).

        I’m happily looking forward to the day when the postman rings with a nice parcel from you guys!

  6. So should we wait for shipment of sentry devices end of February or March?

    1. Some Sentry’s (~100) should go out in the next couple of weeks, but the majority would be in the time-frame.

      1. So is there a way not to ship me any, as I am out of the country till March and I expect the custom offices will block the shipment and return it if they can’t find me.

        I don’t mind waiting for mine in March if that is option, let others enjoy their order first. 🙂

  7. Can you please update the staus about thermo and grow sensors??best regards

    1. As soon as our contract manufacturer can confirm some things, we’ll let folks know. There’s no point in providing vague or optimistic dates.

  8. Any update on shipping that first batch? What about all approvals?

    1. We need to run Thermo and Leak (Water) through ESD one more time, but that’s trivial (harder to book the lab than do the test). First Climates are trickling out.

  9. Thomas Edwards · ·

    If you are starting to ship ( which is really really good news) where do I update my shipping information?

    1. Climate’s are trickling out right now and shoettly we will repatriate thousands of the various Wimoto’s from our CM in Serbia. The Shipping App will send you a note before you shipment is ready confirming address, bugging you for shipping (if applicable), etc.

  10. davesnowdon · ·

    Hey guys, I think we’re due another update. Any response from the contract manufacturer etc? Please don’t return to the long periods of silence that marked 2014

  11. Any more updates? The Wimoto app still isn’t available to download on the iTunes store, so getting the actual product is kind of a moot point right now. Any idea on when the rest of the climates will be going out?

    1. Just replying to myself… Any update? I think we’re do a blog post (good or bad) after three weeks of being told shipping is imminent again.

  12. update for shipping please!

%d bloggers like this: