SENML: smart messages for devices

senml: a response to the tower of babel problemAbout SenML

In today’s post I’d like to talk a little about an upcoming messaging format called SenML. It’s currently still an internet draft but it appears to be on track for becoming a new internet standard soon.

So, what’s it for, you might wonder, and why should I care?  Well, these are pretty good questions. The purpose of SenML is well defined: it describes sensory information, as in temperature readings, pressure measurements or geo-location data like latitude and longitude. In other words, you can use this data structure to send your device’s sensory measurements to other devices or some IOT service. You can also send  actuator commands to your devices in SenML. And Internet-of-things applications can use it as their internal and/or external data format.

Why

Ok, but what’s the relevance for me, you are probably asking. After all, I can already transmit my sensor data and actuator commands in plenty of existing data formats like xml, json, cbor or just plain old C data structs. Well, SenML and xml, json or cbor aren’t mutually exclusive. SenML forms a layer on top of those already existing data formats.  The problem that senml tries to address is this: interoperability. If you have already done something with the internet-of-things before,  like me, you probably came to the realization that most of the systems are currently pretty much closed. Everybody’s got their own data format and doing their own thing: walled gardens everywhere. Or, to put it differently, today’s situtation is a bit like the tower of babel (hence the picture). Everybody speaks a different language.  And SenML tries to become that 1 tongue everybody can understand.

Use case

This forms a problem for all players in the IOT game, even if you don’t realize it yet. Let’s take the example of a hardware manufacturer who’s built a shiny new outdoor temperature sensor with lots of bells and wissels. He’d like to have some IOT platform to go with it so people can actually use it (aka see the results and control it remotely). Now, he is left with a choice:

  • build his own platform and applications, using his own custom data formats
  • tailor his data format specifically to 1 or more existing platforms and restrict usage to those.
  • define his own custom format, publish a spec and hope platform builders integrate it into their eco-system.

Where all of these paths have problems:

  • Should a device manufacturer also be in the business of iot platforms and waste his time building apps?
  • If he limits support to his own or a couple of existing platforms, will he reach a big enough customer base?
  • How about interoperability with other devices that don’t work on his selected platforms, but are owned by his customers and would like to see those devices work together.
  • some very big player might get away with just publishing the specs, but for the rest, platform builders will probably ignore their devices cause of limited resources

Solution

Well, this is where senml tries to fill the gap: if all (or most) platform builders and device manufacturers use the same data format to describe sensory data, this hardware producer no longer needs to worry about which platforms are able to work with his device: they all do. So this way, he doesn’t need to build his own applications, he can rely on already existing tools. Which is a win-win-win situation for all:

  • device manufacturers save precious resources
  • platform builders save resources and can support lots and lots of devices
  • users can work with their preferred and known tools for all the stuff that they own.

Examples

The smallest senml message possible:

[
 {"n":"urn:dev:ow:10e2073a01080063","v":23.1}
]

As you can see, it’s expressed in json. This is simply cause it’s the most readable and compact form (the cbor version would have been a bit more challenging).

This examples provides us the following information:

  • the name of the sensor is urn:dev:ow:10e2073a01080063
  • and the value is 23.1

Here’s a more elaborate example:

[
   {"bn":"gateway","n":"temp","u":"Cel","v":23.1}, 
   {"bn":"dev1", "bv": 200, "n":"distance","u":"m","v":1}
]

This is a message from a gateway.  It contains data from the gateway device itself: a temperature reading in degrees celsius.  And for another device called ‘dev1’, which is a distance measurement in meters.

General concepts

  • A senml message is called a pack
  • A pack is always an array of records.
  • A record contains a single measurement.
  • A record should, at a minimum, always contain a value and a name (or base-name defined in it’s own record or a previous one).
  • value labels can be:
    • v: numeric values
    • vb: boolean values
    • vd: binary data
    • vs: strings
  • You can also put time information in a record:
    • the regular time field (‘t’) to indicate when the measurement was taken.
    • update time (‘ut’) to give an estimate when to expect a new measurement.
  • All the fields that start with a ‘b’ refer to ‘base’. A senml message can have the following ‘base’ labels:
    • base name: this value will be prepended to all ‘names’ (n) that follow until a new base name. Usually, the ‘base’ part contains the name of the device and the ‘name’ is the name of the sensor.
    • base value: this vaue is added to all numeric values that follow. So if the base value says ‘200’ and the value is 1, then the actual measurement was 201. This is for saving space.
    • base time: this indicates the start of the measurements. So, all time labels (‘t’) that follow, are an offset of this base time.
    • base unit: all records that follow which don’t have a unit indicator, should use this value as their unit. This is again to save space.
    • There is also a base sum which works exactly the same as base-value, but for the ‘sum’ field. The specs say that you should either use value or sum, but never both at the same time. Personally though, I haven’t yet come up with a good use case for the sum field
  • A senml messages work for both sensor and actuator data.

Available libraries

Although building small strings of json isn’t too hard, things can get pretty complex real fast when working with multiple measurements across time and with multiple devices. Especially if you want to use features like base value and base time. It becomes even worse when you want to receive actuator commands in senml: parsing json is still doable, but cbor?

For this reason, I prefer to use a library that already does most of the heavy lifting for me.  Things like automatically calculating offsets compared to a base time or base value. And for actuators, it’s much easier to register a callback function that gets triggered whenever the right record is found.

At this point, I should probably make a small note that the author of this article also made the following libraries:

  • senml-c-library: A C++ version, geared towards devices with very small available memory (2K is more than enough). It currently works on most arduino compatible devices and mbed.
  • senml-python-library: a python version with similar features as the C++ version, although this one uses a bit more memory.
  • senml-micropython-library: a port of the python version that works on microcontrollers like the pycom devices.

And an example code snippet for the arduino platform:

void setTemp(float value){
   Serial.print("set the temp of the boiler to ");
   Serial.println(value);
}

SenMLPack doc("device_name");
SenMLFloatActuator rec(KPN_SENML_TEMPERATURE, SENML_UNIT_DEGREES_CELSIUS, setTemp);

void setup(){
   doc.add(&rec); 
}

void loop(){
   const char* in = "[{\"n\":\"temperature\",\"u\":\"Cel\",\"v\":23.1}]";
   doc.fromJson(in);
   char out[80]; 
   memset(out, 0, sizeof(out)); 
   doc.toCbor(out, sizeof(out), SENML_HEX);
}

This example receives a senml message (or simulates it) in json format. The ‘doc’ object parses this and calls the ‘setTemp’ function when it finds the ‘temperature’ record in the input.  Next, the doc object renders the same message in cbor as a hex string so the device can send it back as a sort of ‘ack’ message.

Limitations

Although Senml is an improvement over the current situation quite a bit, it isn’t the holy grail that solves all the interoperability issues in the IOT world. There are still plenty of problems to improve upon. For instance:

  • API discovery:  it would be nice if a device could request the available endpoints and supported features of a platform.
  • device definitions: Senml already declares some extra information about it’s measurements like the measurement unit and the next update time. But it only provides this info when a measurement is sent. Platform builders probably want to know this earlier so that they can build the correct dashboards. They also need more info like min-max, prefered widget type,…
  • firmware management: these days, devices can’t go without firmware updates. The managing platform usually does this, but it requires a common interface on many levels, not just the devices. Platforms also need to discover newly available updates.
  • device discovery: A user needs to add and remove devices from his IOT application.

There are most likely even more improvements possible and maybe someone has already solved one or two of these points. I would love to hear about them, so please let me know.

Leave a Reply

Your email address will not be published. Required fields are marked *