RFM69 AFC failure with OOK

I observed two issues with automatic frequency correction (AFC) and OOK reception:

  • The AFC value does not get cleared by switching to sleep mode and back to receive
  • AFC in continuous OOK mode can yield extreme values, setting the receiver totally off-band.


AFC should not be used in continuous OOK receive mode. I am not sure whether it is designed to work in this mode (I doubt it). When signal is detected by a high RSSI, an controlled AFC may very well be to late for the OOK pulse. But even not enabling AFC in OOK imposses a risk.

When switching between FSK-packet mode for communication with the sensor network, and receiving external OOK signals, the AFC does not get resetted to zero. It can only be done by a HW-reset or power cycle, Or by explicitely clearing the AFC Value:

template< typename SPI >
void RF69A<SPI>::setThd (uint8_t thd) {
    this->writeReg(REG_AFCFEI, (1<<1));
    this->setMode(MODE_SLEEP);
    this->writeReg(REG_OOKFIX, thd);
    chThdSleepMilliseconds(400);
    this->setMode(MODE_RECEIVE);
}

RFM69 OOK signals sampled with RSSI

By polling the RFM69 RSSI register at rates ranging from 25us to 100us, an oscilloscope like picture can be generated for OOK signals.


Read More

RFM69 OOK-mode RSSI behaviour

With the RFM69 in continuous FSK mode, polling the RSSI register at sufficient rate can be used to acquire OOK signals like on oscilloscope. However when doing so with receiver mode set to OOK, it becomes clear that the RSSI register is used internally for some other purposes:


Read More

Plugwise-2-py Web Application

I have extended Plugwise-2-py with a dedicated very light webserver and  two web applications.

  1. Control Switches, Schedules and read actual power.
  2. Configuration: Circle properties and Schedule editor.

Schermafdruk 2014-09-25 23.50.03

Plugwise circles have this nice schedule and standby-killer functionality. Using the standby-killer raises an issue for some devices. Like the need for a button to switch devices on again. And otherwise, one may prefer to turn on lights that otherwise are turned off by the programmed schedule.

For use under linux, in the open source domain, there were not many UI-based solutions available, so I felt the need of solving this. I developed a control app that works on phones, tablets and common webbrowsers on the PC’s. By using Bootstrap 3 and AngularJS technology, I ended up having a compact and dynamic application. And the webserver is standard library python!

Depending on the enabling of 10-seconds monitoring and/or logging of buffered circle recordings, the power reading is updated regularly. Circles which are configured as always-on, can’t be switched on or off, or operated by a schedule. This comes in handy when for example monitoring the production of solar PV installation, or monitoring your fridge.

WebSocket HTML webserver Python

I have written a HTTP and WebSocket handler in python being able to serve them on one and the same port:

https://gist.github.com/SevenW/HTMLWebSocketHandler.py

Although this sounded very obvious as application, I could not find an example only using standard python 2 (2.7) libraries. There are some WebSocket extensions for well-known python web-server applications such as Tornado.

Background

I wanted to make a web-interface for a python application controlling and reading Plugwise devices from small linux computers suchs as Raspberry Pi, Beagle Bone Black or Odroid U3 or XU. It would add value if it was a standalone webserver, not requiring further installation of something like LAMP, or having the need to copy the to be served files to specific webserver locations. Python has nice HTML page serving in its standard library.

As it had to become a dynamically updating application, I was considering server side events (SSE) or WebSockets. I found a few websocket implementaitons for python, but they all used lower level socket libraries. I tried to get two of them work to serve HTTP and websockets on the same port, but did only get it partially working. The upside was that I was starting to get a deeper understanding of standard python SocketServer, BaseHTTPServer and SimpleHTTPRequestHandler. This has led to a combined HTTP- and WebSocket server, which is able to use the same port.

I was inspired by two earlier implementations:

Jamie Kirkpatrick’s jkp/websocketsserver.py https://gist.github.com/jkp/3136208
Opiate’s SimpleWebSocketServer https://github.com/opiate/SimpleWebSocketServer

Design

In its design it is simply using  SimpleHTTPServer which just acts as a basic webserver responding to GET requests to serve files. This is specialized into a new HTTPWebSocketsHandler which overrides handle_one_request(self) to intercept any request to be able to handle it as websocket communication, once the connection is upgraded to a websocket.

The other method it overrides is do_GET() so it can tap into GET requests to detect a WebSocket upgrade request, and put the handler into the websocket state. For other GET requests, it just calls the super-class handler: SimpleHTTPRequestHandler.do_GET(self)

    def do_GET(self):
        if self.headers.get('Upgrade', None) == 'websocket':
            self.handshake()
        else:
            SimpleHTTPRequestHandler.do_GET(self)

In the Plugwise-2-py application, I further specialize HTTPWebSocketsHandler into class PW2PYwebHandler(HTTPWebSocketsHandler) to handle GET and POST requests, where one normally would use PHP scripts to handle for example AJAX calls. If this specialized class decides not to handle the request, it calls the super class methods.

This handler also implemements overrides for the websocket handling, for example to publish and handle received MQTT messages.

See my github for the plugwise application:

https://github.com/SevenW/Plugwise-2-py

Maple / Olimexino STM32 datalogger for JeeNode network

I have been looking into autonomous persistent dataloggers for a JeeNode based sensor network. The requirements include a Real Time Clock, SD-card storage and battery backup operation. the performance of a standard Arduino / JeeNode based solution was a bit risky, in view of the SD-card operation. Maybe libraries have improved by now, but during SD-card operations, interrupts may be blocked for maybe over a tenth of a second, and that can lead to missing messages from the wireless RFM12B sensor network that I want to collect data from. I considered small ARM-soc computers, like the RaspberryPi. This could also be interfaced with a RFM12B module. It has sufficient processing power, but for me there were two downsides: 1. It consumes 2.5W, so there are higher demands on battery backup. 2. Sometimes it requires maintenance, and may require a reboot once in a while. I just do not want to miss data. So I started looking into more powerfull Arduino-like solutions, and stumbled into a Maple (leaflabs) clone.

2013-11-18 22.04.47

The Olimexino-STM32 is a very interesting board to build a SD-card datalogger on. It has the SD-card slot on board, next to the possibility to feed the circuit with a LIPO battery, which automatically takes over when the USB-bus power supply stops. The board contains a LIPO charger and there is a integrated RTC which can be tuned to drift only 10 seconds per year.

Given the power supply and charger options this board is ideal for a datalogger that should work 24/7 365 days per year. Besides power outages, it also allow for maintenance activities on the server collection all data. When power gets disconnected, it can proceed for about 24 hours on the 1400mAh LIPO battery I connected to it, and when power is up again, it recharges the LIPO cell.

I use a 2GB micro-SD-card (yes, the smallest I had). With my current data inflow, it can buffer data for over a year. The power consumption is about 0.5W from memory. (TODO: measure it again).

To the board I connected a RFM12B 868MHz RF module to communicate with standard JeeNodes. This module is soldered to a RFM12B carrier board obtained from the JeeShop. It provides a pinout compatible with arduino shield connectors, which happen also to be implemented on the STM32 board. This results in very clean wiring.

The Maple-clone is a 3.3V circuit, as well as the RFM12B, so I could leave out the voltage conversion stuff from the board.

At reasonable low cost of around 40 euro’s this datalogger could be build.

  • Olimexino-STM32 E.19.95
  • Lipo 1400mAH (Olimex) E.5.95
  • RFM12B (JeeLabs) E.6.00
  • RFM12B board (JeeLabs) E.3.50
  • box E.3.00

Jeenode RFM12B configuration commands

The following table shows the differences in intialisations of the RFM12B for the JeeNode applicaiton to receiver weather station data from both FSK and OOK stations, and send it to the JeeNode home sensor network, and eventually to HouseMon:

JeeLib nativeWH1080 FSKJeeLib OOKJeelib native remarkWH1080 FSK RemarkJeeLib OOK
0x82050x820Ddisable some circuitsdisable some circuits
0x80E70x80E7 868 Mhz, enable tx+rxidenticaldisable TX and RX buf
0xA6400xA67C0xA68Afrequency
0xC6130xC6130xC691bitrateN/A ?
0x94A20x94A00x9489134kHz, -0dBm, 91dBm 134kHz, -0dB, 103dBm200khz, -6db, 97dbm
0xC2AC0xC220datafilter = digitaldatafilter = OOK
0xCA830xCA830xCA00identicalFIFO disabled
0xCEnn0xCED4group IDgroupID=0xD4!
0xC4830xC49F0xC473AFC auto, free, DQD4AFC manual slow +-16 DQD2AFC @PWR, auto, +-4 DQD4
0x9850
0xCC770xCC670xCC67PLL don't carePLL don't care
0xE0000xE105wakeup timerdon't care
0xC8000xC80E0xC800disable low duty cycle
0xC0490xC0060xC040clock out, low batt leveldon't caredon't care
0XB8000XB800clear transmit bufferclear transmit buffer
0x82DD0x82DD0x82C0enable relevant circuitsidenticalenab receiver, baseband

taking a look at the differences between the native-FSK and WH1080-foreign-FSK, there seem to be some obvious required differences, but also some differences that may not matter that much. For example the auto frequency control (AFC) settings, auto-mode, unrestricted, versus manual, slow, restricted to +/-16steps may not a big deal. So I tested this by making a sketch that receives foreign-FSK, and as soon as a signal is received, it programs the RFM12B to native mode and transmits the package into the sensor network. I extended the RF12 driver with functions below, which speak for themselves. The minimal set of changes can be seen in the following functions:

void configureWH1080 () {
    rf12_setGroup(0xD4);
    rf12_setBitrate(0x13);          // 17.24 kbps
    rf12_setFrequency(0x67C);       // 868.300 MHz
    rf12_setFixedLength(LEN_MAX);   // receive fixed number of bytes
}

void deconfigureWH1080 () {
    rf12_setGroup(GROUP_ID);
    rf12_setBitrate(0x06);          // 49.2 kbps
    rf12_setFrequency(0x640);       // 868.000MHz
    rf12_setFixedLength(0);         // number of bytes to be received in packet header
}

This approach avoids using the slightly costly rf12_initialize() funnction which reprograms the SPI bus and resets the RFM12B.

If you made it to here, but think what is this all about. See this:

Decoding the Oregon Scientific V2 protocol
Receiving OOKASK with a modified RFM12B
FSK 868MHz weather stations on JeeNode
WH1080 protocol V2 – FSK

Unidentified 868MHz OOK signals

In my attempts to receive weather stations on the 868MHz band, without exact knowledge on for example transmission frequencies, I have performed sweeps of frequency range 867.850 – 869.000 MHz in steps of 20kHz. In the loggings I have been looking for periodic signal. Using a histogram of pulse durations, both on and off durations, I was able to recognize signatures of signals. This approach led to the identification of the following signals:

  • Two Alecto WS4000 or similar weather stations.
  • An Oregon Scientific THN128 433MHz, received at 868MHz band.
  • A Philips outdoor temperature sensor for Philips clock radios, also a 433MHz module

Besides those identified signals, I have several unidentified periodic signals, many with a periodicity of around or exact 30 seconds. I have created some signal catalogue for my own reference, but any help on those signals is appreciated. send me a message on info at sevenwatt com.

WH1080 protocol V2 – FSK

The Fine Offset weather station switched over to a new RF transmission protocol somewhere in 2012. While the old protocol was a On-Off-Keying (OOK) protocol, the V2 protocol used Frequency-Shift-Keying (FSK). This is natural transmision mode of the RFM02 transmitters and their RFM01 receivers. The RFM12B modules can also receive the same FSK signals. Most of the reverse engineering happened on the Raspberry Pi forum: WH1080 V2 protocol decoded

This post details on the exact transmission protocol.


Package definition:
[
preample 3 bytes 0xAA    synchron word    payload 10 bytes  postample 11bits zero
0xAA    0xAA    0xAA     0x2D    0xD4     nnnnn---nnnnnnnnn 0x00     0x0
101010101010101010101010 0010110111010100 101.............. 00000000 000
]
repeated six times (identical packages) per transmission every 48 seconds
There is no or hardly any spacing between the packages.
Spacing: to be confirmed.

When using the RFM01 or RFM12B, the preample and synchron word will not be in the received data. The preamble is intended to have the frequency synthesiser locked, while the synchron word serves as detection of the proper message.
Open issue: How can an end-of-transmission be detected? Would VDI turn into zero?
The payload contains to types of messages for FO WH1080, and relatives:

Payload definition:
Weather sensor reading Message Format:
AAAABBBBBBBBCCCCCCCCCCCCDDDDDDDDEEEEEEEEFFFFFFFFGGGGHHHHHHHHHHHHIIIIJJJJKKKKKKKK
0xA4    0xF0    0x27    0x47    0x00    0x00    0x03    0xC6    0x0C    0xFE
10100100111100000010011101000111000000000000000000000011110001100000110011111110


with:
AAAA = 1010    Message type: 0xA: sensor readings
BBBBBBBB       Station ID / rolling code: Changes with battery insertion.
CCCCCCCCCCCC   Temperature*10 in celsius. Binary format MSB is sign
DDDDDDDD       Humidity in %. Binary format 0-100. MSB (bit 7) unused.
EEEEEEEE       Windspeed
FFFFFFFF       Wind gust
GGGG           Unknown
HHHHHHHHHHHH   Rainfall cumulative. Binary format, max = 0x3FF,
IIII           Status bits: MSB b3=low batt indicator.
JJJJ           Wind direction
KKKKKKKK       CRC8 - reverse Dallas One-wire CRC

DCF Time Message Format:
AAAABBBBBBBBCCCCDDEEEEEEFFFFFFFFGGGGGGGGHHHHHHHHIIIJJJJJKKKKKKKKLMMMMMMMNNNNNNNN
Hours Minutes Seconds Year       MonthDay      ?      Checksum
0xB4    0xFA    0x59    0x06    0x42    0x13    0x43    0x02    0x45    0x74

with:
AAAA = 1011    Message type: 0xB: DCF77 time stamp
BBBBBBBB       Station ID / rolling code: Changes with battery insertion.
CCCC           Unknown
DD             Unknown
EEEEEE         Hours, BCD
FFFFFFFF       Minutes, BCD
GGGGGGGG       Seconds, BCD
HHHHHHHH       Year, last two digits, BCD
III            Unknown
JJJJJ          Month number, BCD
KKKKKKKK       Day in month, BCD
L              Unknown status bit
MMMMMMM        Unknown
NNNNNNNN       CRC8 - reverse Dallas One-wire CRC

The DCF code is transmitted five times with 48 second intervals between 3-6 minutes past a new hour. The sensor data transmission stops in the 59th minute. Then there are no transmissions for three minutes, apparently to be noise free to acquire the DCF77 signal. On similar OOK weather stations the DCF77 signal is only transmitted every two hours.

The package format was deduced using a long transmision buffer on a JeeNode with some modificaitons in the RF12 driver.
The payload definitions have been described at those pages:
WH1080 V1 OOK protocol
WH1080 V2 FSK protocol
WH1080 V1 OOK and DCF77 message format

This applies at least to the following (later) models:
Fine Offset WH1080
Alecto WS4000
National Geographic 265, at 916MHz

Receiving 868 and 433MHz weather stations

Receiving 868 and 433MHz weather stations

Using an Arduino, JeeNode, Nodo, or Raspberry Pi with RFM12B, RFM01 or superheterodyne receiver, sensors of popular wireless consumer weather stations can be received. Your own, or your neighbors. This article is dedicted to collecting internet source on RF transmission protocols, as the available information seems to be scattered a lot. Oregon scientifc protocols are readily available. Then there semes to be a whole class of OEM weather stations from China, such as Fine Offset. Some of those modesl are avialable as Maplin, Alecto and more. Personally I started with a superheterodyne receiver at 433MHz, with which I was able to recieve my version-1 protocol Orgeon Scientific THN128 sensor, only in the same room. It did not make the next room. But my main sourceof inspiration was this article: http://www.susa.net/wordpress/2012/08/raspberry-pi-reading-wh1081-weather-sensors-using-an-rfm01-and-rfm12b/
which inspired me to invest in HopeRF modules. After I was able to receive Alecto WS4000 or alike stations (two somewhere in the neighborhood, but not the one I aimed at) with a Raspberry Pi, and very noisy with an Arduino Nano, I decided to invest in JeeNodes. Now my “production receiver will be a Jeenode with on-board RFM12B, or added RFM01. To be decided. My experimatal platform is the Raspberry Pi, as I can code easily very sloppy, use large amounts of memory and do all kinds of checks while still being in time for the next pulse.

The following sites/communities have loads of information on receiving weather stations:
http://www.jeelabs.org
http://nodo-domotica.com
The Nodo community is imho a bit difficult to access, as the major source of documentations is the c-code of a userplugin.

Through the following links RF transmission protocols descriptions can be found:
Oregon scientific:
Detailed description of V1, V2, V3 protocols:
http://wmrx00.sourceforge.net/Arduino/OregonScientific-RF-Protocols.pdf
Decoding of the V2 protocol links to jeenode/arduino/atmel code:
http://jeelabs.net/projects/cafe/wiki/Decoding_the_Oregon_Scientific_V2_protocol