Evaluation noise filtering weak OOK signals – RFM69

As explained in a previous post , noise can spoil the reception of OOK signals from remote sensors. One solution to deal with the noise filtering of spikes in the ON/OFF binary signals is to apply a moving average filter of the last n received samples. When n is odd, the “average” is one if more then half the bits in the filter buffer is one, and otherwise zero. With a filter length of n=3 a single outlier sample will not disturb the intended signal. With a filter length of 5, spikes with a duration of two samples can be handled. Clean signals are as snappy as the where, and timings can be determined to the accuracy of the sample rate.

To find the optimal length of the moving average filter, a setup was created with a remote sensor giving a signal that was not received all the time. For each filter length, 100 transmissions where listened to. The graph below shows the success rate as function of the filter length.
433OOK_mov_avg_eval

The binary ON/OFF signal was sampled on the DATA/DIO2 pin of the RFM69 with sample rate 25µs. The bitrate was set to 4000bps. With this setting, the internal sampling rate of the RFM69 is 10µs. This “guarantees” a noisy signal for test purposes. The filter length n=1 is actually unfiltered data at 25µs/sample offered to the decoders. The longest filter length n=15 takes an averaging window of 375µs. This is resiliant to noisy spikes (either ON or OFF) of 7 times the sample rate, thus 175µs. The minimum pulse duration of the received signal, the Oregon Scientific V1 sensor, is about 1100µs, and obviously can stand the long filter periods.

For OOK signals that vary quicker, possibly a shorter filter length should be taken. The shortest pulse durations of OOK sensors and remote controls is around 300µs.

The design of the filter is simply an unbsigned integer, in which the most recent reading (1 or 0) is shifted in from the right. The filter length is enforced by only looking at the least significant n bits. Deciding on the filtered value is a matter of counting the one-bits in the filter.

//moving average buffer, max length = 31!
static uint32_t filter= 0;
static uint8_t avg_len = 7;
static uint32_t sig_mask = (1 << avg_len) - 1;

void receiveOOK() {
    while (....) {
        filter = (filter << 1) | (data_in & 0x01);

        //efficient bit count, from literature
        uint8_t c = 0;
        uint32_t v = filter & sig_mask;
        for (c = 0; v; c++)
        {
          v &= v - 1; // clear the least significant bit set
        }
        uint8_t data_out = (c > (avg_len >> 1)); //bit count moving average

        //decode OOK signal
    }
}

Debugging failure to detect OOK signals

For more remote OOK temperature sensors, occasionally transmissions arrive malformed, and can therefore not be detected as such or properly decoded. The Oregon Scientific THN128 protocol V1 sensor perfoms two transmissions spaced some 200ms apart. When the first transmission is received properly, it is easy to record the raw RSSI and OOK data from a RFM69 during the transmission of the repeated signal. I have recorded two of the transmissions where the first part was succesful, and the second part contained an error.

The errors are in the marked area’s. It appears that such a bit-error is a noise spike that only takes one sample in both cases. Although it is a bit early to proof, it is very likely that failures to detect weak OOK signals are often short time-scale (high frequency) noise on the OOK signal.

Noise

Recording a histogram of all ON and OFF durations of the second transmission, including the preceeding gap after the end of the first transmission gives better insight in the high frequency noise.
rfm69_ook_oscv1_pulse_dur_histo_4000bps_n10

The OFF (ZERO) peak at 1080µs belongs to the received OOK signal. The large distribution at the left hand side of the graph is noise. The red part indicate signal ON (ONE), and tend to be short between 40 and 100µs. For the OFF state the noise tail is a bit longer. It is clear that most noise energy is below the 200µs tiem-scale. Most remote control and sensor OOK pulse durations are above the 300µs time scale, so it should be possible to filter out the high frequency ON/OFF flips to clean-up the signal.

Filtering

So could the reception be improved by getting rid of this type of short spikes on the intended signal? I have been thinking about this. When not using an interupt driven approach to detect ON to OFF and OFF to ON transitions, but by just polling the DATA (DIO2) pin of the RFM69, I can easily implement a kind of moving average filter for the two-valued data. Sampling at for exmaple 40µs and averaging the last 5 samples removes spikes of one and two sample duration. I tested this and it improves the reception of weak signals way better then I expected. I am now collecting more data on the improved performance.

RFM69 OOK internal sampling rate

When the RFM69 is used in continuous OOK mode, the signal can be read from the RFM69 DATA (DIO2) pin. This can be done by polling this line with an ardiuno, jeenode, nucleo STM32 or other MCU. It can also be done interrupt driven by programming interrupts on the rasing and falling edge of the DATA signal. This way accurate timings can be measured. On the nucleo STM32 F103RB with ChibiOS I can measure pulse lengths of OOK signals with an accuracy of one or two microseconds.

I decided to generate some OOK signal (and only noise) histograms to figure out how to deal with some high frequency noise in the OOK signals. I set the OOK bitrate in the RFM to 1000bps as the pulse durations are in the order of one millisecond in the signals I was working with. Then I found that pulse durations I measured were quite discrete. A saw peaks around 40µs, 80µs, etc. Further experimentation revealed that this discretisation internally in the RFM69 depends on the bitrate. With a bitrate of 2000, the peaks occured at 20µs intervals. So I concluded that the internal sampling of OOK signals happens at 25 times the anticipated bitrate (chiprate). More aquirate measurements are required to detemrine the exact periodicity. It could also be 24 times or something like that.

Below are two histograms where the bin size is 10µs, recorded for 30 seconds, with 1000 and 2000 bps.

rfm69_ook_noise_pulse_dur_histo_2000bps

rfm69_ook_noise_pulse_dur_histo_1000bpsIt can be clearly seen that with the lower bitrate also the measured noisy ONES (ON) and ZEORS (OFF) are detected at the 40µs., leaving three bins (nearly) empty in between. With longer durations it can be seen that the mismatch with the 40µs grid slightly increases. This can be due to either the factor of 25 not being accurate, but more likely it can be a mismatch between the RFM69 crystal clock and the nucleo internal oscillator, leading to slight clock differences between the two components.

Conclusion

Internally the RFM69 samples the RSSI at 25 times the programmed bitrate to detect ON or OFF OOK signal. This knowledge can be useful in figuring out the optimal OOK settings for reception of different OOK signals in various formats and bitrates. For example when using one reciever for a light switch control system such as KAKU and Oregon Scientific temerature sensors.

RFM69 Bandwidth investigations part 1

Usually there are several 433MHz devices around the house. Remote controls and various sensors. Normally the exact operating frequency is notknown. A lot of work described on the internet is done with wide-band receivers. The following graphs show the relation between the receiver bandwidth and the bandwidth of the received signal.

Frequency sweeps of continuous OOK transmission with different receiver bandwidth settings. The graphs show the variance on the sampled RSSI signal.

A rule of thumb is that the sweep is performed with the RFM69 receiver OOK-bandwidth being equal to the frequency step size of the sweep. This gives about 3 datapoints with elevated variance, providing good view to estimate the the peak frequency (433.81MHz for the KAKU remote)First a coarse sweep with steps of 100kHz can be perfoemd, followed by a narrow sweep with step size of 5 or 10kHz.

RFM69 OOK signals sampled with RSSI

By polling the RFM69 RSSI register at rates ranging from 25us to 100us, an oscilloscope like picture can be generated for OOK signals.


Read More

RFM69 OOK-mode RSSI behaviour

With the RFM69 in continuous FSK mode, polling the RSSI register at sufficient rate can be used to acquire OOK signals like on oscilloscope. However when doing so with receiver mode set to OOK, it becomes clear that the RSSI register is used internally for some other purposes:


Read More

Plugwise-2-py Web Application

I have extended Plugwise-2-py with a dedicated very light webserver and  two web applications.

  1. Control Switches, Schedules and read actual power.
  2. Configuration: Circle properties and Schedule editor.

Schermafdruk 2014-09-25 23.50.03

Plugwise circles have this nice schedule and standby-killer functionality. Using the standby-killer raises an issue for some devices. Like the need for a button to switch devices on again. And otherwise, one may prefer to turn on lights that otherwise are turned off by the programmed schedule.

For use under linux, in the open source domain, there were not many UI-based solutions available, so I felt the need of solving this. I developed a control app that works on phones, tablets and common webbrowsers on the PC’s. By using Bootstrap 3 and AngularJS technology, I ended up having a compact and dynamic application. And the webserver is standard library python!

Depending on the enabling of 10-seconds monitoring and/or logging of buffered circle recordings, the power reading is updated regularly. Circles which are configured as always-on, can’t be switched on or off, or operated by a schedule. This comes in handy when for example monitoring the production of solar PV installation, or monitoring your fridge.

Jeenode RFM12B configuration commands

The following table shows the differences in intialisations of the RFM12B for the JeeNode applicaiton to receiver weather station data from both FSK and OOK stations, and send it to the JeeNode home sensor network, and eventually to HouseMon:

JeeLib nativeWH1080 FSKJeeLib OOKJeelib native remarkWH1080 FSK RemarkJeeLib OOK
0x82050x820Ddisable some circuitsdisable some circuits
0x80E70x80E7 868 Mhz, enable tx+rxidenticaldisable TX and RX buf
0xA6400xA67C0xA68Afrequency
0xC6130xC6130xC691bitrateN/A ?
0x94A20x94A00x9489134kHz, -0dBm, 91dBm 134kHz, -0dB, 103dBm200khz, -6db, 97dbm
0xC2AC0xC220datafilter = digitaldatafilter = OOK
0xCA830xCA830xCA00identicalFIFO disabled
0xCEnn0xCED4group IDgroupID=0xD4!
0xC4830xC49F0xC473AFC auto, free, DQD4AFC manual slow +-16 DQD2AFC @PWR, auto, +-4 DQD4
0x9850
0xCC770xCC670xCC67PLL don't carePLL don't care
0xE0000xE105wakeup timerdon't care
0xC8000xC80E0xC800disable low duty cycle
0xC0490xC0060xC040clock out, low batt leveldon't caredon't care
0XB8000XB800clear transmit bufferclear transmit buffer
0x82DD0x82DD0x82C0enable relevant circuitsidenticalenab receiver, baseband

taking a look at the differences between the native-FSK and WH1080-foreign-FSK, there seem to be some obvious required differences, but also some differences that may not matter that much. For example the auto frequency control (AFC) settings, auto-mode, unrestricted, versus manual, slow, restricted to +/-16steps may not a big deal. So I tested this by making a sketch that receives foreign-FSK, and as soon as a signal is received, it programs the RFM12B to native mode and transmits the package into the sensor network. I extended the RF12 driver with functions below, which speak for themselves. The minimal set of changes can be seen in the following functions:

void configureWH1080 () {
    rf12_setGroup(0xD4);
    rf12_setBitrate(0x13);          // 17.24 kbps
    rf12_setFrequency(0x67C);       // 868.300 MHz
    rf12_setFixedLength(LEN_MAX);   // receive fixed number of bytes
}

void deconfigureWH1080 () {
    rf12_setGroup(GROUP_ID);
    rf12_setBitrate(0x06);          // 49.2 kbps
    rf12_setFrequency(0x640);       // 868.000MHz
    rf12_setFixedLength(0);         // number of bytes to be received in packet header
}

This approach avoids using the slightly costly rf12_initialize() funnction which reprograms the SPI bus and resets the RFM12B.

If you made it to here, but think what is this all about. See this:

Decoding the Oregon Scientific V2 protocol
Receiving OOKASK with a modified RFM12B
FSK 868MHz weather stations on JeeNode
WH1080 protocol V2 – FSK

Receiving 868 and 433MHz weather stations

Receiving 868 and 433MHz weather stations

Using an Arduino, JeeNode, Nodo, or Raspberry Pi with RFM12B, RFM01 or superheterodyne receiver, sensors of popular wireless consumer weather stations can be received. Your own, or your neighbors. This article is dedicted to collecting internet source on RF transmission protocols, as the available information seems to be scattered a lot. Oregon scientifc protocols are readily available. Then there semes to be a whole class of OEM weather stations from China, such as Fine Offset. Some of those modesl are avialable as Maplin, Alecto and more. Personally I started with a superheterodyne receiver at 433MHz, with which I was able to recieve my version-1 protocol Orgeon Scientific THN128 sensor, only in the same room. It did not make the next room. But my main sourceof inspiration was this article: http://www.susa.net/wordpress/2012/08/raspberry-pi-reading-wh1081-weather-sensors-using-an-rfm01-and-rfm12b/
which inspired me to invest in HopeRF modules. After I was able to receive Alecto WS4000 or alike stations (two somewhere in the neighborhood, but not the one I aimed at) with a Raspberry Pi, and very noisy with an Arduino Nano, I decided to invest in JeeNodes. Now my “production receiver will be a Jeenode with on-board RFM12B, or added RFM01. To be decided. My experimatal platform is the Raspberry Pi, as I can code easily very sloppy, use large amounts of memory and do all kinds of checks while still being in time for the next pulse.

The following sites/communities have loads of information on receiving weather stations:
http://www.jeelabs.org
http://nodo-domotica.com
The Nodo community is imho a bit difficult to access, as the major source of documentations is the c-code of a userplugin.

Through the following links RF transmission protocols descriptions can be found:
Oregon scientific:
Detailed description of V1, V2, V3 protocols:
http://wmrx00.sourceforge.net/Arduino/OregonScientific-RF-Protocols.pdf
Decoding of the V2 protocol links to jeenode/arduino/atmel code:
http://jeelabs.net/projects/cafe/wiki/Decoding_the_Oregon_Scientific_V2_protocol