Smart Nestbox Data Processing

This page explains the different data types being logged on the smart nestbox and guides the field scientists in their work to interpret the data.

Trigger mechanism of the smart nestbox

It is very important to understand the underlying state machine that is determining the data being logged on the smart nestbox. A typical measurement event has the following sequence:

  1. Standby: no weight measurements are logged, only battery voltage (P) and, once per hour, the no-load weight (or “offset”) is being re-measured (O).
  2. Detectiom: a weight of 100g about the initial offset is measured (D). Potentially this is an owl sitting on the perch, but it did not yet get identified. The values measured periodically during this phase cannot directly be transformed into grams, as they are unprecise measurements (short sampling period). It allows though to get an idea about how still the bird was sitting on the perch before it got identified.
  3. Identification: the RFID tag of the owl could be read out and is logged (R) up to three times. If it is three times the same value, one can be confident that the ID was read out correctly.
  4. Exact Measurement: because the owl was identified, the smart nestbox now switches to a continuous weight measurement mode with hight precision (X) and over multiple seconds averaged values (A, only if values are within a certain tolerance).
  5. Identification Re-try: After 10 - 20 measurement points, the RFID tag of the owl is being read out again. If it can still get detected, R is logged. If not, U is logged (as in undefined).
  6. Owl leaves: If the weight falls below ca. 100 g again, X measurements continue for 10 - 20 data points, then immediately the weight is labelled as offset (O) for 100 more data points.
  7. Turn back to standby mode.

Data on SD card

All logging events are first stored in a temprary buffer and periodically written to the SD card into files named LOGXXXXX.txt, where XXXXX is an ascending number. The logging events are represented as a single line in those files. Each of those lines has the following common strucutre:

Logging Character, Timestamp, Data [,complement]

The Logging Character is a predefined ASCII character and encodes the type of data being logged on this line (explained below).

The timestamp is represented as UNIX epoch timestamp, i.e. the number of seconds since 1.1.1970. It is furthermore stored as UTC value, i.e. you need to add one or two hours to convert it into European winter time (CET) resp. summer time (CEST).

Data Type Encoding

Measurement Data

Description Field 1 (Character) Field 2 Field 3 (Data) Field 4 Conversion
Battery voltage P Timestamp Battery voltage in mV 5250 mV = Full, 3600 mV = Empty
RFID identificator R Timestamp Tag ID (hexadecimal) compare with database
Exact Weight Value X Timestamp Weight sensor raw value Deviation from average value during averaging period Weight in grams = ([Raw value] - [Calibration Offset])/[Calibration slope]
Inexact Weight Value (before RFID detection) D Timestamp Approximate Weight sensor raw value No direct relationship to weight in grams, but an indicator if the owl was standing still before its ID was read out.
Weight Offset Value (Current Zero of the scale) O Timestamp Weight sensor raw value See X
Temperature Value T Timestamp Temperature in deci-Kelvin Temperature in °C = Value/10 - 273.15

Meta Data and Status Information

Description Field 1 (Character) Field 2 Field 3 (Data) Field 4 Conversion
Energy safe mode ON/OFF E Timestamp 0 or 1 Field 4 0 = system shuts down for the day. 1 = system wakes up for the night
Average of the last 10 weight values that were within a certain tolerance A Timestamp Weight average Tolerance See X
Two possible meanings: Data=0 –> Timestamp was set by user or reloaded from memory after shutdown. Data>0 –> RFID re-detection was unsuccessful U Timestamp 0 or the last 16 bits of the non-detected RFID number - For information only
Stable Weight Measured and going back to detection mode S Timestamp Weight average Tolerance See X
Header information (stands on top of each file H Timestamp

When reading out an SD card, the logged data is spread accross a big number of files (LOGXXXX.txt with ascending numbers). For the subsequent data processing, it is necessary to consolidate these files into one single file, and then filter them according to the log character.

On Windows machines, the following software can be used to consolidate the log files in a particular folder: Link to github

On UNIX machines (Mac, Linux), the same task can be done in a terminal by going into the directory of the log files

cd "path-to-log-files"

and executing the commands below:

cat LOG* > log_all.csv 
grep "X," log_all.csv > log_X.csv
grep "D," log_all.csv > log_D.csv
grep "I," log_all.csv > log_I.csv
grep "P," log_all.csv > log_P.csv
grep "T," log_all.csv > log_T.csv
grep "R," log_all.csv > log_R.csv
grep "O," log_all.csv > log_O.csv
sed 's/,X,/,/g' log_X.csv > log_X_clean.csv
sed 's/,D,/,/g' log_D.csv > log_D_clean.csv
sed 's/,I,/,/g' log_I.csv > log_I_clean.csv
sed 's/,P,/,/g' log_P.csv > log_P_clean.csv
sed 's/,T,/,/g' log_T.csv > log_T_clean.csv
sed 's/,R,/,/g' log_R.csv > log_R_clean.csv
sed 's/,O,/,/g' log_O.csv > log_O_clean.csv
sed '/,R,/d' log_X_clean.csv > X.csv
sed '/,R,/d' log_D_clean.csv > D.csv
sed '/,R,/d' log_I_clean.csv > I.csv
sed '/,R,/d' log_P_clean.csv > P.csv
sed '/,R,/d' log_T_clean.csv > T.csv
sed '/,R,/d' log_R_clean.csv > R.csv
sed '/,R,/d' log_O_clean.csv > O.csv

Basic Calibration

Calibration should be done after installing the smart nestbox for the first time and then subsequently as often as possible and at different ambient temperatures (in the starting phase of the project ideally after every battery change). Use the Configuration interface to calculate slope and offset of the linear relationship between weight and ADC value.

Conversion

The relationship between ADC values (the numbers logged and marked as X in the log file) and real weight is a linear function:

ADC_value = slope * weight [g] + offset

Temperature Calibration

To be documented

Continuous Offset Measurement

While the slope of a once calibrated weight sensor should always remain the same (except for temperature variations), the offset of the scale will continuously change, due to fatigue, rain, dirt or other weight getting accumulated on the perch. This is why the smart nestbox makes periodic measurements of the weight at zero, i.e while no weight is being detected. These measurements are logged as O .

A measured weight value (X) should therefore always be corrected by subtracting the last measured offset value (O) from it.

An example of a long-term offset variation is found below (measurement period starts after fresh installation of the perch and lasts for 5 weeks). Note that in this case, the weight gain is quite substantial, potentially due to soaked in water during the measurement period.

The datalogger considers any weight measurement above 100g on the perch as an owl, meaning that it will start the RFID readout and trigger or remain in the X continuous weight measurement mode. If the accumulated offset since the last user input/battery change surpasses 100g, this may continuously trigger this mode and result in decreased battery life.

Offset Measurements after Owl leaves

After an owl leaves, the series of X measurements will always continue for 10 - 20 more measurement points, directly followed by ca. 100 O measurement points. This is in order to determine the offset value that is directly following an applied force to the weight sensor.

It still has to be confirmed with enough gathered data which offset should be taken into account: the one before the measurement or the one right after.

Due to mechanical properties, there is a certain “recovery period” in which the sensor value slowly converges to its default value at rest. See the image on the right for a typical pattern.