Smart Nestbox Data Processing

This page explains the different data types being logged on the smart nestbox and guides the field scientists in their work to interpret the data.

It is very important to understand the underlying state machine that is determining the data being logged on the smart nestbox. A typical measurement event has the following sequence:

1. Standby: no weight measurements are logged, only battery voltage (P) and, once per hour, the no-load weight (or “offset”) is being re-measured (O).
2. Detectiom: a weight of 100g about the initial offset is measured (D). Potentially this is an owl sitting on the perch, but it did not yet get identified. The values measured periodically during this phase cannot directly be transformed into grams, as they are unprecise measurements (short sampling period). It allows though to get an idea about how still the bird was sitting on the perch before it got identified.
3. Identification: the RFID tag of the owl could be read out and is logged (R) up to three times. If it is three times the same value, one can be confident that the ID was read out correctly.
4. Exact Measurement: because the owl was identified, the smart nestbox now switches to a continuous weight measurement mode with hight precision (X) and over multiple seconds averaged values (A, only if values are within a certain tolerance).
5. Identification Re-try: After 10 - 20 measurement points, the RFID tag of the owl is being read out again. If it can still get detected, R is logged. If not, U is logged (as in undefined).
6. Owl leaves: If the weight falls below ca. 100 g again, X measurements continue for 10 - 20 data points, then immediately the weight is labelled as offset (O) for 100 more data points.
7. Turn back to standby mode.

All logging events are first stored in a temprary buffer and periodically written to the SD card into files named LOGXXXXX.txt, where XXXXX is an ascending number. The logging events are represented as a single line in those files. Each of those lines has the following common strucutre:

Logging Character, Timestamp, Data [,complement]

The Logging Character is a predefined ASCII character and encodes the type of data being logged on this line (explained below).

The timestamp is represented as UNIX epoch timestamp, i.e. the number of seconds since 1.1.1970. It is furthermore stored as UTC value, i.e. you need to add one or two hours to convert it into European winter time (CET) resp. summer time (CEST).

When reading out an SD card, the logged data is spread accross a big number of files (LOGXXXX.txt with ascending numbers). For the subsequent data processing, it is necessary to consolidate these files into one single file, and then filter them according to the log character.

On Windows machines, the following software can be used to consolidate the log files in a particular folder: Link to github

On UNIX machines (Mac, Linux), the same task can be done in a terminal by going into the directory of the log files

cd "path-to-log-files"

and executing the commands below:

cat LOG* > log_all.csv 
grep "X," log_all.csv > log_X.csv
grep "D," log_all.csv > log_D.csv
grep "I," log_all.csv > log_I.csv
grep "P," log_all.csv > log_P.csv
grep "T," log_all.csv > log_T.csv
grep "R," log_all.csv > log_R.csv
grep "O," log_all.csv > log_O.csv
sed 's/,X,/,/g' log_X.csv > log_X_clean.csv
sed 's/,D,/,/g' log_D.csv > log_D_clean.csv
sed 's/,I,/,/g' log_I.csv > log_I_clean.csv
sed 's/,P,/,/g' log_P.csv > log_P_clean.csv
sed 's/,T,/,/g' log_T.csv > log_T_clean.csv
sed 's/,R,/,/g' log_R.csv > log_R_clean.csv
sed 's/,O,/,/g' log_O.csv > log_O_clean.csv
sed '/,R,/d' log_X_clean.csv > X.csv
sed '/,R,/d' log_D_clean.csv > D.csv
sed '/,R,/d' log_I_clean.csv > I.csv
sed '/,R,/d' log_P_clean.csv > P.csv
sed '/,R,/d' log_T_clean.csv > T.csv
sed '/,R,/d' log_R_clean.csv > R.csv
sed '/,R,/d' log_O_clean.csv > O.csv

Calibration should be done after installing the smart nestbox for the first time and then subsequently as often as possible and at different ambient temperatures (in the starting phase of the project ideally after every battery change). Use the Configuration interface to calculate slope and offset of the linear relationship between weight and ADC value.

The relationship between ADC values (the numbers logged and marked as X in the log file) and real weight is a linear function:

ADC_value = slope * weight [g] + offset

While the slope of a once calibrated weight sensor should always remain the same (except for temperature variations), the offset of the scale will continuously change, due to fatigue, rain, dirt or other weight getting accumulated on the perch. This is why the smart nestbox makes periodic measurements of the weight at zero, i.e while no weight is being detected. These measurements are logged as O .

A measured weight value (X) should therefore always be corrected by subtracting the last measured offset value (O) from it.

An example of a long-term offset variation is found below (measurement period starts after fresh installation of the perch and lasts for 5 weeks). Note that in this case, the weight gain is quite substantial, potentially due to soaked in water during the measurement period.

After an owl leaves, the series of X measurements will always continue for 10 - 20 more measurement points, directly followed by ca. 100 O measurement points. This is in order to determine the offset value that is directly following an applied force to the weight sensor.

It still has to be confirmed with enough gathered data which offset should be taken into account: the one before the measurement or the one right after.

Due to mechanical properties, there is a certain “recovery period” in which the sensor value slowly converges to its default value at rest. See the image on the right for a typical pattern.