External data collection and quality checks · Learning

Topics covered

Check both the HTML structure and extraction rules so that missing fields on a page do not cause other records to be discarded.

Overall workflow

JSON and HTML responses → field and unit checks → valid and rejected data → CSV storage

Successful collection and value quality

Receiving JSON does not make it ready for analysis. Check field presence, data types, physical ranges, and units in order. Replacing missing values with actual 0 values can distort means and alarm thresholds.

Boundaries to preserve when the input format changes

Separate collection, cleaning, and storage into functions to limit the impact of API field or HTML structure changes. Record reasons for rejected rows, and check that numbers become strings during a CSV round trip.

Worked example

The sensor response contains device, timestamp, and temperature_c. The second row lacks a temperature, and the third has a numeric string.

import json
payload='[{"device":"a","timestamp":0,"temperature_c":21.2},{"device":"a","timestamp":1},{"device":"b","timestamp":0,"temperature_c":"22.1"}]'
rows=json.loads(payload)
def clean(row):
    if 'temperature_c' not in row:return None,'missing temperature'
    try:return dict(row,temperature_c=float(row['temperature_c'])),'ok'
    except (ValueError,TypeError):return None,'invalid temperature'
for row in rows:print(clean(row))

({'device': 'a', 'timestamp': 0, 'temperature_c': 21.2}, 'ok')
(None, 'missing temperature')
({'device': 'b', 'timestamp': 0, 'temperature_c': 22.1}, 'ok')

Reference example

View more examples

Reference materials