External data collection and quality checks

Learning objectives

Check both the HTML structure and extraction rules so that missing fields on a page do not cause other records to be discarded.

Select the required values from JSON

Starting conditions

The sensor response contains device, timestamp, and temperature_c. The second row lacks a temperature, and the third has a numeric string.

Procedure

1. Read the provided JSON and check each row's fields and data types.
2. Distinguish successful collection from values suitable for analysis.

Results to retain

• A table of valid, missing-value, and format conversion cases.

import json
payload='[{"device":"a","timestamp":0,"temperature_c":21.2},{"device":"a","timestamp":1},{"device":"b","timestamp":0,"temperature_c":"22.1"}]'
rows=json.loads(payload)
def clean(row):
    if 'temperature_c' not in row:return None,'missing temperature'
    try:return dict(row,temperature_c=float(row['temperature_c'])),'ok'
    except (ValueError,TypeError):return None,'invalid temperature'
for row in rows:print(clean(row))

Record results and self-check

Row:

Original value and data type:

Cleaned value:

Reason for handling:

Points to check

• Did you avoid treating missing temperatures as actual 0 degrees?
• Did you record reasons for conversion failures instead of silently discarding them?

When the response structure changes

Starting conditions

Test additional inputs temperature_c=null, “error”, 1000, and a new field unit=“F”.

Procedure

1. Process the four rows of the same variants. First check numeric conversion results; for conversion failures, record both the original value and the error reason.
2. Make the units that can be interpreted as Celsius and the sensor's valid temperature range arguments to the validation function. State the chosen units, range, and whether boundaries are included, and distinguish out-of-range rows from rows with different units. Since sensor specifications are not provided, label this range as your chosen validation condition.
3. For each row, record the input value, unit, converted value, acceptance, and rejection reason. Collect valid and rejected rows separately, and state what additional information is needed to determine whether the sensor has failed.

Results to retain

• A validation function and an error case table.

variants=[dict(device='a',timestamp=2,temperature_c=None),dict(device='a',timestamp=3,temperature_c='error'),dict(device='a',timestamp=4,temperature_c=1000),dict(device='a',timestamp=5,temperature_c=70,unit='F')]
# TODO: Extend clean with each validation stage.

Record results and self-check

Input:

Format check:

Range and unit checks:

Handling:

Points to check

• Did you check the sensor range even when the format was numeric?
• Did you avoid arbitrarily converting units whose meaning you do not know?

Compare with data available only as HTML

Starting conditions

Compare a situation with only an HTML table and no API response. Keep the existing selector examples as reference materials.

Procedure

1. Give one cause each of broken API fields and HTML selectors.
2. Separate collection, cleaning, and storage into three functions, and define their inputs and outputs.
3. Save the cleaned temperature data as CSV and read it back to check that the values and missing-value handling are preserved.

Results to retain

• A collection pipeline diagram and a CSV round-trip check.

import csv,io
valid=[clean(row)[0] for row in rows if clean(row)[0] is not None]
stream=io.StringIO();writer=csv.DictWriter(stream,fieldnames=['device','timestamp','temperature_c'])
writer.writeheader();writer.writerows(valid);stream.seek(0)
print(list(csv.DictReader(stream)))

Record results and self-check

Step:

Input:

Output:

Errors to check:

Points to check

• Did you test the handling rules with fixed responses without repeatedly requesting external sites?
• Did you identify terms of use and collection intervals as items to check before making a real connection?