NiiL

External data collection and quality checks

English translation

Check both the HTML structure and extraction rules so that missing fields on a page do not cause other records to be discarded.

Before practice

01 Can you use a value immediately if it converts to a number?

Description

Check sensor range, units, and missing values as well as format.

02 Is it safe to fill missing measurements with 0?

Description

Failed observations and actual 0 values are different situations, so the rule must reflect the meaning of the data.

Practice

Preview practice notebookDownload practice notebook

1. Select the required values from JSON

Starting conditions

The sensor response contains device, timestamp, and temperature_c. The second row lacks a temperature, and the third has a numeric string.

Procedure

  1. Read the provided JSON and check each row's fields and data types.
  2. Distinguish successful collection from values suitable for analysis.

Results to retain

Items to record and points to check
Row
Original value and data type
Cleaned value
Reason for handling

Points to check

  • Did you avoid treating missing temperatures as actual 0 degrees?
  • Did you record reasons for conversion failures instead of silently discarding them?

2. When the response structure changes

Starting conditions

Test additional inputs temperature_c=null, "error", 1000, and a new field unit="F".

Procedure

  1. Process the four rows of the same variants. First check numeric conversion results; for conversion failures, record both the original value and the error reason.
  2. Make the units that can be interpreted as Celsius and the sensor's valid temperature range arguments to the validation function. State the chosen units, range, and whether boundaries are included, and distinguish out-of-range rows from rows with different units. Since sensor specifications are not provided, label this range as your chosen validation condition.
  3. For each row, record the input value, unit, converted value, acceptance, and rejection reason. Collect valid and rejected rows separately, and state what additional information is needed to determine whether the sensor has failed.

Results to retain

Items to record and points to check
Input
Format check
Range and unit checks
Processing

Points to check

  • Did you check the sensor range even when the format was numeric?
  • Did you avoid arbitrarily converting units whose meaning you do not know?

3. Compare with data available only as HTML

Starting conditions

Compare a situation with only an HTML table and no API response. Keep the existing selector examples as reference materials.

Procedure

  1. Give one cause each of broken API fields and HTML selectors.
  2. Separate collection, cleaning, and storage into three functions, and define their inputs and outputs.
  3. Save the cleaned temperature data as CSV and read it back to check that the values and missing-value handling are preserved.

Results to retain

Items to record and points to check
Step
Input
Output
Errors to check

Points to check

  • Did you test the handling rules with fixed responses without repeatedly requesting external sites?
  • Did you identify terms of use and collection intervals as items to check before making a real connection?