Parquet round-trip and row-count assertions
A successful Glue run is not the same as a correct output. The job can succeed and still drop half the rows because of a misconfigured window. Read the output back through pyarrow, assert row counts, and validate the schema before you trust the curated layer.
uv run python -c "
import pyarrow.parquet as pq
for name in [\"carrier_analysis\", \"route_analysis\"]:
table = pq.read_table(f\"out/{name}\")
assert table.num_rows > 0, f\"{name} is empty\"
print(f\"{name} schema: {table.schema}\")
print(f\"{name} rows: {table.num_rows}\")
"The smoke test reads the Parquet output and asserts non-zero rows. Same pattern works for production: download, read, assert.
Why pyarrow and not Spark? Reading Parquet through pyarrow uses zero Spark startup cost. A smoke check that takes 800 ms instead of 30 seconds is one a developer will actually run before pushing.
pyarrow reads from S3 directly with pyarrow.fs.S3FileSystem. In a downstream Lambda or a Glue post-processing step, run the same assert and emit a CloudWatch metric. If the row count is below threshold, page the team. The Lakehouse course goes deep on this pattern.
Quiz: Quiz
Loading practice…
AI prompt: Try it: design a row-count threshold
Loading practice…