← Back to Projects

Data Pipeline with Tests

beginnerPhase 01 · Programming + DSAPythonPytestPandas

Build a robust local data pipeline that reads messy CSVs, applies an algorithmic transformation, validates data, and writes clean JSON.

What you'll learn

  • Reading, parsing, and cleaning messy tabular data
  • Implementing a core algorithmic logic (e.g., graph search to link related records)
  • Writing strict, robust unit tests to verify algorithmic correctness
  • Outputting validated data into JSON

Architecture

Raw CSV
  → Pandas Data Cleaning (handle nulls, formats)
  → Algorithmic Transformation (DSA Core)
  → Validation (Pydantic / assert checks)
  → Clean JSON Output

Steps

  1. Find a messy real-world CSV dataset (e.g., flight delays, housing data, or sales records).
  2. Write a Python script to ingest the dataset using Pandas or the standard csv library.
  3. Implement a non-trivial algorithmic step: for example, group and link records using a graph algorithm, or calculate a dynamic-programming based optimal sequence.
  4. Crucial Step: Write at least 5 unit tests using pytest specifically to test your core algorithmic function in isolation.
  5. Validate the final output structure and write it to a well-formatted JSON file.
  6. Refactor the script into modular functions (extract, transform, load).

← Back to the roadmap