A practical framework for choosing transformation tools without treating either as a religion.
Development sample: this tutorial outline is illustrative and awaits Ken's editorial review.
Start with the people who maintain it
SQL is a strong default when a transformation is relational and the team can reason about it clearly. PySpark becomes useful when the job needs richer control flow, reusable typed functions, or direct interaction with distributed data structures.
Make the simple path obvious
select customer_id, sum(amount) as lifetime_value
from orders
group by customer_id;The useful question is not which tool is more impressive. It is which version makes correctness, review, operations, and change easiest for the team that owns the pipeline.