Data Engineering — The Plumbing of Data

What you'll learn

• What data engineers do (and why data scientists depend on them)
• ETL: Extract, Transform, Load
• Data warehouses vs data lakes
• Why "garbage in, garbage out" rules everything

The water system behind the tap 🚰

When a data scientist opens clean data in pandas, someone built the plumbing that delivered it. That someone is a data engineer. Data scientists use data; data engineers move and prepare it at scale — building the pipes so clean water reaches every tap.

The core pattern is ETL (or its modern cousin ELT):

• Extract — pull raw data from many sources: app databases, APIs, log files, spreadsheets, sensors.
• Transform — clean and reshape it: fix missing values, standardize formats (is it "USA", "U.S.", or "United States"?), join sources together, remove duplicates.
• Load — put the clean result somewhere analysts can use it.

That "somewhere" is usually one of two:

• Data warehouse — highly organized, structured storage optimized for analytics queries (like a tidy library with a strict catalog). Tools: Snowflake, BigQuery.
• Data lake — stores raw data of any shape (structured or not) cheaply, to be processed later (like a giant storage unit). Good for huge, varied, or not-yet-defined data.

Why this matters to you: the famous rule is "garbage in, garbage out." The fanciest ML model trained on messy, wrong data produces confident nonsense. Studies say data scientists spend up to 80% of their time on data prep for exactly this reason. A good data engineer makes that faster and more reliable — which is why data engineering is one of the best-paid, most in-demand roles in the whole field.

Data validation: the pipeline's seatbelt 🛡️

Building a pipeline that moves data is only half the job — a good data engineer also builds checks that stop bad data before it spreads. This is called data validation (or "data quality checks"), and it runs automatically every time new data flows through.

Common checks you'll see in real pipelines:

• Schema checks — does this row have the columns we expect, in the right types? (A "price" column suddenly full of text means something upstream broke.)
• Range checks — is a person's age between 0 and 120? Is a percentage between 0 and 100? Impossible values usually mean a bug, not reality.
• Null checks — did a required field (like a customer ID) come through empty? Missing keys silently break every join downstream.
• Uniqueness checks — do we have the same order ID twice? Duplicates quietly inflate totals.
• Freshness checks — did today's data actually arrive today, or is the pipeline stuck showing yesterday's numbers as if they were current?

Why this matters more than it sounds: a broken pipeline that crashes is annoying but safe — someone notices immediately. A broken pipeline that silently loads corrupted data is dangerous — it can feed a wrong number into an executive dashboard or a machine learning model for weeks before anyone notices. Modern tools like Great Expectations or dbt tests let engineers write these checks as code, so every pipeline run automatically verifies its own output before anyone downstream trusts it.

This is the disciplined half of "garbage in, garbage out": it's not enough to hope data is clean, you build automated guards that catch it when it isn't.

Checklist: would this pipeline pass a review? ✅

Before trusting a data pipeline in production, a data engineer walks through checks like these:

1. Does every run validate its output (schema, range, null, uniqueness) before downstream tools touch it?
2. Does it fail loudly, not silently? An alert on failure beats a dashboard quietly showing stale or wrong numbers.
3. Is the source of truth documented? Anyone should be able to trace a number back to where it originated.
4. Can it be re-run safely? Running it twice on the same day shouldn't double-count anything.
5. Is sensitive data (names, emails, payment info) handled with care — masked or access-controlled, not just dumped raw into a lake?

You don't need to write a professional-grade pipeline to internalize this — just get in the habit of asking these questions any time you load a CSV in pandas: do I actually know this data is clean, or am I hoping?

Real-world case study: the Mars Climate Orbiter unit mix-up 🔍

In 1999, NASA lost the Mars Climate Orbiter, a $125 million spacecraft, because of a data pipeline problem — not a dramatic one, a mundane one. One engineering team's software calculated thruster force in pound-seconds (imperial units); NASA's navigation team expected the values in newton-seconds (metric units). The two pieces of software passed numbers between each other with no unit check in between.

The mismatch was small enough on paper — the spacecraft still made it all the way to Mars — but it accumulated over months of navigation corrections until the orbiter approached the planet at the wrong altitude and was destroyed entering the atmosphere.

This is a textbook data engineering failure: two systems exchanged data without a schema/unit validation step confirming they agreed on what the numbers meant. It's the exact same category of bug as a modern pipeline silently loading a "price" column that switched from dollars to cents, or a date column that switched from MM/DD to DD/MM. The fix NASA adopted afterward — and the fix every data engineer builds today — is the same: never trust that two systems agree on units, formats, or meaning; verify it automatically, every time.

Summary

• Data engineers build the pipelines that deliver clean data
• ETL = Extract → Transform → Load
• Warehouse (structured, for analytics) vs Lake (raw, cheap, any shape)
• Garbage in, garbage out — data quality caps everything
• Data validation (schema, range, null, uniqueness, freshness checks) stops bad data before it spreads
• The Mars Climate Orbiter shows how an un-validated unit mismatch between systems can be catastrophic

Next: Business Intelligence — turning all this data into dashboards that drive decisions.