Ingest
Four feeds, zero special cases
CRM customers, a product catalog, a lending core, and a payment ledger arrive as separate CSVs with realistically inconsistent shapes and values.
Data engineering portfolio
Financial data,
made legible.
Four messy operational feeds. One guarded Python ETL pipeline. A PostgreSQL star schema you can trust — and inspect down to the last rejected row.
Pipeline successful
0 critical failures · 3 warnings
- Raw rows ingested
- 1,713 from four operational feeds
- Rows quarantined
- 54 rejected with a reason, never lost
- Automated checks
- 24 integrity, counts, and totals
- Warehouse rows
- 3,704 dimensions, facts, and dates
Results from the seeded demo run · 0 critical validation failures
01 The project
Built to show the plumbing,
not just the chart.
Helix Finance is a small, end-to-end data warehouse of a fictional lender. It covers the less glamorous parts of analytics work: cleaning, rejecting, reconciling, and proving the result.
Validate
Quality gates that block bad loads
Uniqueness, foreign keys, row counts, and source-to-warehouse totals are verified before the transaction commits.
Model
A star schema with clear grain
Four conformed dimensions and two facts make loan originations and transaction servicing easy to query separately — and together.
Observe
Rejections are first-class
Every rejected row keeps its source, row number, reason code, and complete raw JSON payload. Nothing disappears into a log file.
Analyze
Decision-ready SQL, audited end to end
Six reporting queries cover portfolio mix, originations, transactions, top customers, loan status, and customer segments.
Ship
Lightweight on purpose
Python, Pandas, PostgreSQL, SQLAlchemy, and Streamlit. One Docker Compose service, no orchestration tax.
02 Architecture
One pipeline.
Five deliberate stages.
A full atomic refresh keeps the demo simple to reason about. The pipeline is intentionally readable, not hidden behind an orchestration framework.
-
01
Extract
Read four source CSVs as strings so natural IDs never lose their shape.
-
02
Clean
Normalize dates, amounts, categories, and IDs. Resolve duplicates and missing values.
-
03
Validate
Check uniqueness, referential integrity, row counts, and monetary reconciliation.
-
04
Load
Replace the star schema atomically through SQLAlchemy. Blocking failures roll back.
-
05
Explore
Inspect run history, rejects, checks, warehouse tables, and analytical reports.
Validation runs before commit.
Read the orchestration
A blocking failure rolls back the replacement, leaving the previous warehouse snapshot intact.
03 Data quality
Not every row makes it in.
That's the point.
Real operational data is messy. The warehouse should make that visible, not paper over it. Rejected rows are quarantined with enough context to fix the source.
- Uniqueness enforced on every natural key
- Referential integrity checked on every fact-to-dimension edge
- Source and warehouse amounts reconciled to the cent
- Warnings distinguished from blocking failures
0
critical failures
3
policy warnings
19
reason codes
The seed data intentionally includes duplicates, missing emails, malformed dates, inconsistent casing, and orphan foreign keys.
04 The warehouse
Small model.
Honest answers.
The star schema keeps the grain explicit. A separate audit area keeps observability out of the analytical path.
4
Conformed dimensions
Date, customer, product, and loan. Stable natural keys, clean surrogate keys.
2
Purpose-built facts
Loan originations at contract grain; transactions at event grain.
3
Open the data dictionary
Audit tables
Runs, rejected rows, and validation results persist across refreshes.
Dimensions
DIM
dim_customer
customer_key
DIM
dim_product
product_key
DIM
dim_loan
loan_key
DIM
dim_date
date_key · YYYYMMDD
surrogate keys
Facts
FACT
fact_loan_originations
one row per loan
FACT
fact_transactions
one row per event
audit / append-only
etl_pipeline_runs
etl_rejected_rows
etl_validation_results
05 Quickstart
Run it in
under a minute.
The repository includes deterministic seed data, a PostgreSQL Compose service, and the complete ETL pipeline. No API keys. No external services. All data is synthetic.
- Docker Compose
- Python 3.11+
- Seed data included
git clone https://github.com/Mo-Shakib/Helix-Finance.git
cd Helix-Finance
cp .env.example .env
docker compose up -d
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python -m finance_dwh.pipeline
streamlit run app.py06 The source
Read the pipeline.
Question the schema.
Every transformation, check, rejected row, and SQL report is in the repository. The whole build is designed to be read.