Problem statement
- Keeping track of the houses an user is interested gets tiring over time.
- historical price tracking and drops is behind a pay wall.
- As a data nerd, I found the whole search experience boring and wanted to reimagine it using some custom search preferences.
Approach
Ingestion
Data from platform vendors is collected while respecting their terms of service on data scraping, and is ingested as parquet onto a lakehouse.
Modeling
To start with, this pulls data from one source (Funda listings) and there is always a risk of them retracting the current scraping method this project uses. So to keep it platform-agnostic and make adding and removing data sources scalable without disrupting the project, data vault methodology is used for integration, on top of which a dimensional model is built out. That model then acts as the foundation for a semantic layer powering both the final human-facing user interface and AI-assisted search.
Status
Currently the scraping and integration of data into a vault is worked out, which can be found in the GitHub repo. This project is being actively built out and be ready for usage by 15th of July.