Pinned Loading

1. DataEngineeringProject-ADF-Databricks-1 DataEngineeringProject-ADF-Databricks-1 Public

   This project is created using Medallion architecture, using Self hosted runtime, ADF, Databricks , Spark SQL, Postgres DB

   Jupyter Notebook

2. Databricks-Kafka-LiveStreaming Databricks-Kafka-LiveStreaming Public

   Stream data from Alpha Vantage and store it in Confluent Kafka and process with Databricks

   Jupyter Notebook

3. Databricks-Streaming-AutoLoader Databricks-Streaming-AutoLoader Public

   Load Stream data to delta lake using Autoloader in Databricks

   Jupyter Notebook

4. ADF-CICD-pipeline ADF-CICD-pipeline Public

   CI CD pipeline to deploy Azure Data Factory project from Dev to Stage. Sample code provided is to flattern json file and store in csv format using Dataflow in ADF

5. KubernetesFlaskMicroservice KubernetesFlaskMicroservice Public

   Python

6. LLM-Notebook-Qualitycheck LLM-Notebook-Qualitycheck Public

   Spark notebooks quality check using Ollama LLM

   Jupyter Notebook