ankitm/data_pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

14-06-26 1st comit

Browse Source
This commit is contained in:
Ankit Malik
2026-06-15 12:44:13 +05:30
parent 418a8847b2
commit 166a2a1d19
14 changed files with 1981 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files
logs/etl_20260612.log
+40
View File
@@ -982,3 +982,43 @@
2026-06-12 19:04:53 | INFO | Fetched 5,984 stores
2026-06-12 19:04:53 | INFO | Table Store_Master truncated successfully.
2026-06-12 19:04:53 | INFO | Truncate a ClickHouse table - Store_Master
2026-06-12 19:07:43 | INFO | ================================================================================
2026-06-12 19:07:43 | INFO | Hello from data-move Python data pipeline !
2026-06-12 19:07:43 | INFO | Data-pipeline running Date is -:2026-06-11
2026-06-12 19:07:43 | INFO | connecting with both db servers sql-serveras well as clickhouse DB
2026-06-12 19:07:44 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001148C85BE30>
2026-06-12 19:07:45 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001148DA15BD0>
2026-06-12 19:07:46 | INFO | Both databases connected successfully
2026-06-12 19:07:46 | INFO | Collecting MIDs for: 2026-06-11
2026-06-12 19:07:46 | INFO | Found 818 MIDs
2026-06-12 19:07:46 | INFO | ================================================================================
2026-06-12 19:07:46 | INFO | TABLE=SOS_OneApp | TYPE=FACT | OPERATION=INSERT
2026-06-12 19:07:46 | INFO | Fetching data for 818 MIDs
2026-06-12 19:07:47 | INFO | Fetched 3,677 rows from SQL Server
2026-06-12 19:07:47 | INFO | Delete command submitted successfully.
2026-06-12 19:07:47 | INFO | SOS_OneApp: inserted 3,677 rows into ClickHouse
2026-06-12 19:07:47 | INFO | ================================================================================
2026-06-12 19:07:47 | INFO | TABLE=OQaD | TYPE=FACT | OPERATION=INSERT
2026-06-12 19:07:47 | INFO | Fetching OQaD data for 818 MIDs
2026-06-12 19:07:50 | INFO | Fetched 464 rows
2026-06-12 19:09:36 | INFO | ================================================================================
2026-06-12 19:09:36 | INFO | Hello from data-move Python data pipeline !
2026-06-12 19:09:36 | INFO | Data-pipeline running Date is -:2026-06-11
2026-06-12 19:09:36 | INFO | connecting with both db servers sql-serveras well as clickhouse DB
2026-06-12 19:09:37 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001732452BCD0>
2026-06-12 19:09:38 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000173256E1BD0>
2026-06-12 19:09:39 | INFO | Both databases connected successfully
2026-06-12 19:09:39 | INFO | Collecting MIDs for: 2026-06-11
2026-06-12 19:09:39 | INFO | Found 818 MIDs
2026-06-12 19:09:39 | INFO | ================================================================================
2026-06-12 19:09:39 | INFO | TABLE=SOS_OneApp | TYPE=FACT | OPERATION=INSERT
2026-06-12 19:09:39 | INFO | Fetching data for 818 MIDs
2026-06-12 19:09:40 | INFO | Fetched 3,677 rows from SQL Server
2026-06-12 19:09:40 | INFO | Deleting rows from SOS_OneApp
2026-06-12 19:09:40 | INFO | Delete command submitted successfully.
2026-06-12 19:09:41 | INFO | SOS_OneApp: inserted 3,677 rows into ClickHouse
2026-06-12 19:09:41 | INFO | ================================================================================
2026-06-12 19:09:41 | INFO | TABLE=OQaD | TYPE=FACT | OPERATION=INSERT
2026-06-12 19:09:41 | INFO | Fetching OQaD data for 818 MIDs
2026-06-12 19:09:44 | INFO | Fetched 464 rows
2026-06-12 19:09:44 | INFO | Deleting rows from OQaD
logs/etl_20260615.log
+1103
View File
File diff suppressed because one or more lines are too long