ankitm/data_pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

22-06-2026 2nd commit

Browse Source
This commit is contained in:
Ankit Malik
2026-06-22 13:22:16 +05:30
parent e1bd387849
commit 75fc088c68
2 changed files with 305 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
logs/etl_20260622.log
+265
View File
@@ -926,3 +926,268 @@ KeyError: 'fetch_Promotion'
2026-06-22 13:07:12 | INFO | ================================================================================
2026-06-22 13:07:12 | INFO | Pipeline Completed Successfully
2026-06-22 13:07:12 | INFO | ================================================================================
2026-06-22 13:15:45 | INFO | ================================================================================
2026-06-22 13:15:45 | INFO | Hello from data-move Python data pipeline!
2026-06-22 13:15:45 | INFO | Pipeline Run Date: 2026-06-21
2026-06-22 13:15:45 | INFO | Connecting to databases...
2026-06-22 13:15:46 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000015AA76E09F0>
2026-06-22 13:15:47 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000015AA879C7D0>
2026-06-22 13:15:48 | INFO | Database connections established
2026-06-22 13:15:48 | INFO | Collecting MIDs for: 2026-06-21
2026-06-22 13:15:49 | INFO | Found 569 MIDs
2026-06-22 13:15:49 | INFO | ================================================================================
2026-06-22 13:15:49 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT
2026-06-22 13:15:49 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
2026-06-22 13:15:49 | INFO | Start Fetching data for these 569 MIDs
2026-06-22 13:15:49 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
2026-06-22 13:15:49 | INFO | Fetching data for 569 MIDs
2026-06-22 13:15:51 | INFO | Fetched 1,907 rows from SQL Server
2026-06-22 13:15:51 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!!
2026-06-22 13:15:51 | INFO | Fetched 1907 rows
2026-06-22 13:15:51 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
2026-06-22 13:15:51 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:15:52 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse
2026-06-22 13:15:52 | INFO | SOS_OneApp loaded successfully (1907 rows)
2026-06-22 13:15:52 | INFO | ================================================================================
2026-06-22 13:15:52 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-22 13:15:52 | INFO | Fetching Data from sql server for table-: OQaD ..............
2026-06-22 13:15:52 | INFO | Fetching quiz_empids data for EMPID and Visitid
2026-06-22 13:16:09 | INFO | Fetched 291,166 total empid and visitdate fetched for OQAD from SQL Server
2026-06-22 13:16:10 | INFO | Fetched 290,722 matched empids fetched for OQAD
2026-06-22 13:16:10 | INFO | Exists: True
2026-06-22 13:16:10 | INFO | Path: D:\z\data_move\src\sql\fact\OQaD.sql
2026-06-22 13:16:10 | INFO | Fetching data for 290,722 EMPIDs
2026-06-22 13:16:10 | INFO | Fetching OQaD data for run_date=2026-06-21
2026-06-22 13:16:46 | INFO | fn name is fetch_OQad ------Fetched 0 rows
2026-06-22 13:16:46 | INFO | Fetched 0 rows from SQL Server
2026-06-22 13:16:46 | INFO | Fetched total row -: 0 from sql server for table-:OQaD ...........!!!
2026-06-22 13:16:46 | WARNING | OQaD returned no rows
2026-06-22 13:16:46 | INFO | ================================================================================
2026-06-22 13:16:46 | INFO | Processing Table: Survey | Table type is -: FACT | Based on -mids and operation is used -INSERT
2026-06-22 13:16:46 | INFO | Fetching Data from sql server for table-: Survey ..............
2026-06-22 13:16:46 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
2026-06-22 13:16:46 | INFO | Fetching data for 569 MIDs
2026-06-22 13:16:47 | INFO | Fetched 165 rows from SQL Server
2026-06-22 13:16:47 | INFO | Fetched total row -: 165 from sql server for table-:Survey ...........!!!
2026-06-22 13:16:47 | INFO | Fetched 165 rows
2026-06-22 13:16:47 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
2026-06-22 13:16:47 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:16:47 | INFO | Survey: inserted 165 rows into ClickHouse
2026-06-22 13:16:47 | INFO | Survey loaded successfully (165 rows)
2026-06-22 13:16:47 | INFO | ================================================================================
2026-06-22 13:16:47 | INFO | Processing Table: additional_visibility | Table type is -: FACT | Based on -mids and operation is used -INSERT
2026-06-22 13:16:47 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
2026-06-22 13:16:47 | INFO | Start Fetching data for these 569 MIDs
2026-06-22 13:16:47 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
2026-06-22 13:16:47 | INFO | Fetching data for 569 MIDs
2026-06-22 13:16:47 | INFO | Fetched 1,579 rows from SQL Server
2026-06-22 13:16:47 | INFO | Fetched total row -: 1579 from sql server for table-:additional_visibility ...........!!!
2026-06-22 13:16:47 | INFO | Fetched 1579 rows
2026-06-22 13:16:48 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
2026-06-22 13:16:48 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:16:48 | INFO | additional_visibility: inserted 1,579 rows into ClickHouse
2026-06-22 13:16:48 | INFO | additional_visibility loaded successfully (1579 rows)
2026-06-22 13:16:48 | INFO | ================================================================================
2026-06-22 13:16:48 | INFO | Processing Table: Coverage | Table type is -: FACT | Based on -mids and operation is used -INSERT
2026-06-22 13:16:48 | INFO | Fetching Data from sql server for table-: Coverage ..............
2026-06-22 13:16:48 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
2026-06-22 13:16:48 | INFO | Fetching data for 569 MIDs
2026-06-22 13:17:03 | INFO | Fetched 512 rows from SQL Server
2026-06-22 13:17:03 | INFO | Fetched total row -: 512 from sql server for table-:Coverage ...........!!!
2026-06-22 13:17:03 | INFO | Fetched 512 rows
2026-06-22 13:17:03 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
2026-06-22 13:17:03 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:03 | INFO | Coverage: inserted 512 rows into ClickHouse
2026-06-22 13:17:03 | INFO | Coverage loaded successfully (512 rows)
2026-06-22 13:17:03 | INFO | ================================================================================
2026-06-22 13:17:03 | INFO | Processing Table: Login | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-22 13:17:03 | INFO | Fetching Data from sql server for table-: Login ..............
2026-06-22 13:17:03 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
2026-06-22 13:17:03 | INFO | Fetching data for 569 MIDs
2026-06-22 13:17:05 | INFO | Fetched 454 rows from SQL Server
2026-06-22 13:17:05 | INFO | Fetched total row -: 454 from sql server for table-:Login ...........!!!
2026-06-22 13:17:05 | INFO | Fetched 454 rows
2026-06-22 13:17:05 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _
2026-06-22 13:17:05 | INFO | No delete logic required for Login
2026-06-22 13:17:05 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:05 | INFO | Login: inserted 454 rows into ClickHouse
2026-06-22 13:17:05 | INFO | Login loaded successfully (454 rows)
2026-06-22 13:17:05 | INFO | ================================================================================
2026-06-22 13:17:05 | INFO | Processing Table: Stock_Details | Table type is -: FACT | Based on -mids and operation is used -INSERT
2026-06-22 13:17:05 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
2026-06-22 13:17:05 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
2026-06-22 13:17:05 | INFO | Fetching data for 569 MIDs
2026-06-22 13:17:10 | INFO | Fetched 27,359 rows from SQL Server
2026-06-22 13:17:10 | INFO | Fetched total row -: 27359 from sql server for table-:Stock_Details ...........!!!
2026-06-22 13:17:10 | INFO | Fetched 27359 rows
2026-06-22 13:17:10 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _
2026-06-22 13:17:10 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:11 | INFO | Stock_Details: inserted 27,359 rows into ClickHouse
2026-06-22 13:17:11 | INFO | Stock_Details loaded successfully (27359 rows)
2026-06-22 13:17:11 | INFO | ================================================================================
2026-06-22 13:17:11 | INFO | Processing Table: Attendance | Table type is -: FACT | Based on -run_date and operation is used -DELETE+INSERT
2026-06-22 13:17:11 | INFO | Fetching Data from sql server for table-: Attendance ..............
2026-06-22 13:17:11 | INFO | Fetching Attendance data from 2026-06-06 to 2026-06-21
2026-06-22 13:17:27 | INFO | Fetched 121,639 attendance rows for 544 employees
2026-06-22 13:17:27 | INFO | Fetched total row -: 121639 from sql server for table-:Attendance ...........!!!
2026-06-22 13:17:27 | INFO | Fetched 121639 rows
2026-06-22 13:17:27 | INFO | Truncated table Attendance
2026-06-22 13:17:27 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:28 | INFO | Attendance: inserted 121,639 rows into ClickHouse
2026-06-22 13:17:28 | INFO | Attendance loaded successfully (121639 rows)
2026-06-22 13:17:28 | INFO | ================================================================================
2026-06-22 13:17:28 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-22 13:17:28 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
2026-06-22 13:17:28 | INFO | Start Fetching data for these 569 MIDs
2026-06-22 13:17:28 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
2026-06-22 13:17:28 | INFO | Fetching data for 569 MIDs
2026-06-22 13:17:29 | INFO | Fetched 21,625 rows from SQL Server
2026-06-22 13:17:29 | INFO | Fetched total row -: 21625 from sql server for table-:Journey_Plan ...........!!!
2026-06-22 13:17:29 | INFO | Fetched 21625 rows
2026-06-22 13:17:30 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
2026-06-22 13:17:30 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:30 | INFO | Journey_Plan: inserted 21,625 rows into ClickHouse
2026-06-22 13:17:30 | INFO | Journey_Plan loaded successfully (21625 rows)
2026-06-22 13:17:30 | INFO | ================================================================================
2026-06-22 13:17:30 | INFO | Processing Table: Web_Logins | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-22 13:17:30 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
2026-06-22 13:17:30 | INFO | Start Fetching data for these 569 MIDs
2026-06-22 13:17:30 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
2026-06-22 13:17:30 | INFO | Fetching data for 569 MIDs
2026-06-22 13:17:31 | INFO | Fetched 12 rows from SQL Server
2026-06-22 13:17:31 | INFO | Fetched total row -: 12 from sql server for table-:Web_Logins ...........!!!
2026-06-22 13:17:31 | INFO | Fetched 12 rows
2026-06-22 13:17:31 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
2026-06-22 13:17:31 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:31 | INFO | Web_Logins: inserted 12 rows into ClickHouse
2026-06-22 13:17:31 | INFO | Web_Logins loaded successfully (12 rows)
2026-06-22 13:17:31 | INFO | ================================================================================
2026-06-22 13:17:31 | INFO | Processing Table: Promotion | Table type is -: FACT | Based on -mids and operation is used -INSERT
2026-06-22 13:17:31 | INFO | Fetching Data from sql server for table-: Promotion ..............
2026-06-22 13:17:31 | INFO | Fetching data for 569 MIDs
2026-06-22 13:17:32 | INFO | Fetched 4,223 rows from SQL Server
2026-06-22 13:17:32 | INFO | Fetched total row -: 4223 from sql server for table-:Promotion ...........!!!
2026-06-22 13:17:32 | INFO | Fetched 4223 rows
2026-06-22 13:17:32 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
2026-06-22 13:17:32 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:33 | INFO | Promotion: inserted 4,223 rows into ClickHouse
2026-06-22 13:17:33 | INFO | Promotion loaded successfully (4223 rows)
2026-06-22 13:17:33 | INFO | ================================================================================
2026-06-22 13:17:33 | INFO | Processing Table: PaidVisibility | Table type is -: FACT | Based on -mids and operation is used -INSERT
2026-06-22 13:17:33 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
2026-06-22 13:17:33 | INFO | Start Fetching data for these 569 MIDs
2026-06-22 13:17:33 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
2026-06-22 13:17:33 | INFO | Fetching data for 569 MIDs
2026-06-22 13:17:34 | INFO | Fetched 1,284 rows from SQL Server
2026-06-22 13:17:34 | INFO | Fetched total row -: 1284 from sql server for table-:PaidVisibility ...........!!!
2026-06-22 13:17:34 | INFO | Fetched 1284 rows
2026-06-22 13:17:34 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _
2026-06-22 13:17:34 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:34 | INFO | PaidVisibility: inserted 1,284 rows into ClickHouse
2026-06-22 13:17:34 | INFO | PaidVisibility loaded successfully (1284 rows)
2026-06-22 13:17:34 | INFO | ================================================================================
2026-06-22 13:17:34 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-22 13:17:34 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-22 13:17:34 | INFO | Fetching data from sql server for Master table......
2026-06-22 13:17:34 | INFO | Fetching in progress ....
2026-06-22 13:17:38 | INFO | Fetched 5,998 rows from SQL Server
2026-06-22 13:17:38 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
2026-06-22 13:17:38 | INFO | Fetched 5998 rows
2026-06-22 13:17:38 | INFO | Truncated table Store_Master
2026-06-22 13:17:38 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:39 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
2026-06-22 13:17:39 | INFO | Store_Master loaded successfully (5998 rows)
2026-06-22 13:17:39 | INFO | ================================================================================
2026-06-22 13:17:39 | INFO | Processing Table: SKU_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-22 13:17:39 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
2026-06-22 13:17:39 | INFO | Fetching data from sql server for Master table......
2026-06-22 13:17:39 | INFO | Fetching in progress ....
2026-06-22 13:17:39 | INFO | Fetched 160 rows from SQL Server
2026-06-22 13:17:39 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
2026-06-22 13:17:39 | INFO | Fetched 160 rows
2026-06-22 13:17:39 | INFO | Truncated table SKU_Master
2026-06-22 13:17:39 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:40 | INFO | SKU_Master: inserted 160 rows into ClickHouse
2026-06-22 13:17:40 | INFO | SKU_Master loaded successfully (160 rows)
2026-06-22 13:17:40 | INFO | ================================================================================
2026-06-22 13:17:40 | INFO | Processing Table: display_master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-22 13:17:40 | INFO | Fetching Data from sql server for table-: display_master ..............
2026-06-22 13:17:40 | INFO | Fetching data from sql server for Master table......
2026-06-22 13:17:40 | INFO | Fetching in progress ....
2026-06-22 13:17:40 | INFO | Fetched 135 rows from SQL Server
2026-06-22 13:17:40 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
2026-06-22 13:17:40 | INFO | Fetched 135 rows
2026-06-22 13:17:40 | INFO | Truncated table display_master
2026-06-22 13:17:40 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:40 | INFO | display_master: inserted 135 rows into ClickHouse
2026-06-22 13:17:40 | INFO | display_master loaded successfully (135 rows)
2026-06-22 13:17:40 | INFO | ================================================================================
2026-06-22 13:17:40 | INFO | Processing Table: Employee_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-22 13:17:40 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
2026-06-22 13:17:40 | INFO | Fetching data from sql server for Master table......
2026-06-22 13:17:40 | INFO | Fetching in progress ....
2026-06-22 13:17:41 | INFO | Fetched 2,288 rows from SQL Server
2026-06-22 13:17:41 | INFO | Fetched total row -: 2288 from sql server for table-:Employee_Master ...........!!!
2026-06-22 13:17:41 | INFO | Fetched 2288 rows
2026-06-22 13:17:41 | INFO | Truncated table Employee_Master
2026-06-22 13:17:41 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:41 | INFO | Employee_Master: inserted 2,288 rows into ClickHouse
2026-06-22 13:17:41 | INFO | Employee_Master loaded successfully (2288 rows)
2026-06-22 13:17:41 | INFO | ================================================================================
2026-06-22 13:17:41 | INFO | Processing Table: coverage_remarks | Table type is -: DIMENSION | Based on -reason_id and operation is used -DELETE+INSERT
2026-06-22 13:17:41 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
2026-06-22 13:17:42 | INFO | Fetching data from sql server for Master table......
2026-06-22 13:17:42 | INFO | Fetching in progress ....
2026-06-22 13:17:42 | INFO | Fetched 0 rows from SQL Server
2026-06-22 13:17:42 | INFO | Fetched 0 rows from SQL Server
2026-06-22 13:17:42 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
2026-06-22 13:17:42 | WARNING | coverage_remarks returned no rows
2026-06-22 13:17:42 | INFO | ================================================================================
2026-06-22 13:17:42 | INFO | Processing Table: mapping_storevisibility | Table type is -: BRIDGE | Based on -run_date and operation is used -ONLY_INSERT
2026-06-22 13:17:42 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
2026-06-22 13:17:42 | WARNING | Table 'mapping_storevisibility' does not exist. During collecting store_ids
2026-06-22 13:17:42 | INFO | Fetching data from sql server for Master table......
2026-06-22 13:17:42 | INFO | Fetching in progress ....
2026-06-22 13:17:42 | INFO | Fetched 0 rows from SQL Server
2026-06-22 13:17:42 | INFO | Fetched 0 rows from SQL Server
2026-06-22 13:17:42 | INFO | Fetched total row -: 0 from sql server for table-:mapping_storevisibility ...........!!!
2026-06-22 13:17:42 | WARNING | mapping_storevisibility returned no rows
2026-06-22 13:17:42 | INFO | ================================================================================
2026-06-22 13:17:42 | INFO | Processing Table: Master_VisibilityReason | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
2026-06-22 13:17:42 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason ..............
2026-06-22 13:17:42 | INFO | Fetching data from sql server for Master table......
2026-06-22 13:17:42 | INFO | Fetching in progress ....
2026-06-22 13:17:42 | INFO | Fetched 17 rows from SQL Server
2026-06-22 13:17:42 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!!
2026-06-22 13:17:42 | INFO | Fetched 17 rows
2026-06-22 13:17:43 | INFO | Truncated table Master_VisibilityReason
2026-06-22 13:17:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:43 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
2026-06-22 13:17:43 | INFO | Master_VisibilityReason loaded successfully (17 rows)
2026-06-22 13:17:43 | INFO | ================================================================================
2026-06-22 13:17:43 | INFO | Processing Table: Master_VisibilityDefinition | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
2026-06-22 13:17:43 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition ..............
2026-06-22 13:17:43 | INFO | Fetching data from sql server for Master table......
2026-06-22 13:17:43 | INFO | Fetching in progress ....
2026-06-22 13:17:43 | INFO | Fetched 861 rows from SQL Server
2026-06-22 13:17:43 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!!
2026-06-22 13:17:43 | INFO | Fetched 861 rows
2026-06-22 13:17:43 | INFO | Truncated table Master_VisibilityDefinition
2026-06-22 13:17:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:43 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
2026-06-22 13:17:43 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
2026-06-22 13:17:43 | INFO | ================================================================================
2026-06-22 13:17:43 | INFO | Processing Table: Master_Salesterritorylayer | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
2026-06-22 13:17:43 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer ..............
2026-06-22 13:17:43 | INFO | Fetching data from sql server for Master table......
2026-06-22 13:17:43 | INFO | Fetching in progress ....
2026-06-22 13:17:43 | INFO | Fetched 33 rows from SQL Server
2026-06-22 13:17:43 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!!
2026-06-22 13:17:43 | INFO | Fetched 33 rows
2026-06-22 13:17:43 | INFO | Truncated table Master_Salesterritorylayer
2026-06-22 13:17:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-22 13:17:44 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
2026-06-22 13:17:44 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
2026-06-22 13:17:44 | INFO | ================================================================================
2026-06-22 13:17:44 | INFO | Pipeline Completed Successfully
2026-06-22 13:17:44 | INFO | ================================================================================
main2.py
+40 -2
View File
@@ -14,7 +14,7 @@
from __future__ import annotations
from time import sleep
import sys
from datetime import date, datetime, timedelta
@@ -254,4 +254,42 @@ def main():
if __name__ == "__main__":
main()
for attempt in range(3):
try:
main()
with open("Pipeline_config.yml", "r") as f:
config = yaml.safe_load(f)
config["pipeline"]["last_successful_run_date"] = str(date.today())
with open("Pipeline_config.yml", "w") as f:
yaml.safe_dump(config, f, sort_keys=False)
log.info(
f"Pipeline completed successfully. "
f"last_successful_run_date={date.today()}"
)
break
except Exception as e:
with open("Pipeline_config.yml", "r") as f:
config = yaml.safe_load(f)
config["pipeline"]["run_date"] = str(date.today())
with open("Pipeline_config.yml", "w") as f:
yaml.safe_dump(config, f, sort_keys=False)
if attempt == 2:
raise
log.warning(
f"Pipeline failed. Retry {attempt + 1}/3. Error: {e}"
)
sleep(5)