Ankit Malik
2188 lines
190 KiB
Plaintext
2188 lines
190 KiB
Plaintext
2026-06-23 11:59:53 | INFO | ================================================================================
|
|
2026-06-23 11:59:53 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 11:59:53 | INFO | Pipeline Run Date: 2023-01-01
|
|
2026-06-23 11:59:53 | INFO | Connecting to databases...
|
|
2026-06-23 11:59:55 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000023BE73F8CB0>
|
|
2026-06-23 11:59:58 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000023BE84B07D0>
|
|
2026-06-23 11:59:59 | INFO | Database connections established
|
|
2026-06-23 11:59:59 | INFO | Collecting MIDs for: 2023-01-01
|
|
2026-06-23 11:59:59 | INFO | Found 0 MIDs
|
|
2026-06-23 12:00:00 | INFO | ================================================================================
|
|
2026-06-23 12:00:00 | INFO | Processing Table-:SOS_OneApp | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 12:00:00 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-23 12:00:00 | WARNING | No MIDs — nothing to fetch.
|
|
2026-06-23 12:00:00 | INFO | Fetched total row -: 0 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-23 12:00:00 | WARNING | SOS_OneApp returned no rows
|
|
2026-06-23 12:00:00 | INFO | ================================================================================
|
|
2026-06-23 12:00:00 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 12:00:00 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-23 12:00:01 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-23 12:00:09 | INFO | Fetched 0 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-23 12:00:09 | WARNING | Table 'OQaD' does not exist.
|
|
2026-06-23 12:00:09 | WARNING | OQaD Matched df in OQaD returned no rows
|
|
2026-06-23 12:00:09 | INFO | Fetched 1 matched empids fetched for OQAD
|
|
2026-06-23 12:00:09 | INFO | Exists: True
|
|
2026-06-23 12:00:09 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-23 12:00:09 | INFO | Fetching data for 1 EMPIDs
|
|
2026-06-23 12:00:09 | INFO | Fetching OQaD data for run_date=2023-01-01
|
|
2026-06-23 12:00:10 | INFO | fn name is fetch_OQad ------Fetched 0 rows
|
|
2026-06-23 12:00:10 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 12:00:10 | INFO | Fetched total row -: 0 from sql server for table-:OQaD ...........!!!
|
|
2026-06-23 12:00:10 | WARNING | OQaD returned no rows
|
|
2026-06-23 12:00:10 | INFO | ================================================================================
|
|
2026-06-23 12:00:10 | INFO | Processing Table-:Survey | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 12:00:10 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-23 12:00:10 | WARNING | No MIDs — nothing to fetch.
|
|
2026-06-23 12:00:10 | INFO | Fetched total row -: 0 from sql server for table-:Survey ...........!!!
|
|
2026-06-23 12:00:10 | WARNING | Survey returned no rows
|
|
2026-06-23 12:00:10 | INFO | ================================================================================
|
|
2026-06-23 12:00:10 | INFO | Processing Table-:additional_visibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 12:00:10 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-23 12:00:10 | WARNING | No MIDs — nothing to fetch.
|
|
2026-06-23 12:00:10 | INFO | Fetched total row -: 0 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-23 12:00:10 | WARNING | additional_visibility returned no rows
|
|
2026-06-23 12:00:10 | INFO | ================================================================================
|
|
2026-06-23 12:00:10 | INFO | Processing Table-:Coverage | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 12:00:10 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-23 12:00:10 | WARNING | No MIDs — nothing to fetch.
|
|
2026-06-23 12:00:10 | INFO | Fetched total row -: 0 from sql server for table-:Coverage ...........!!!
|
|
2026-06-23 12:00:10 | WARNING | Coverage returned no rows
|
|
2026-06-23 12:00:10 | INFO | ================================================================================
|
|
2026-06-23 12:00:10 | INFO | Processing Table-:Login | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 12:00:10 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-23 12:00:10 | INFO | Start Fetching data for these 0 MIDs or based on this date 2023-01-01
|
|
2026-06-23 12:00:10 | INFO | Fetching data for 0 MIDs
|
|
2026-06-23 12:00:12 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 12:00:12 | INFO | Fetched total row -: 0 from sql server for table-:Login ...........!!!
|
|
2026-06-23 12:00:12 | WARNING | Login returned no rows
|
|
2026-06-23 12:00:12 | INFO | ================================================================================
|
|
2026-06-23 12:00:12 | INFO | Processing Table-:Stock_Details | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 12:00:12 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-23 12:00:12 | WARNING | No MIDs — nothing to fetch.
|
|
2026-06-23 12:00:12 | INFO | Fetched total row -: 0 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-23 12:00:12 | WARNING | Stock_Details returned no rows
|
|
2026-06-23 12:00:12 | INFO | ================================================================================
|
|
2026-06-23 12:00:12 | INFO | Processing Table-:Attendance | Table type -:FACT | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 12:00:12 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-23 12:00:12 | INFO | Fetching Attendance data from 2022-12-17 to 2023-01-01
|
|
2026-06-23 12:00:16 | INFO | Fetched 5,481 attendance rows for 347 employees
|
|
2026-06-23 12:00:16 | INFO | Fetched total row -: 5481 from sql server for table-:Attendance ...........!!!
|
|
2026-06-23 12:00:16 | INFO | Fetched 5481 rows
|
|
2026-06-23 12:00:16 | INFO | Creating table Attendance
|
|
2026-06-23 12:00:16 | INFO | Table ready: Attendance
|
|
2026-06-23 12:00:16 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 12:00:17 | INFO | Attendance: inserted 5,481 rows into ClickHouse
|
|
2026-06-23 12:00:17 | INFO | Attendance loaded successfully (5481 rows)
|
|
2026-06-23 12:00:17 | INFO | ================================================================================
|
|
2026-06-23 12:00:17 | INFO | Processing Table-:Journey_Plan | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 12:00:17 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-23 12:00:17 | WARNING | No MIDs — nothing to fetch.
|
|
2026-06-23 12:00:17 | INFO | Fetched total row -: 0 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-23 12:00:17 | WARNING | Journey_Plan returned no rows
|
|
2026-06-23 12:00:17 | INFO | ================================================================================
|
|
2026-06-23 12:00:17 | INFO | Processing Table-:Web_Logins | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 12:00:17 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-23 12:00:17 | WARNING | No MIDs — nothing to fetch.
|
|
2026-06-23 12:00:17 | INFO | Fetched total row -: 0 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-23 12:00:17 | WARNING | Web_Logins returned no rows
|
|
2026-06-23 12:00:17 | INFO | ================================================================================
|
|
2026-06-23 12:00:17 | INFO | Processing Table-:Promotion | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 12:00:17 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-23 12:00:17 | WARNING | No MIDs — nothing to fetch.
|
|
2026-06-23 12:00:17 | INFO | Fetched total row -: 0 from sql server for table-:Promotion ...........!!!
|
|
2026-06-23 12:00:17 | WARNING | Promotion returned no rows
|
|
2026-06-23 12:00:17 | INFO | ================================================================================
|
|
2026-06-23 12:00:17 | INFO | Processing Table-:PaidVisibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 12:00:17 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
|
|
2026-06-23 12:00:17 | WARNING | No MIDs — nothing to fetch.
|
|
2026-06-23 12:00:17 | INFO | Fetched total row -: 0 from sql server for table-:PaidVisibility ...........!!!
|
|
2026-06-23 12:00:17 | WARNING | PaidVisibility returned no rows
|
|
2026-06-23 12:00:17 | INFO | ================================================================================
|
|
2026-06-23 12:00:17 | INFO | Processing Table-:Store_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 12:00:17 | INFO | Fetching Data from sql server for table-: Store_Master ..............
|
|
2026-06-23 12:00:17 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 12:00:17 | INFO | Fetching in progress ....
|
|
2026-06-23 12:00:20 | INFO | Fetched 5,998 rows from SQL Server
|
|
2026-06-23 12:00:20 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
|
|
2026-06-23 12:00:20 | INFO | Fetched 5998 rows
|
|
2026-06-23 12:00:20 | INFO | Creating table Store_Master
|
|
2026-06-23 12:00:21 | INFO | Table ready: Store_Master
|
|
2026-06-23 12:00:21 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 12:00:21 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
|
|
2026-06-23 12:00:21 | INFO | Store_Master loaded successfully (5998 rows)
|
|
2026-06-23 12:00:21 | INFO | ================================================================================
|
|
2026-06-23 12:00:21 | INFO | Processing Table-:SKU_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 12:00:21 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
|
|
2026-06-23 12:00:21 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 12:00:21 | INFO | Fetching in progress ....
|
|
2026-06-23 12:00:21 | INFO | Fetched 160 rows from SQL Server
|
|
2026-06-23 12:00:21 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
|
|
2026-06-23 12:00:21 | INFO | Fetched 160 rows
|
|
2026-06-23 12:00:21 | INFO | Creating table SKU_Master
|
|
2026-06-23 12:00:22 | INFO | Table ready: SKU_Master
|
|
2026-06-23 12:00:22 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 12:00:22 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-23 12:00:22 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-23 12:00:22 | INFO | ================================================================================
|
|
2026-06-23 12:00:22 | INFO | Processing Table-:display_master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 12:00:22 | INFO | Fetching Data from sql server for table-: display_master ..............
|
|
2026-06-23 12:00:22 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 12:00:22 | INFO | Fetching in progress ....
|
|
2026-06-23 12:00:22 | INFO | Fetched 135 rows from SQL Server
|
|
2026-06-23 12:00:22 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
|
|
2026-06-23 12:00:22 | INFO | Fetched 135 rows
|
|
2026-06-23 12:00:22 | INFO | Creating table display_master
|
|
2026-06-23 12:00:22 | INFO | Table ready: display_master
|
|
2026-06-23 12:00:22 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 12:00:22 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-23 12:00:22 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-23 12:00:22 | INFO | ================================================================================
|
|
2026-06-23 12:00:22 | INFO | Processing Table-:Employee_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 12:00:22 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
|
|
2026-06-23 12:00:22 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 12:00:22 | INFO | Fetching in progress ....
|
|
2026-06-23 12:00:23 | INFO | Fetched 2,289 rows from SQL Server
|
|
2026-06-23 12:00:23 | INFO | Fetched total row -: 2289 from sql server for table-:Employee_Master ...........!!!
|
|
2026-06-23 12:00:23 | INFO | Fetched 2289 rows
|
|
2026-06-23 12:00:23 | INFO | Creating table Employee_Master
|
|
2026-06-23 12:00:23 | INFO | Table ready: Employee_Master
|
|
2026-06-23 12:00:23 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 12:00:24 | INFO | Employee_Master: inserted 2,289 rows into ClickHouse
|
|
2026-06-23 12:00:24 | INFO | Employee_Master loaded successfully (2289 rows)
|
|
2026-06-23 12:00:24 | INFO | ================================================================================
|
|
2026-06-23 12:00:24 | INFO | Processing Table-:coverage_remarks | Table type -:DIMENSION | fetcht by-:reason_id | operation-:DELETE+INSERT
|
|
2026-06-23 12:00:24 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
|
|
2026-06-23 12:00:24 | WARNING | Table 'coverage_remarks' does not exist. During collecting reason_ids
|
|
2026-06-23 12:00:24 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 12:00:24 | INFO | Fetching in progress ....
|
|
2026-06-23 12:00:24 | INFO | Fetched 29 rows from SQL Server
|
|
2026-06-23 12:00:24 | INFO | Fetched 29 rows from SQL Server
|
|
2026-06-23 12:00:24 | INFO | Fetched total row -: 29 from sql server for table-:coverage_remarks ...........!!!
|
|
2026-06-23 12:00:24 | INFO | Fetched 29 rows
|
|
2026-06-23 12:00:25 | INFO | Creating table coverage_remarks
|
|
2026-06-23 12:00:25 | INFO | Table ready: coverage_remarks
|
|
2026-06-23 12:00:25 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 12:00:25 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-23 12:00:25 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-23 12:00:25 | INFO | ================================================================================
|
|
2026-06-23 12:00:25 | INFO | Processing Table-:mapping_storevisibility | Table type -:BRIDGE | fetcht by-:run_date | operation-:ONLY_INSERT
|
|
2026-06-23 12:00:25 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
|
|
2026-06-23 12:00:25 | WARNING | Table 'mapping_storevisibility' does not exist. During collecting store_ids
|
|
2026-06-23 12:00:25 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 12:00:25 | INFO | Fetching in progress ....
|
|
2026-06-23 12:00:25 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 12:00:25 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 12:00:25 | INFO | Fetched total row -: 0 from sql server for table-:mapping_storevisibility ...........!!!
|
|
2026-06-23 12:00:25 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-23 12:00:25 | INFO | ================================================================================
|
|
2026-06-23 12:00:25 | INFO | Processing Table-:Master_VisibilityReason | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 12:00:25 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason ..............
|
|
2026-06-23 12:00:25 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 12:00:25 | INFO | Fetching in progress ....
|
|
2026-06-23 12:00:25 | INFO | Fetched 17 rows from SQL Server
|
|
2026-06-23 12:00:25 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!!
|
|
2026-06-23 12:00:25 | INFO | Fetched 17 rows
|
|
2026-06-23 12:00:26 | INFO | Creating table Master_VisibilityReason
|
|
2026-06-23 12:00:26 | INFO | Table ready: Master_VisibilityReason
|
|
2026-06-23 12:00:26 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 12:00:26 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-23 12:00:26 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-23 12:00:26 | INFO | ================================================================================
|
|
2026-06-23 12:00:26 | INFO | Processing Table-:Master_VisibilityDefinition | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 12:00:26 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition ..............
|
|
2026-06-23 12:00:26 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 12:00:26 | INFO | Fetching in progress ....
|
|
2026-06-23 12:00:26 | INFO | Fetched 861 rows from SQL Server
|
|
2026-06-23 12:00:26 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!!
|
|
2026-06-23 12:00:26 | INFO | Fetched 861 rows
|
|
2026-06-23 12:00:26 | INFO | Creating table Master_VisibilityDefinition
|
|
2026-06-23 12:00:26 | INFO | Table ready: Master_VisibilityDefinition
|
|
2026-06-23 12:00:26 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 12:00:26 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-23 12:00:26 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-23 12:00:26 | INFO | ================================================================================
|
|
2026-06-23 12:00:26 | INFO | Processing Table-:Master_Salesterritorylayer | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 12:00:26 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer ..............
|
|
2026-06-23 12:00:26 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 12:00:26 | INFO | Fetching in progress ....
|
|
2026-06-23 12:00:27 | INFO | Fetched 33 rows from SQL Server
|
|
2026-06-23 12:00:27 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!!
|
|
2026-06-23 12:00:27 | INFO | Fetched 33 rows
|
|
2026-06-23 12:00:27 | INFO | Creating table Master_Salesterritorylayer
|
|
2026-06-23 12:00:27 | INFO | Table ready: Master_Salesterritorylayer
|
|
2026-06-23 12:00:27 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 12:00:27 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-23 12:00:27 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-23 12:00:27 | INFO | ================================================================================
|
|
2026-06-23 12:00:27 | INFO | Pipeline Completed Successfully
|
|
2026-06-23 12:00:27 | INFO | ================================================================================
|
|
2026-06-23 12:00:27 | INFO | Pipeline completed successfully. last_successful_run_date=2026-06-23
|
|
2026-06-23 16:01:09 | INFO | Pipeline Start Date: 2026-06-22
|
|
2026-06-23 16:01:09 | INFO | ================================================================================
|
|
2026-06-23 16:01:09 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 16:01:09 | INFO | Pipeline Run Date: 2026-06-22
|
|
2026-06-23 16:01:09 | INFO | Connecting to databases...
|
|
2026-06-23 16:01:10 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001EB528B49F0>
|
|
2026-06-23 16:01:13 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001EB539687D0>
|
|
2026-06-23 16:01:13 | INFO | Database connections established
|
|
2026-06-23 16:01:13 | INFO | Collecting MIDs for: 2026-06-22
|
|
2026-06-23 16:01:14 | INFO | Found 882 MIDs
|
|
2026-06-23 16:01:14 | INFO | ================================================================================
|
|
2026-06-23 16:01:14 | INFO | Processing Table-:SOS_OneApp | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:01:14 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-23 16:01:14 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:01:14 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:01:14 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:01:25 | INFO | Fetched 3,888 rows from SQL Server
|
|
2026-06-23 16:01:25 | INFO | Fetched total row -: 3888 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-23 16:01:25 | INFO | Fetched 3888 rows
|
|
2026-06-23 16:01:26 | INFO | Creating table SOS_OneApp
|
|
2026-06-23 16:01:26 | INFO | Table ready: SOS_OneApp
|
|
2026-06-23 16:01:26 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:01:26 | INFO | SOS_OneApp: inserted 3,888 rows into ClickHouse
|
|
2026-06-23 16:01:26 | INFO | SOS_OneApp loaded successfully (3888 rows)
|
|
2026-06-23 16:01:26 | INFO | ================================================================================
|
|
2026-06-23 16:01:26 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:01:26 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-23 16:01:27 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-23 16:02:05 | INFO | Fetched 291,599 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-23 16:02:05 | WARNING | Table 'OQaD' does not exist.
|
|
2026-06-23 16:02:05 | WARNING | OQaD Matched df in OQaD returned no rows
|
|
2026-06-23 16:02:05 | INFO | Fetched 1 matched empids fetched for OQAD
|
|
2026-06-23 16:02:05 | INFO | Exists: True
|
|
2026-06-23 16:02:05 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-23 16:02:05 | INFO | Fetching data for 1 EMPIDs
|
|
2026-06-23 16:02:05 | INFO | Fetching OQaD data for run_date=2026-06-22
|
|
2026-06-23 16:08:39 | INFO | Pipeline Start Date: 2026-06-22
|
|
2026-06-23 16:08:39 | INFO | ================================================================================
|
|
2026-06-23 16:08:39 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 16:08:39 | INFO | Pipeline Run Date: 2026-06-22
|
|
2026-06-23 16:08:39 | INFO | Connecting to databases...
|
|
2026-06-23 16:08:40 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B6FA530730>
|
|
2026-06-23 16:08:41 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B6FB6447D0>
|
|
2026-06-23 16:08:41 | INFO | Database connections established
|
|
2026-06-23 16:08:41 | INFO | Collecting MIDs for: 2026-06-22
|
|
2026-06-23 16:08:42 | INFO | Found 882 MIDs
|
|
2026-06-23 16:08:42 | INFO | ================================================================================
|
|
2026-06-23 16:08:42 | INFO | Processing Table-:SOS_OneApp | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:08:42 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-23 16:08:42 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:08:42 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:08:42 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:08:52 | INFO | Fetched 3,888 rows from SQL Server
|
|
2026-06-23 16:08:52 | INFO | Fetched total row -: 3888 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-23 16:08:52 | INFO | Fetched 3888 rows
|
|
2026-06-23 16:08:52 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-23 16:08:52 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:08:52 | INFO | SOS_OneApp: inserted 3,888 rows into ClickHouse
|
|
2026-06-23 16:08:52 | INFO | SOS_OneApp loaded successfully (3888 rows)
|
|
2026-06-23 16:08:52 | INFO | ================================================================================
|
|
2026-06-23 16:08:52 | INFO | Processing Table-:Survey | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:08:52 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-23 16:08:52 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:08:52 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:08:53 | INFO | Fetched 152 rows from SQL Server
|
|
2026-06-23 16:08:53 | INFO | Fetched total row -: 152 from sql server for table-:Survey ...........!!!
|
|
2026-06-23 16:08:53 | INFO | Fetched 152 rows
|
|
2026-06-23 16:08:53 | INFO | Creating table Survey
|
|
2026-06-23 16:08:53 | INFO | Table ready: Survey
|
|
2026-06-23 16:08:53 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:08:53 | INFO | Survey: inserted 152 rows into ClickHouse
|
|
2026-06-23 16:08:53 | INFO | Survey loaded successfully (152 rows)
|
|
2026-06-23 16:08:53 | INFO | ================================================================================
|
|
2026-06-23 16:08:53 | INFO | Processing Table-:additional_visibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:08:53 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-23 16:08:53 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:08:53 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:08:53 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:08:54 | INFO | Fetched 2,136 rows from SQL Server
|
|
2026-06-23 16:08:54 | INFO | Fetched total row -: 2136 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-23 16:08:54 | INFO | Fetched 2136 rows
|
|
2026-06-23 16:08:54 | INFO | Creating table additional_visibility
|
|
2026-06-23 16:08:54 | INFO | Table ready: additional_visibility
|
|
2026-06-23 16:08:54 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:08:54 | INFO | additional_visibility: inserted 2,136 rows into ClickHouse
|
|
2026-06-23 16:08:54 | INFO | additional_visibility loaded successfully (2136 rows)
|
|
2026-06-23 16:08:54 | INFO | ================================================================================
|
|
2026-06-23 16:08:54 | INFO | Processing Table-:Coverage | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:08:54 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-23 16:08:54 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:08:54 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:09:06 | INFO | Fetched 825 rows from SQL Server
|
|
2026-06-23 16:09:06 | INFO | Fetched total row -: 825 from sql server for table-:Coverage ...........!!!
|
|
2026-06-23 16:09:06 | INFO | Fetched 825 rows
|
|
2026-06-23 16:09:06 | INFO | Creating table Coverage
|
|
2026-06-23 16:09:06 | INFO | Table ready: Coverage
|
|
2026-06-23 16:09:06 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:09:07 | INFO | Coverage: inserted 825 rows into ClickHouse
|
|
2026-06-23 16:09:07 | INFO | Coverage loaded successfully (825 rows)
|
|
2026-06-23 16:09:07 | INFO | ================================================================================
|
|
2026-06-23 16:09:07 | INFO | Processing Table-:Login | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:09:07 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-23 16:09:07 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:09:07 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:09:09 | INFO | Fetched 501 rows from SQL Server
|
|
2026-06-23 16:09:09 | INFO | Fetched total row -: 501 from sql server for table-:Login ...........!!!
|
|
2026-06-23 16:09:09 | INFO | Fetched 501 rows
|
|
2026-06-23 16:09:09 | INFO | Creating table Login
|
|
2026-06-23 16:09:09 | INFO | Table ready: Login
|
|
2026-06-23 16:09:09 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:09:09 | INFO | Login: inserted 501 rows into ClickHouse
|
|
2026-06-23 16:09:09 | INFO | Login loaded successfully (501 rows)
|
|
2026-06-23 16:09:09 | INFO | ================================================================================
|
|
2026-06-23 16:09:09 | INFO | Processing Table-:Stock_Details | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:09:09 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-23 16:09:09 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:09:09 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:09:13 | INFO | Fetched 44,061 rows from SQL Server
|
|
2026-06-23 16:09:13 | INFO | Fetched total row -: 44061 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-23 16:09:13 | INFO | Fetched 44061 rows
|
|
2026-06-23 16:09:13 | INFO | Creating table Stock_Details
|
|
2026-06-23 16:09:13 | INFO | Table ready: Stock_Details
|
|
2026-06-23 16:09:13 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:09:14 | INFO | Stock_Details: inserted 44,061 rows into ClickHouse
|
|
2026-06-23 16:09:14 | INFO | Stock_Details loaded successfully (44061 rows)
|
|
2026-06-23 16:09:14 | INFO | ================================================================================
|
|
2026-06-23 16:09:14 | INFO | Processing Table-:Attendance | Table type -:FACT | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 16:09:14 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-23 16:09:14 | INFO | Fetching Attendance data from 2026-06-07 to 2026-06-22
|
|
2026-06-23 16:09:29 | INFO | Fetched 121,144 attendance rows for 542 employees
|
|
2026-06-23 16:09:29 | INFO | Fetched total row -: 121144 from sql server for table-:Attendance ...........!!!
|
|
2026-06-23 16:09:29 | INFO | Fetched 121144 rows
|
|
2026-06-23 16:09:29 | INFO | Truncated table Attendance
|
|
2026-06-23 16:09:29 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:09:34 | INFO | Attendance: inserted 121,144 rows into ClickHouse
|
|
2026-06-23 16:09:34 | INFO | Attendance loaded successfully (121144 rows)
|
|
2026-06-23 16:09:34 | INFO | ================================================================================
|
|
2026-06-23 16:09:34 | INFO | Processing Table-:Journey_Plan | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:09:34 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-23 16:09:34 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:09:34 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:09:34 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:09:35 | INFO | Fetched 21,737 rows from SQL Server
|
|
2026-06-23 16:09:35 | INFO | Fetched total row -: 21737 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-23 16:09:35 | INFO | Fetched 21737 rows
|
|
2026-06-23 16:09:35 | INFO | Creating table Journey_Plan
|
|
2026-06-23 16:09:36 | INFO | Table ready: Journey_Plan
|
|
2026-06-23 16:09:36 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:09:36 | INFO | Journey_Plan: inserted 21,737 rows into ClickHouse
|
|
2026-06-23 16:09:36 | INFO | Journey_Plan loaded successfully (21737 rows)
|
|
2026-06-23 16:09:36 | INFO | ================================================================================
|
|
2026-06-23 16:09:36 | INFO | Processing Table-:Web_Logins | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:09:36 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-23 16:09:36 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:09:36 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:09:36 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:09:37 | INFO | Fetched 214 rows from SQL Server
|
|
2026-06-23 16:09:37 | INFO | Fetched total row -: 214 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-23 16:09:37 | INFO | Fetched 214 rows
|
|
2026-06-23 16:09:37 | INFO | Creating table Web_Logins
|
|
2026-06-23 16:09:37 | INFO | Table ready: Web_Logins
|
|
2026-06-23 16:09:37 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:09:37 | INFO | Web_Logins: inserted 214 rows into ClickHouse
|
|
2026-06-23 16:09:37 | INFO | Web_Logins loaded successfully (214 rows)
|
|
2026-06-23 16:09:37 | INFO | ================================================================================
|
|
2026-06-23 16:09:37 | INFO | Processing Table-:Promotion | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:09:37 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-23 16:09:37 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:09:40 | INFO | Fetched 8,835 rows from SQL Server
|
|
2026-06-23 16:09:40 | INFO | Fetched total row -: 8835 from sql server for table-:Promotion ...........!!!
|
|
2026-06-23 16:09:40 | INFO | Fetched 8835 rows
|
|
2026-06-23 16:09:40 | INFO | Creating table Promotion
|
|
2026-06-23 16:09:40 | INFO | Table ready: Promotion
|
|
2026-06-23 16:09:40 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:09:41 | INFO | Promotion: inserted 8,835 rows into ClickHouse
|
|
2026-06-23 16:09:41 | INFO | Promotion loaded successfully (8835 rows)
|
|
2026-06-23 16:09:41 | INFO | ================================================================================
|
|
2026-06-23 16:09:41 | INFO | Processing Table-:PaidVisibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:09:41 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
|
|
2026-06-23 16:09:41 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:09:41 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:09:41 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:09:43 | INFO | Fetched 1,029 rows from SQL Server
|
|
2026-06-23 16:09:43 | INFO | Fetched total row -: 1029 from sql server for table-:PaidVisibility ...........!!!
|
|
2026-06-23 16:09:43 | INFO | Fetched 1029 rows
|
|
2026-06-23 16:09:43 | INFO | Creating table PaidVisibility
|
|
2026-06-23 16:09:43 | INFO | Table ready: PaidVisibility
|
|
2026-06-23 16:09:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:09:43 | INFO | PaidVisibility: inserted 1,029 rows into ClickHouse
|
|
2026-06-23 16:09:43 | INFO | PaidVisibility loaded successfully (1029 rows)
|
|
2026-06-23 16:09:43 | INFO | ================================================================================
|
|
2026-06-23 16:09:43 | INFO | Processing Table-:Store_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:09:43 | INFO | Fetching Data from sql server for table-: Store_Master ..............
|
|
2026-06-23 16:09:43 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:09:43 | INFO | Fetching in progress ....
|
|
2026-06-23 16:09:45 | INFO | Fetched 5,998 rows from SQL Server
|
|
2026-06-23 16:09:45 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
|
|
2026-06-23 16:09:45 | INFO | Fetched 5998 rows
|
|
2026-06-23 16:09:45 | INFO | Truncated table Store_Master
|
|
2026-06-23 16:09:45 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:09:48 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
|
|
2026-06-23 16:09:48 | INFO | Store_Master loaded successfully (5998 rows)
|
|
2026-06-23 16:09:48 | INFO | ================================================================================
|
|
2026-06-23 16:09:48 | INFO | Processing Table-:SKU_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:09:48 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
|
|
2026-06-23 16:09:48 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:09:48 | INFO | Fetching in progress ....
|
|
2026-06-23 16:09:48 | INFO | Fetched 160 rows from SQL Server
|
|
2026-06-23 16:09:48 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
|
|
2026-06-23 16:09:48 | INFO | Fetched 160 rows
|
|
2026-06-23 16:09:49 | INFO | Truncated table SKU_Master
|
|
2026-06-23 16:09:49 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:09:49 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-23 16:09:49 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-23 16:09:49 | INFO | ================================================================================
|
|
2026-06-23 16:09:49 | INFO | Processing Table-:display_master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:09:49 | INFO | Fetching Data from sql server for table-: display_master ..............
|
|
2026-06-23 16:09:49 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:09:49 | INFO | Fetching in progress ....
|
|
2026-06-23 16:09:49 | INFO | Fetched 135 rows from SQL Server
|
|
2026-06-23 16:09:49 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
|
|
2026-06-23 16:09:49 | INFO | Fetched 135 rows
|
|
2026-06-23 16:09:49 | INFO | Truncated table display_master
|
|
2026-06-23 16:09:49 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:09:49 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-23 16:09:49 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-23 16:09:49 | INFO | ================================================================================
|
|
2026-06-23 16:09:49 | INFO | Processing Table-:Employee_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:09:49 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
|
|
2026-06-23 16:09:49 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:09:49 | INFO | Fetching in progress ....
|
|
2026-06-23 16:09:50 | INFO | Fetched 2,289 rows from SQL Server
|
|
2026-06-23 16:09:50 | INFO | Fetched total row -: 2289 from sql server for table-:Employee_Master ...........!!!
|
|
2026-06-23 16:09:50 | INFO | Fetched 2289 rows
|
|
2026-06-23 16:09:50 | INFO | Truncated table Employee_Master
|
|
2026-06-23 16:09:50 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:09:51 | INFO | Employee_Master: inserted 2,289 rows into ClickHouse
|
|
2026-06-23 16:09:51 | INFO | Employee_Master loaded successfully (2289 rows)
|
|
2026-06-23 16:09:51 | INFO | ================================================================================
|
|
2026-06-23 16:09:51 | INFO | Processing Table-:coverage_remarks | Table type -:DIMENSION | fetcht by-:reason_id | operation-:DELETE+INSERT
|
|
2026-06-23 16:09:51 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
|
|
2026-06-23 16:09:52 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:09:52 | INFO | Fetching in progress ....
|
|
2026-06-23 16:09:52 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 16:09:52 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 16:09:52 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
|
|
2026-06-23 16:09:52 | WARNING | coverage_remarks returned no rows
|
|
2026-06-23 16:09:52 | INFO | ================================================================================
|
|
2026-06-23 16:09:52 | INFO | Processing Table-:mapping_storevisibility | Table type -:BRIDGE | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 16:09:52 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
|
|
2026-06-23 16:09:52 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:09:52 | ERROR | Failed processing table mapping_storevisibility
|
|
Traceback (most recent call last):
|
|
File "D:\z\airflow-project\main2.py", line 194, in elt
|
|
df = fetch_data(sql_engine ,table_name,table_type)
|
|
File "D:\z\airflow-project\src\dim.py", line 38, in fetch_data
|
|
with open(sql_file, "r", encoding="utf-8") as f:
|
|
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
FileNotFoundError: [Errno 2] No such file or directory: 'src\\sql\\dim\\mapping_storevisibility.sql'
|
|
2026-06-23 16:09:52 | WARNING | Pipeline failed. Retry 1/3. Error: [Errno 2] No such file or directory: 'src\\sql\\dim\\mapping_storevisibility.sql'
|
|
2026-06-23 16:09:57 | INFO | ================================================================================
|
|
2026-06-23 16:09:57 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 16:09:57 | INFO | Pipeline Run Date: 2026-06-22
|
|
2026-06-23 16:09:57 | INFO | Connecting to databases...
|
|
2026-06-23 16:09:57 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B6FB64D6D0>
|
|
2026-06-23 16:09:58 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B6FB64DE50>
|
|
2026-06-23 16:09:59 | INFO | Database connections established
|
|
2026-06-23 16:09:59 | INFO | Collecting MIDs for: 2026-06-22
|
|
2026-06-23 16:09:59 | INFO | Found 882 MIDs
|
|
2026-06-23 16:09:59 | INFO | ================================================================================
|
|
2026-06-23 16:09:59 | INFO | Processing Table-:SOS_OneApp | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:09:59 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-23 16:09:59 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:09:59 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:09:59 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:12:37 | INFO | Pipeline Start Date: 2026-06-22
|
|
2026-06-23 16:12:37 | INFO | ================================================================================
|
|
2026-06-23 16:12:37 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 16:12:37 | INFO | Pipeline Run Date: 2026-06-22
|
|
2026-06-23 16:12:37 | INFO | Connecting to databases...
|
|
2026-06-23 16:12:38 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B677380890>
|
|
2026-06-23 16:12:39 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B6784A87D0>
|
|
2026-06-23 16:12:39 | INFO | Database connections established
|
|
2026-06-23 16:12:39 | INFO | Collecting MIDs for: 2026-06-22
|
|
2026-06-23 16:12:40 | INFO | Found 882 MIDs
|
|
2026-06-23 16:12:40 | INFO | ================================================================================
|
|
2026-06-23 16:12:40 | INFO | Processing Table-:SOS_OneApp | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:12:40 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-23 16:12:40 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:12:40 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:12:40 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:12:45 | INFO | Fetched 3,888 rows from SQL Server
|
|
2026-06-23 16:12:45 | INFO | Fetched total row -: 3888 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-23 16:12:45 | INFO | Fetched 3888 rows
|
|
2026-06-23 16:12:45 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-23 16:12:45 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:12:45 | INFO | SOS_OneApp: inserted 3,888 rows into ClickHouse
|
|
2026-06-23 16:12:45 | INFO | SOS_OneApp loaded successfully (3888 rows)
|
|
2026-06-23 16:12:45 | INFO | ================================================================================
|
|
2026-06-23 16:12:46 | INFO | Processing Table-:Survey | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:12:46 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-23 16:12:46 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:12:46 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:12:46 | INFO | Fetched 152 rows from SQL Server
|
|
2026-06-23 16:12:46 | INFO | Fetched total row -: 152 from sql server for table-:Survey ...........!!!
|
|
2026-06-23 16:12:46 | INFO | Fetched 152 rows
|
|
2026-06-23 16:12:46 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
|
|
2026-06-23 16:12:46 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:12:46 | INFO | Survey: inserted 152 rows into ClickHouse
|
|
2026-06-23 16:12:46 | INFO | Survey loaded successfully (152 rows)
|
|
2026-06-23 16:12:46 | INFO | ================================================================================
|
|
2026-06-23 16:12:46 | INFO | Processing Table-:additional_visibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:12:46 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-23 16:12:46 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:12:46 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:12:46 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:12:47 | INFO | Fetched 2,136 rows from SQL Server
|
|
2026-06-23 16:12:47 | INFO | Fetched total row -: 2136 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-23 16:12:47 | INFO | Fetched 2136 rows
|
|
2026-06-23 16:12:47 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
|
|
2026-06-23 16:12:47 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:12:47 | INFO | additional_visibility: inserted 2,136 rows into ClickHouse
|
|
2026-06-23 16:12:47 | INFO | additional_visibility loaded successfully (2136 rows)
|
|
2026-06-23 16:12:47 | INFO | ================================================================================
|
|
2026-06-23 16:12:47 | INFO | Processing Table-:Coverage | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:12:47 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-23 16:12:47 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:12:47 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:12:58 | INFO | Fetched 825 rows from SQL Server
|
|
2026-06-23 16:12:58 | INFO | Fetched total row -: 825 from sql server for table-:Coverage ...........!!!
|
|
2026-06-23 16:12:58 | INFO | Fetched 825 rows
|
|
2026-06-23 16:12:58 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
|
|
2026-06-23 16:12:58 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:12:58 | INFO | Coverage: inserted 825 rows into ClickHouse
|
|
2026-06-23 16:12:58 | INFO | Coverage loaded successfully (825 rows)
|
|
2026-06-23 16:12:58 | INFO | ================================================================================
|
|
2026-06-23 16:12:58 | INFO | Processing Table-:Login | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:12:58 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-23 16:12:58 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:12:58 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:13:00 | INFO | Fetched 501 rows from SQL Server
|
|
2026-06-23 16:13:00 | INFO | Fetched total row -: 501 from sql server for table-:Login ...........!!!
|
|
2026-06-23 16:13:00 | INFO | Fetched 501 rows
|
|
2026-06-23 16:13:00 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _
|
|
2026-06-23 16:13:01 | INFO | No delete logic required for Login
|
|
2026-06-23 16:13:01 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:13:01 | INFO | Login: inserted 501 rows into ClickHouse
|
|
2026-06-23 16:13:01 | INFO | Login loaded successfully (501 rows)
|
|
2026-06-23 16:13:01 | INFO | ================================================================================
|
|
2026-06-23 16:13:01 | INFO | Processing Table-:Stock_Details | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:13:01 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-23 16:13:01 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:13:01 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:13:05 | INFO | Fetched 44,061 rows from SQL Server
|
|
2026-06-23 16:13:05 | INFO | Fetched total row -: 44061 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-23 16:13:05 | INFO | Fetched 44061 rows
|
|
2026-06-23 16:13:05 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _
|
|
2026-06-23 16:13:05 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:13:05 | INFO | Stock_Details: inserted 44,061 rows into ClickHouse
|
|
2026-06-23 16:13:05 | INFO | Stock_Details loaded successfully (44061 rows)
|
|
2026-06-23 16:13:05 | INFO | ================================================================================
|
|
2026-06-23 16:13:05 | INFO | Processing Table-:Attendance | Table type -:FACT | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 16:13:05 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-23 16:13:05 | INFO | Fetching Attendance data from 2026-06-07 to 2026-06-22
|
|
2026-06-23 16:13:24 | INFO | Fetched 121,144 attendance rows for 542 employees
|
|
2026-06-23 16:13:24 | INFO | Fetched total row -: 121144 from sql server for table-:Attendance ...........!!!
|
|
2026-06-23 16:13:24 | INFO | Fetched 121144 rows
|
|
2026-06-23 16:13:25 | INFO | Truncated table Attendance
|
|
2026-06-23 16:13:25 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:13:27 | INFO | Attendance: inserted 121,144 rows into ClickHouse
|
|
2026-06-23 16:13:27 | INFO | Attendance loaded successfully (121144 rows)
|
|
2026-06-23 16:13:27 | INFO | ================================================================================
|
|
2026-06-23 16:13:27 | INFO | Processing Table-:Journey_Plan | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:13:27 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-23 16:13:27 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:13:27 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:13:27 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:13:29 | INFO | Fetched 21,737 rows from SQL Server
|
|
2026-06-23 16:13:29 | INFO | Fetched total row -: 21737 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-23 16:13:29 | INFO | Fetched 21737 rows
|
|
2026-06-23 16:13:29 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
|
|
2026-06-23 16:13:29 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:13:30 | INFO | Journey_Plan: inserted 21,737 rows into ClickHouse
|
|
2026-06-23 16:13:30 | INFO | Journey_Plan loaded successfully (21737 rows)
|
|
2026-06-23 16:13:30 | INFO | ================================================================================
|
|
2026-06-23 16:13:30 | INFO | Processing Table-:Web_Logins | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:13:30 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-23 16:13:30 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:13:30 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:13:30 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:13:30 | INFO | Fetched 214 rows from SQL Server
|
|
2026-06-23 16:13:30 | INFO | Fetched total row -: 214 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-23 16:13:30 | INFO | Fetched 214 rows
|
|
2026-06-23 16:13:30 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
|
|
2026-06-23 16:13:30 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:13:31 | INFO | Web_Logins: inserted 214 rows into ClickHouse
|
|
2026-06-23 16:13:31 | INFO | Web_Logins loaded successfully (214 rows)
|
|
2026-06-23 16:13:31 | INFO | ================================================================================
|
|
2026-06-23 16:13:31 | INFO | Processing Table-:Promotion | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:13:31 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-23 16:13:31 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:13:39 | INFO | Fetched 8,835 rows from SQL Server
|
|
2026-06-23 16:13:39 | INFO | Fetched total row -: 8835 from sql server for table-:Promotion ...........!!!
|
|
2026-06-23 16:13:39 | INFO | Fetched 8835 rows
|
|
2026-06-23 16:13:39 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-23 16:13:39 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:13:39 | INFO | Promotion: inserted 8,835 rows into ClickHouse
|
|
2026-06-23 16:13:39 | INFO | Promotion loaded successfully (8835 rows)
|
|
2026-06-23 16:13:39 | INFO | ================================================================================
|
|
2026-06-23 16:13:39 | INFO | Processing Table-:PaidVisibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:13:39 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
|
|
2026-06-23 16:13:39 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:13:39 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:13:39 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:13:41 | INFO | Fetched 1,029 rows from SQL Server
|
|
2026-06-23 16:13:41 | INFO | Fetched total row -: 1029 from sql server for table-:PaidVisibility ...........!!!
|
|
2026-06-23 16:13:41 | INFO | Fetched 1029 rows
|
|
2026-06-23 16:13:42 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _
|
|
2026-06-23 16:13:42 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:13:42 | INFO | PaidVisibility: inserted 1,029 rows into ClickHouse
|
|
2026-06-23 16:13:42 | INFO | PaidVisibility loaded successfully (1029 rows)
|
|
2026-06-23 16:13:42 | INFO | ================================================================================
|
|
2026-06-23 16:13:42 | INFO | Processing Table-:Store_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:13:42 | INFO | Fetching Data from sql server for table-: Store_Master ..............
|
|
2026-06-23 16:13:42 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:13:42 | INFO | Fetching in progress ....
|
|
2026-06-23 16:13:45 | INFO | Fetched 5,998 rows from SQL Server
|
|
2026-06-23 16:13:45 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
|
|
2026-06-23 16:13:45 | INFO | Fetched 5998 rows
|
|
2026-06-23 16:13:46 | INFO | Truncated table Store_Master
|
|
2026-06-23 16:13:46 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:13:46 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
|
|
2026-06-23 16:13:46 | INFO | Store_Master loaded successfully (5998 rows)
|
|
2026-06-23 16:13:46 | INFO | ================================================================================
|
|
2026-06-23 16:13:46 | INFO | Processing Table-:SKU_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:13:46 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
|
|
2026-06-23 16:13:46 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:13:46 | INFO | Fetching in progress ....
|
|
2026-06-23 16:13:46 | INFO | Fetched 160 rows from SQL Server
|
|
2026-06-23 16:13:46 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
|
|
2026-06-23 16:13:46 | INFO | Fetched 160 rows
|
|
2026-06-23 16:13:46 | INFO | Truncated table SKU_Master
|
|
2026-06-23 16:13:46 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:13:47 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-23 16:13:47 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-23 16:13:47 | INFO | ================================================================================
|
|
2026-06-23 16:13:47 | INFO | Processing Table-:display_master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:13:47 | INFO | Fetching Data from sql server for table-: display_master ..............
|
|
2026-06-23 16:13:47 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:13:47 | INFO | Fetching in progress ....
|
|
2026-06-23 16:13:47 | INFO | Fetched 135 rows from SQL Server
|
|
2026-06-23 16:13:47 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
|
|
2026-06-23 16:13:47 | INFO | Fetched 135 rows
|
|
2026-06-23 16:13:47 | INFO | Truncated table display_master
|
|
2026-06-23 16:13:47 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:13:47 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-23 16:13:47 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-23 16:13:47 | INFO | ================================================================================
|
|
2026-06-23 16:13:47 | INFO | Processing Table-:Employee_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:13:47 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
|
|
2026-06-23 16:13:47 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:13:47 | INFO | Fetching in progress ....
|
|
2026-06-23 16:13:48 | INFO | Fetched 2,289 rows from SQL Server
|
|
2026-06-23 16:13:48 | INFO | Fetched total row -: 2289 from sql server for table-:Employee_Master ...........!!!
|
|
2026-06-23 16:13:48 | INFO | Fetched 2289 rows
|
|
2026-06-23 16:13:48 | INFO | Truncated table Employee_Master
|
|
2026-06-23 16:13:48 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:13:48 | INFO | Employee_Master: inserted 2,289 rows into ClickHouse
|
|
2026-06-23 16:13:48 | INFO | Employee_Master loaded successfully (2289 rows)
|
|
2026-06-23 16:13:48 | INFO | ================================================================================
|
|
2026-06-23 16:13:48 | INFO | Processing Table-:coverage_remarks | Table type -:DIMENSION | fetcht by-:reason_id | operation-:DELETE+INSERT
|
|
2026-06-23 16:13:48 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
|
|
2026-06-23 16:13:49 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:13:49 | INFO | Fetching in progress ....
|
|
2026-06-23 16:13:49 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 16:13:49 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 16:13:49 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
|
|
2026-06-23 16:13:49 | WARNING | coverage_remarks returned no rows
|
|
2026-06-23 16:13:49 | INFO | ================================================================================
|
|
2026-06-23 16:13:49 | INFO | Processing Table-:mapping_storevisibility | Table type -:BRIDGE | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 16:13:49 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
|
|
2026-06-23 16:13:49 | INFO | Fetching data from sql server for BRIDGE table......
|
|
2026-06-23 16:13:49 | ERROR | Failed processing table mapping_storevisibility
|
|
Traceback (most recent call last):
|
|
File "D:\z\airflow-project\main2.py", line 192, in elt
|
|
df=fn(sql_engine, table_name , table_type, mids, run_date)
|
|
File "D:\z\airflow-project\src\bridge.py", line 34, in fetch_mapping_storevisibility
|
|
sql = sql_template.format( )
|
|
KeyError: 'store_id_list'
|
|
2026-06-23 16:13:49 | WARNING | Pipeline failed. Retry 1/3. Error: 'store_id_list'
|
|
2026-06-23 16:13:54 | INFO | ================================================================================
|
|
2026-06-23 16:13:54 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 16:13:54 | INFO | Pipeline Run Date: 2026-06-22
|
|
2026-06-23 16:13:54 | INFO | Connecting to databases...
|
|
2026-06-23 16:13:54 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B6784B1130>
|
|
2026-06-23 16:13:55 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B6784B1F40>
|
|
2026-06-23 16:13:55 | INFO | Database connections established
|
|
2026-06-23 16:13:55 | INFO | Collecting MIDs for: 2026-06-22
|
|
2026-06-23 16:13:55 | INFO | Found 882 MIDs
|
|
2026-06-23 16:13:56 | INFO | ================================================================================
|
|
2026-06-23 16:13:56 | INFO | Processing Table-:SOS_OneApp | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:13:56 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-23 16:13:56 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:13:56 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:13:56 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:14:03 | INFO | Fetched 3,888 rows from SQL Server
|
|
2026-06-23 16:14:03 | INFO | Fetched total row -: 3888 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-23 16:14:03 | INFO | Fetched 3888 rows
|
|
2026-06-23 16:14:03 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-23 16:14:04 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:14:04 | INFO | SOS_OneApp: inserted 3,888 rows into ClickHouse
|
|
2026-06-23 16:14:04 | INFO | SOS_OneApp loaded successfully (3888 rows)
|
|
2026-06-23 16:14:04 | INFO | ================================================================================
|
|
2026-06-23 16:14:04 | INFO | Processing Table-:Survey | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:14:04 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-23 16:14:04 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:14:04 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:14:04 | INFO | Fetched 152 rows from SQL Server
|
|
2026-06-23 16:14:04 | INFO | Fetched total row -: 152 from sql server for table-:Survey ...........!!!
|
|
2026-06-23 16:14:04 | INFO | Fetched 152 rows
|
|
2026-06-23 16:14:04 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
|
|
2026-06-23 16:14:04 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:14:04 | INFO | Survey: inserted 152 rows into ClickHouse
|
|
2026-06-23 16:14:04 | INFO | Survey loaded successfully (152 rows)
|
|
2026-06-23 16:14:04 | INFO | ================================================================================
|
|
2026-06-23 16:14:04 | INFO | Processing Table-:additional_visibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:14:04 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-23 16:14:04 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:14:04 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:14:04 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:14:08 | INFO | Fetched 2,136 rows from SQL Server
|
|
2026-06-23 16:14:08 | INFO | Fetched total row -: 2136 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-23 16:14:08 | INFO | Fetched 2136 rows
|
|
2026-06-23 16:14:08 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
|
|
2026-06-23 16:14:08 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:14:08 | INFO | additional_visibility: inserted 2,136 rows into ClickHouse
|
|
2026-06-23 16:14:08 | INFO | additional_visibility loaded successfully (2136 rows)
|
|
2026-06-23 16:14:08 | INFO | ================================================================================
|
|
2026-06-23 16:14:08 | INFO | Processing Table-:Coverage | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:14:08 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-23 16:14:08 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:14:08 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:14:18 | INFO | Fetched 825 rows from SQL Server
|
|
2026-06-23 16:14:18 | INFO | Fetched total row -: 825 from sql server for table-:Coverage ...........!!!
|
|
2026-06-23 16:14:18 | INFO | Fetched 825 rows
|
|
2026-06-23 16:14:18 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
|
|
2026-06-23 16:14:18 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:14:19 | INFO | Coverage: inserted 825 rows into ClickHouse
|
|
2026-06-23 16:14:19 | INFO | Coverage loaded successfully (825 rows)
|
|
2026-06-23 16:14:19 | INFO | ================================================================================
|
|
2026-06-23 16:14:19 | INFO | Processing Table-:Login | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:14:19 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-23 16:14:19 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:14:19 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:14:20 | INFO | Fetched 501 rows from SQL Server
|
|
2026-06-23 16:14:20 | INFO | Fetched total row -: 501 from sql server for table-:Login ...........!!!
|
|
2026-06-23 16:14:20 | INFO | Fetched 501 rows
|
|
2026-06-23 16:14:20 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _
|
|
2026-06-23 16:14:20 | INFO | No delete logic required for Login
|
|
2026-06-23 16:14:20 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:14:20 | INFO | Login: inserted 501 rows into ClickHouse
|
|
2026-06-23 16:14:20 | INFO | Login loaded successfully (501 rows)
|
|
2026-06-23 16:14:20 | INFO | ================================================================================
|
|
2026-06-23 16:14:20 | INFO | Processing Table-:Stock_Details | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:14:20 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-23 16:14:20 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:14:20 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:14:36 | INFO | Fetched 44,061 rows from SQL Server
|
|
2026-06-23 16:14:36 | INFO | Fetched total row -: 44061 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-23 16:14:36 | INFO | Fetched 44061 rows
|
|
2026-06-23 16:14:36 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _
|
|
2026-06-23 16:14:36 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:14:37 | INFO | Stock_Details: inserted 44,061 rows into ClickHouse
|
|
2026-06-23 16:14:37 | INFO | Stock_Details loaded successfully (44061 rows)
|
|
2026-06-23 16:14:37 | INFO | ================================================================================
|
|
2026-06-23 16:14:37 | INFO | Processing Table-:Attendance | Table type -:FACT | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 16:14:37 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-23 16:14:37 | INFO | Fetching Attendance data from 2026-06-07 to 2026-06-22
|
|
2026-06-23 16:16:13 | INFO | Fetched 121,144 attendance rows for 542 employees
|
|
2026-06-23 16:16:13 | INFO | Fetched total row -: 121144 from sql server for table-:Attendance ...........!!!
|
|
2026-06-23 16:16:13 | INFO | Fetched 121144 rows
|
|
2026-06-23 16:16:13 | INFO | Truncated table Attendance
|
|
2026-06-23 16:16:13 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:16:14 | INFO | Attendance: inserted 121,144 rows into ClickHouse
|
|
2026-06-23 16:16:14 | INFO | Attendance loaded successfully (121144 rows)
|
|
2026-06-23 16:16:14 | INFO | ================================================================================
|
|
2026-06-23 16:16:14 | INFO | Processing Table-:Journey_Plan | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:16:14 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-23 16:16:14 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:16:14 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:16:14 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:16:18 | INFO | Fetched 21,737 rows from SQL Server
|
|
2026-06-23 16:16:18 | INFO | Fetched total row -: 21737 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-23 16:16:18 | INFO | Fetched 21737 rows
|
|
2026-06-23 16:16:18 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
|
|
2026-06-23 16:16:18 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:16:19 | INFO | Journey_Plan: inserted 21,737 rows into ClickHouse
|
|
2026-06-23 16:16:19 | INFO | Journey_Plan loaded successfully (21737 rows)
|
|
2026-06-23 16:16:19 | INFO | ================================================================================
|
|
2026-06-23 16:16:19 | INFO | Processing Table-:Web_Logins | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:16:19 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-23 16:16:19 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:16:19 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:16:19 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:16:20 | INFO | Fetched 214 rows from SQL Server
|
|
2026-06-23 16:16:20 | INFO | Fetched total row -: 214 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-23 16:16:20 | INFO | Fetched 214 rows
|
|
2026-06-23 16:16:20 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
|
|
2026-06-23 16:16:20 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:16:20 | INFO | Web_Logins: inserted 214 rows into ClickHouse
|
|
2026-06-23 16:16:20 | INFO | Web_Logins loaded successfully (214 rows)
|
|
2026-06-23 16:16:20 | INFO | ================================================================================
|
|
2026-06-23 16:16:20 | INFO | Processing Table-:Promotion | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:16:20 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-23 16:16:20 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:16:47 | INFO | Fetched 8,835 rows from SQL Server
|
|
2026-06-23 16:16:47 | INFO | Fetched total row -: 8835 from sql server for table-:Promotion ...........!!!
|
|
2026-06-23 16:16:47 | INFO | Fetched 8835 rows
|
|
2026-06-23 16:16:47 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-23 16:16:47 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:16:48 | INFO | Promotion: inserted 8,835 rows into ClickHouse
|
|
2026-06-23 16:16:48 | INFO | Promotion loaded successfully (8835 rows)
|
|
2026-06-23 16:16:48 | INFO | ================================================================================
|
|
2026-06-23 16:16:48 | INFO | Processing Table-:PaidVisibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:16:48 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
|
|
2026-06-23 16:16:48 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:16:48 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:16:48 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:16:56 | INFO | Fetched 1,029 rows from SQL Server
|
|
2026-06-23 16:16:56 | INFO | Fetched total row -: 1029 from sql server for table-:PaidVisibility ...........!!!
|
|
2026-06-23 16:16:56 | INFO | Fetched 1029 rows
|
|
2026-06-23 16:16:56 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _
|
|
2026-06-23 16:16:57 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:16:57 | INFO | PaidVisibility: inserted 1,029 rows into ClickHouse
|
|
2026-06-23 16:16:57 | INFO | PaidVisibility loaded successfully (1029 rows)
|
|
2026-06-23 16:16:57 | INFO | ================================================================================
|
|
2026-06-23 16:16:57 | INFO | Processing Table-:Store_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:16:57 | INFO | Fetching Data from sql server for table-: Store_Master ..............
|
|
2026-06-23 16:16:57 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:16:57 | INFO | Fetching in progress ....
|
|
2026-06-23 16:17:15 | INFO | Fetched 5,998 rows from SQL Server
|
|
2026-06-23 16:17:15 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
|
|
2026-06-23 16:17:15 | INFO | Fetched 5998 rows
|
|
2026-06-23 16:17:15 | INFO | Truncated table Store_Master
|
|
2026-06-23 16:17:15 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:17:17 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
|
|
2026-06-23 16:17:17 | INFO | Store_Master loaded successfully (5998 rows)
|
|
2026-06-23 16:17:17 | INFO | ================================================================================
|
|
2026-06-23 16:17:17 | INFO | Processing Table-:SKU_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:17:17 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
|
|
2026-06-23 16:17:17 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:17:17 | INFO | Fetching in progress ....
|
|
2026-06-23 16:17:18 | INFO | Fetched 160 rows from SQL Server
|
|
2026-06-23 16:17:18 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
|
|
2026-06-23 16:17:18 | INFO | Fetched 160 rows
|
|
2026-06-23 16:17:18 | INFO | Truncated table SKU_Master
|
|
2026-06-23 16:17:18 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:17:19 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-23 16:17:19 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-23 16:17:19 | INFO | ================================================================================
|
|
2026-06-23 16:17:19 | INFO | Processing Table-:display_master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:17:19 | INFO | Fetching Data from sql server for table-: display_master ..............
|
|
2026-06-23 16:17:19 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:17:19 | INFO | Fetching in progress ....
|
|
2026-06-23 16:17:19 | INFO | Fetched 135 rows from SQL Server
|
|
2026-06-23 16:17:19 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
|
|
2026-06-23 16:17:19 | INFO | Fetched 135 rows
|
|
2026-06-23 16:17:19 | INFO | Truncated table display_master
|
|
2026-06-23 16:17:19 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:17:19 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-23 16:17:19 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-23 16:17:19 | INFO | ================================================================================
|
|
2026-06-23 16:17:19 | INFO | Processing Table-:Employee_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:17:19 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
|
|
2026-06-23 16:17:19 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:17:19 | INFO | Fetching in progress ....
|
|
2026-06-23 16:17:24 | INFO | Fetched 2,289 rows from SQL Server
|
|
2026-06-23 16:17:24 | INFO | Fetched total row -: 2289 from sql server for table-:Employee_Master ...........!!!
|
|
2026-06-23 16:17:24 | INFO | Fetched 2289 rows
|
|
2026-06-23 16:17:25 | INFO | Truncated table Employee_Master
|
|
2026-06-23 16:17:25 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:17:25 | INFO | Employee_Master: inserted 2,289 rows into ClickHouse
|
|
2026-06-23 16:17:25 | INFO | Employee_Master loaded successfully (2289 rows)
|
|
2026-06-23 16:17:25 | INFO | ================================================================================
|
|
2026-06-23 16:17:25 | INFO | Processing Table-:coverage_remarks | Table type -:DIMENSION | fetcht by-:reason_id | operation-:DELETE+INSERT
|
|
2026-06-23 16:17:25 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
|
|
2026-06-23 16:17:26 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:17:26 | INFO | Fetching in progress ....
|
|
2026-06-23 16:17:26 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 16:17:26 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 16:17:26 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
|
|
2026-06-23 16:17:26 | WARNING | coverage_remarks returned no rows
|
|
2026-06-23 16:17:26 | INFO | ================================================================================
|
|
2026-06-23 16:17:26 | INFO | Processing Table-:mapping_storevisibility | Table type -:BRIDGE | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 16:17:26 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
|
|
2026-06-23 16:17:26 | INFO | Fetching data from sql server for BRIDGE table......
|
|
2026-06-23 16:17:26 | ERROR | Failed processing table mapping_storevisibility
|
|
Traceback (most recent call last):
|
|
File "D:\z\airflow-project\main2.py", line 192, in elt
|
|
df=fn(sql_engine, table_name , table_type, mids, run_date)
|
|
File "D:\z\airflow-project\src\bridge.py", line 34, in fetch_mapping_storevisibility
|
|
sql = sql_template.format( )
|
|
KeyError: 'store_id_list'
|
|
2026-06-23 16:17:26 | WARNING | Pipeline failed. Retry 2/3. Error: 'store_id_list'
|
|
2026-06-23 16:17:31 | INFO | ================================================================================
|
|
2026-06-23 16:17:31 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 16:17:31 | INFO | Pipeline Run Date: 2026-06-22
|
|
2026-06-23 16:17:31 | INFO | Connecting to databases...
|
|
2026-06-23 16:17:31 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B6784B2300>
|
|
2026-06-23 16:17:32 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B6784B14F0>
|
|
2026-06-23 16:17:32 | INFO | Database connections established
|
|
2026-06-23 16:17:32 | INFO | Collecting MIDs for: 2026-06-22
|
|
2026-06-23 16:17:32 | INFO | Found 882 MIDs
|
|
2026-06-23 16:17:32 | INFO | ================================================================================
|
|
2026-06-23 16:17:32 | INFO | Processing Table-:SOS_OneApp | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:17:32 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-23 16:17:32 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:17:32 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:17:32 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:17:43 | INFO | Fetched 3,888 rows from SQL Server
|
|
2026-06-23 16:17:43 | INFO | Fetched total row -: 3888 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-23 16:17:43 | INFO | Fetched 3888 rows
|
|
2026-06-23 16:17:43 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-23 16:17:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:17:44 | INFO | SOS_OneApp: inserted 3,888 rows into ClickHouse
|
|
2026-06-23 16:17:44 | INFO | SOS_OneApp loaded successfully (3888 rows)
|
|
2026-06-23 16:17:44 | INFO | ================================================================================
|
|
2026-06-23 16:17:44 | INFO | Processing Table-:Survey | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:17:44 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-23 16:17:44 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:17:44 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:17:44 | INFO | Fetched 152 rows from SQL Server
|
|
2026-06-23 16:17:44 | INFO | Fetched total row -: 152 from sql server for table-:Survey ...........!!!
|
|
2026-06-23 16:17:44 | INFO | Fetched 152 rows
|
|
2026-06-23 16:17:44 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
|
|
2026-06-23 16:17:44 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:17:44 | INFO | Survey: inserted 152 rows into ClickHouse
|
|
2026-06-23 16:17:44 | INFO | Survey loaded successfully (152 rows)
|
|
2026-06-23 16:17:44 | INFO | ================================================================================
|
|
2026-06-23 16:17:44 | INFO | Processing Table-:additional_visibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:17:44 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-23 16:17:44 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:17:44 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:17:44 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:17:47 | INFO | Fetched 2,136 rows from SQL Server
|
|
2026-06-23 16:17:47 | INFO | Fetched total row -: 2136 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-23 16:17:47 | INFO | Fetched 2136 rows
|
|
2026-06-23 16:17:47 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
|
|
2026-06-23 16:17:48 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:17:48 | INFO | additional_visibility: inserted 2,136 rows into ClickHouse
|
|
2026-06-23 16:17:48 | INFO | additional_visibility loaded successfully (2136 rows)
|
|
2026-06-23 16:17:48 | INFO | ================================================================================
|
|
2026-06-23 16:17:48 | INFO | Processing Table-:Coverage | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:17:48 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-23 16:17:48 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:17:48 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:17:58 | INFO | Fetched 825 rows from SQL Server
|
|
2026-06-23 16:17:58 | INFO | Fetched total row -: 825 from sql server for table-:Coverage ...........!!!
|
|
2026-06-23 16:17:58 | INFO | Fetched 825 rows
|
|
2026-06-23 16:17:58 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
|
|
2026-06-23 16:17:58 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:17:58 | INFO | Coverage: inserted 825 rows into ClickHouse
|
|
2026-06-23 16:17:58 | INFO | Coverage loaded successfully (825 rows)
|
|
2026-06-23 16:17:58 | INFO | ================================================================================
|
|
2026-06-23 16:17:58 | INFO | Processing Table-:Login | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:17:58 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-23 16:17:58 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:17:58 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:18:01 | INFO | Fetched 501 rows from SQL Server
|
|
2026-06-23 16:18:01 | INFO | Fetched total row -: 501 from sql server for table-:Login ...........!!!
|
|
2026-06-23 16:18:01 | INFO | Fetched 501 rows
|
|
2026-06-23 16:18:01 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _
|
|
2026-06-23 16:18:01 | INFO | No delete logic required for Login
|
|
2026-06-23 16:18:01 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:18:01 | INFO | Login: inserted 501 rows into ClickHouse
|
|
2026-06-23 16:18:01 | INFO | Login loaded successfully (501 rows)
|
|
2026-06-23 16:18:01 | INFO | ================================================================================
|
|
2026-06-23 16:18:01 | INFO | Processing Table-:Stock_Details | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:18:01 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-23 16:18:01 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:18:01 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:18:56 | INFO | Fetched 44,061 rows from SQL Server
|
|
2026-06-23 16:18:56 | INFO | Fetched total row -: 44061 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-23 16:18:56 | INFO | Fetched 44061 rows
|
|
2026-06-23 16:18:57 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _
|
|
2026-06-23 16:18:57 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:18:59 | INFO | Stock_Details: inserted 44,061 rows into ClickHouse
|
|
2026-06-23 16:18:59 | INFO | Stock_Details loaded successfully (44061 rows)
|
|
2026-06-23 16:18:59 | INFO | ================================================================================
|
|
2026-06-23 16:18:59 | INFO | Processing Table-:Attendance | Table type -:FACT | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 16:18:59 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-23 16:18:59 | INFO | Fetching Attendance data from 2026-06-07 to 2026-06-22
|
|
2026-06-23 16:20:19 | INFO | Fetched 121,144 attendance rows for 542 employees
|
|
2026-06-23 16:20:19 | INFO | Fetched total row -: 121144 from sql server for table-:Attendance ...........!!!
|
|
2026-06-23 16:20:19 | INFO | Fetched 121144 rows
|
|
2026-06-23 16:20:20 | INFO | Truncated table Attendance
|
|
2026-06-23 16:20:20 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:20:25 | INFO | Attendance: inserted 121,144 rows into ClickHouse
|
|
2026-06-23 16:20:25 | INFO | Attendance loaded successfully (121144 rows)
|
|
2026-06-23 16:20:25 | INFO | ================================================================================
|
|
2026-06-23 16:20:25 | INFO | Processing Table-:Journey_Plan | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:20:25 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-23 16:20:25 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:20:25 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:20:25 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:20:29 | INFO | Fetched 21,737 rows from SQL Server
|
|
2026-06-23 16:20:29 | INFO | Fetched total row -: 21737 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-23 16:20:29 | INFO | Fetched 21737 rows
|
|
2026-06-23 16:20:29 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
|
|
2026-06-23 16:20:29 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:20:30 | INFO | Journey_Plan: inserted 21,737 rows into ClickHouse
|
|
2026-06-23 16:20:30 | INFO | Journey_Plan loaded successfully (21737 rows)
|
|
2026-06-23 16:20:30 | INFO | ================================================================================
|
|
2026-06-23 16:20:30 | INFO | Processing Table-:Web_Logins | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:20:30 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-23 16:20:30 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:20:30 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:20:30 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:20:30 | INFO | Fetched 214 rows from SQL Server
|
|
2026-06-23 16:20:30 | INFO | Fetched total row -: 214 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-23 16:20:30 | INFO | Fetched 214 rows
|
|
2026-06-23 16:20:30 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
|
|
2026-06-23 16:20:30 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:20:30 | INFO | Web_Logins: inserted 214 rows into ClickHouse
|
|
2026-06-23 16:20:30 | INFO | Web_Logins loaded successfully (214 rows)
|
|
2026-06-23 16:20:30 | INFO | ================================================================================
|
|
2026-06-23 16:20:30 | INFO | Processing Table-:Promotion | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:20:30 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-23 16:20:30 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:20:45 | INFO | Fetched 8,835 rows from SQL Server
|
|
2026-06-23 16:20:45 | INFO | Fetched total row -: 8835 from sql server for table-:Promotion ...........!!!
|
|
2026-06-23 16:20:45 | INFO | Fetched 8835 rows
|
|
2026-06-23 16:20:45 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-23 16:20:45 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:20:46 | INFO | Promotion: inserted 8,835 rows into ClickHouse
|
|
2026-06-23 16:20:46 | INFO | Promotion loaded successfully (8835 rows)
|
|
2026-06-23 16:20:46 | INFO | ================================================================================
|
|
2026-06-23 16:20:46 | INFO | Processing Table-:PaidVisibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:20:46 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
|
|
2026-06-23 16:20:46 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 16:20:46 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 16:20:46 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 16:20:50 | INFO | Fetched 1,029 rows from SQL Server
|
|
2026-06-23 16:20:50 | INFO | Fetched total row -: 1029 from sql server for table-:PaidVisibility ...........!!!
|
|
2026-06-23 16:20:50 | INFO | Fetched 1029 rows
|
|
2026-06-23 16:20:50 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _
|
|
2026-06-23 16:20:50 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:20:50 | INFO | PaidVisibility: inserted 1,029 rows into ClickHouse
|
|
2026-06-23 16:20:50 | INFO | PaidVisibility loaded successfully (1029 rows)
|
|
2026-06-23 16:20:50 | INFO | ================================================================================
|
|
2026-06-23 16:20:50 | INFO | Processing Table-:Store_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:20:50 | INFO | Fetching Data from sql server for table-: Store_Master ..............
|
|
2026-06-23 16:20:50 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:20:50 | INFO | Fetching in progress ....
|
|
2026-06-23 16:21:04 | INFO | Fetched 5,998 rows from SQL Server
|
|
2026-06-23 16:21:04 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
|
|
2026-06-23 16:21:04 | INFO | Fetched 5998 rows
|
|
2026-06-23 16:21:04 | INFO | Truncated table Store_Master
|
|
2026-06-23 16:21:04 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:21:05 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
|
|
2026-06-23 16:21:05 | INFO | Store_Master loaded successfully (5998 rows)
|
|
2026-06-23 16:21:05 | INFO | ================================================================================
|
|
2026-06-23 16:21:05 | INFO | Processing Table-:SKU_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:21:05 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
|
|
2026-06-23 16:21:05 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:21:05 | INFO | Fetching in progress ....
|
|
2026-06-23 16:21:05 | INFO | Fetched 160 rows from SQL Server
|
|
2026-06-23 16:21:05 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
|
|
2026-06-23 16:21:05 | INFO | Fetched 160 rows
|
|
2026-06-23 16:21:05 | INFO | Truncated table SKU_Master
|
|
2026-06-23 16:21:05 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:21:05 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-23 16:21:05 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-23 16:21:05 | INFO | ================================================================================
|
|
2026-06-23 16:21:05 | INFO | Processing Table-:display_master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:21:05 | INFO | Fetching Data from sql server for table-: display_master ..............
|
|
2026-06-23 16:21:05 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:21:05 | INFO | Fetching in progress ....
|
|
2026-06-23 16:21:06 | INFO | Fetched 135 rows from SQL Server
|
|
2026-06-23 16:21:06 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
|
|
2026-06-23 16:21:06 | INFO | Fetched 135 rows
|
|
2026-06-23 16:21:06 | INFO | Truncated table display_master
|
|
2026-06-23 16:21:06 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:21:06 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-23 16:21:06 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-23 16:21:06 | INFO | ================================================================================
|
|
2026-06-23 16:21:06 | INFO | Processing Table-:Employee_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:21:06 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
|
|
2026-06-23 16:21:06 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:21:06 | INFO | Fetching in progress ....
|
|
2026-06-23 16:21:11 | INFO | Fetched 2,289 rows from SQL Server
|
|
2026-06-23 16:21:11 | INFO | Fetched total row -: 2289 from sql server for table-:Employee_Master ...........!!!
|
|
2026-06-23 16:21:11 | INFO | Fetched 2289 rows
|
|
2026-06-23 16:21:11 | INFO | Truncated table Employee_Master
|
|
2026-06-23 16:21:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:21:12 | INFO | Employee_Master: inserted 2,289 rows into ClickHouse
|
|
2026-06-23 16:21:12 | INFO | Employee_Master loaded successfully (2289 rows)
|
|
2026-06-23 16:21:12 | INFO | ================================================================================
|
|
2026-06-23 16:21:12 | INFO | Processing Table-:coverage_remarks | Table type -:DIMENSION | fetcht by-:reason_id | operation-:DELETE+INSERT
|
|
2026-06-23 16:21:12 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
|
|
2026-06-23 16:21:12 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:21:12 | INFO | Fetching in progress ....
|
|
2026-06-23 16:21:12 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 16:21:12 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 16:21:12 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
|
|
2026-06-23 16:21:12 | WARNING | coverage_remarks returned no rows
|
|
2026-06-23 16:21:12 | INFO | ================================================================================
|
|
2026-06-23 16:21:12 | INFO | Processing Table-:mapping_storevisibility | Table type -:BRIDGE | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 16:21:12 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
|
|
2026-06-23 16:21:12 | INFO | Fetching data from sql server for BRIDGE table......
|
|
2026-06-23 16:21:12 | INFO | Fetching in progress ....
|
|
2026-06-23 16:21:13 | INFO | Fetched 805 rows from SQL Server
|
|
2026-06-23 16:21:13 | INFO | Fetched total row -: 805 from sql server for table-:mapping_storevisibility ...........!!!
|
|
2026-06-23 16:21:13 | INFO | Fetched 805 rows
|
|
2026-06-23 16:21:13 | INFO | Creating table mapping_storevisibility
|
|
2026-06-23 16:21:13 | INFO | Table ready: mapping_storevisibility
|
|
2026-06-23 16:21:13 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:21:13 | INFO | mapping_storevisibility: inserted 805 rows into ClickHouse
|
|
2026-06-23 16:21:13 | INFO | mapping_storevisibility loaded successfully (805 rows)
|
|
2026-06-23 16:21:13 | INFO | ================================================================================
|
|
2026-06-23 16:21:13 | INFO | Processing Table-:Master_VisibilityReason | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 16:21:13 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason ..............
|
|
2026-06-23 16:21:13 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:21:13 | INFO | Fetching in progress ....
|
|
2026-06-23 16:21:13 | INFO | Fetched 17 rows from SQL Server
|
|
2026-06-23 16:21:13 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!!
|
|
2026-06-23 16:21:13 | INFO | Fetched 17 rows
|
|
2026-06-23 16:21:13 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-23 16:21:13 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:21:13 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-23 16:21:13 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-23 16:21:13 | INFO | ================================================================================
|
|
2026-06-23 16:21:13 | INFO | Processing Table-:Master_VisibilityDefinition | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 16:21:13 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition ..............
|
|
2026-06-23 16:21:13 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:21:13 | INFO | Fetching in progress ....
|
|
2026-06-23 16:21:14 | INFO | Fetched 861 rows from SQL Server
|
|
2026-06-23 16:21:14 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!!
|
|
2026-06-23 16:21:14 | INFO | Fetched 861 rows
|
|
2026-06-23 16:21:14 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-23 16:21:14 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:21:15 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-23 16:21:15 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-23 16:21:15 | INFO | ================================================================================
|
|
2026-06-23 16:21:15 | INFO | Processing Table-:Master_Salesterritorylayer | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 16:21:15 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer ..............
|
|
2026-06-23 16:21:15 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:21:15 | INFO | Fetching in progress ....
|
|
2026-06-23 16:21:15 | INFO | Fetched 33 rows from SQL Server
|
|
2026-06-23 16:21:15 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!!
|
|
2026-06-23 16:21:15 | INFO | Fetched 33 rows
|
|
2026-06-23 16:21:15 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-23 16:21:15 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:21:15 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-23 16:21:15 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-23 16:21:15 | INFO | ================================================================================
|
|
2026-06-23 16:21:15 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:21:15 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-23 16:21:15 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-23 16:21:45 | INFO | Fetched 291,599 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-23 16:21:45 | WARNING | Table 'OQaD' does not exist.
|
|
2026-06-23 16:21:45 | WARNING | OQaD Matched df in OQaD returned no rows
|
|
2026-06-23 16:21:45 | INFO | Fetched 1 matched empids fetched for OQAD
|
|
2026-06-23 16:21:45 | INFO | Exists: True
|
|
2026-06-23 16:21:45 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-23 16:21:45 | INFO | Fetching data for 1 EMPIDs
|
|
2026-06-23 16:21:45 | INFO | Fetching OQaD data for run_date=2026-06-22
|
|
2026-06-23 16:27:23 | INFO | fn name is fetch_OQad ------Fetched 291222 rows
|
|
2026-06-23 16:27:23 | INFO | Fetched 291,222 rows from SQL Server
|
|
2026-06-23 16:27:23 | INFO | Fetched total row -: 291222 from sql server for table-:OQaD ...........!!!
|
|
2026-06-23 16:27:23 | INFO | Fetched 291222 rows
|
|
2026-06-23 16:27:23 | INFO | Creating table OQaD
|
|
2026-06-23 16:27:23 | INFO | Table ready: OQaD
|
|
2026-06-23 16:27:23 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:27:26 | INFO | OQaD: inserted 291,222 rows into ClickHouse
|
|
2026-06-23 16:27:26 | INFO | OQaD loaded successfully (291222 rows)
|
|
2026-06-23 16:27:26 | INFO | ================================================================================
|
|
2026-06-23 16:27:26 | INFO | Pipeline Completed Successfully
|
|
2026-06-23 16:27:26 | INFO | ================================================================================
|
|
2026-06-23 16:27:26 | INFO | Pipeline completed successfully. last_successful_run_date=2026-06-23
|
|
2026-06-23 16:27:26 | INFO | ================================================================================
|
|
2026-06-23 16:27:26 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 16:27:26 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-23 16:27:26 | INFO | Connecting to databases...
|
|
2026-06-23 16:27:26 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B6784B16D0>
|
|
2026-06-23 16:27:27 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B6784B0410>
|
|
2026-06-23 16:27:27 | INFO | Database connections established
|
|
2026-06-23 16:27:27 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-23 16:27:27 | INFO | Found 857 MIDs
|
|
2026-06-23 16:27:28 | INFO | ================================================================================
|
|
2026-06-23 16:27:28 | INFO | Processing Table-:SOS_OneApp | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:27:28 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-23 16:27:28 | INFO | Start Fetching data for these 857 MIDs
|
|
2026-06-23 16:27:28 | INFO | Start Fetching data for these 857 MIDs or based on this date 2026-06-23
|
|
2026-06-23 16:27:28 | INFO | Fetching data for 857 MIDs
|
|
2026-06-23 16:27:30 | INFO | Fetched 1,748 rows from SQL Server
|
|
2026-06-23 16:27:30 | INFO | Fetched total row -: 1748 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-23 16:27:30 | INFO | Fetched 1748 rows
|
|
2026-06-23 16:27:31 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-23 16:27:31 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:27:31 | INFO | SOS_OneApp: inserted 1,748 rows into ClickHouse
|
|
2026-06-23 16:27:31 | INFO | SOS_OneApp loaded successfully (1748 rows)
|
|
2026-06-23 16:27:31 | INFO | ================================================================================
|
|
2026-06-23 16:27:31 | INFO | Processing Table-:Survey | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:27:31 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-23 16:27:31 | INFO | Start Fetching data for these 857 MIDs or based on this date 2026-06-23
|
|
2026-06-23 16:27:31 | INFO | Fetching data for 857 MIDs
|
|
2026-06-23 16:27:31 | INFO | Fetched 55 rows from SQL Server
|
|
2026-06-23 16:27:31 | INFO | Fetched total row -: 55 from sql server for table-:Survey ...........!!!
|
|
2026-06-23 16:27:31 | INFO | Fetched 55 rows
|
|
2026-06-23 16:27:31 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
|
|
2026-06-23 16:27:31 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:27:31 | INFO | Survey: inserted 55 rows into ClickHouse
|
|
2026-06-23 16:27:31 | INFO | Survey loaded successfully (55 rows)
|
|
2026-06-23 16:27:31 | INFO | ================================================================================
|
|
2026-06-23 16:27:31 | INFO | Processing Table-:additional_visibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:27:31 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-23 16:27:31 | INFO | Start Fetching data for these 857 MIDs
|
|
2026-06-23 16:27:31 | INFO | Start Fetching data for these 857 MIDs or based on this date 2026-06-23
|
|
2026-06-23 16:27:31 | INFO | Fetching data for 857 MIDs
|
|
2026-06-23 16:27:32 | INFO | Fetched 642 rows from SQL Server
|
|
2026-06-23 16:27:32 | INFO | Fetched total row -: 642 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-23 16:27:32 | INFO | Fetched 642 rows
|
|
2026-06-23 16:27:32 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
|
|
2026-06-23 16:27:32 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:27:33 | INFO | additional_visibility: inserted 642 rows into ClickHouse
|
|
2026-06-23 16:27:33 | INFO | additional_visibility loaded successfully (642 rows)
|
|
2026-06-23 16:27:33 | INFO | ================================================================================
|
|
2026-06-23 16:27:33 | INFO | Processing Table-:Coverage | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:27:33 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-23 16:27:33 | INFO | Start Fetching data for these 857 MIDs or based on this date 2026-06-23
|
|
2026-06-23 16:27:33 | INFO | Fetching data for 857 MIDs
|
|
2026-06-23 16:27:33 | INFO | Fetched 800 rows from SQL Server
|
|
2026-06-23 16:27:33 | INFO | Fetched total row -: 800 from sql server for table-:Coverage ...........!!!
|
|
2026-06-23 16:27:33 | INFO | Fetched 800 rows
|
|
2026-06-23 16:27:33 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
|
|
2026-06-23 16:27:34 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:27:34 | INFO | Coverage: inserted 800 rows into ClickHouse
|
|
2026-06-23 16:27:34 | INFO | Coverage loaded successfully (800 rows)
|
|
2026-06-23 16:27:34 | INFO | ================================================================================
|
|
2026-06-23 16:27:34 | INFO | Processing Table-:Login | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:27:34 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-23 16:27:34 | INFO | Start Fetching data for these 857 MIDs or based on this date 2026-06-23
|
|
2026-06-23 16:27:34 | INFO | Fetching data for 857 MIDs
|
|
2026-06-23 16:27:35 | INFO | Fetched 451 rows from SQL Server
|
|
2026-06-23 16:27:35 | INFO | Fetched total row -: 451 from sql server for table-:Login ...........!!!
|
|
2026-06-23 16:27:35 | INFO | Fetched 451 rows
|
|
2026-06-23 16:27:35 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _
|
|
2026-06-23 16:27:35 | INFO | No delete logic required for Login
|
|
2026-06-23 16:27:35 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:27:35 | INFO | Login: inserted 451 rows into ClickHouse
|
|
2026-06-23 16:27:35 | INFO | Login loaded successfully (451 rows)
|
|
2026-06-23 16:27:35 | INFO | ================================================================================
|
|
2026-06-23 16:27:35 | INFO | Processing Table-:Stock_Details | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:27:35 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-23 16:27:35 | INFO | Start Fetching data for these 857 MIDs or based on this date 2026-06-23
|
|
2026-06-23 16:27:35 | INFO | Fetching data for 857 MIDs
|
|
2026-06-23 16:27:42 | INFO | Fetched 19,243 rows from SQL Server
|
|
2026-06-23 16:27:42 | INFO | Fetched total row -: 19243 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-23 16:27:42 | INFO | Fetched 19243 rows
|
|
2026-06-23 16:27:43 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _
|
|
2026-06-23 16:27:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:27:43 | INFO | Stock_Details: inserted 19,243 rows into ClickHouse
|
|
2026-06-23 16:27:43 | INFO | Stock_Details loaded successfully (19243 rows)
|
|
2026-06-23 16:27:43 | INFO | ================================================================================
|
|
2026-06-23 16:27:43 | INFO | Processing Table-:Attendance | Table type -:FACT | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 16:27:43 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-23 16:27:43 | INFO | Fetching Attendance data from 2026-06-08 to 2026-06-23
|
|
2026-06-23 16:29:09 | INFO | Fetched 121,962 attendance rows for 542 employees
|
|
2026-06-23 16:29:09 | INFO | Fetched total row -: 121962 from sql server for table-:Attendance ...........!!!
|
|
2026-06-23 16:29:09 | INFO | Fetched 121962 rows
|
|
2026-06-23 16:29:10 | INFO | Truncated table Attendance
|
|
2026-06-23 16:29:10 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:29:11 | INFO | Attendance: inserted 121,962 rows into ClickHouse
|
|
2026-06-23 16:29:11 | INFO | Attendance loaded successfully (121962 rows)
|
|
2026-06-23 16:29:11 | INFO | ================================================================================
|
|
2026-06-23 16:29:11 | INFO | Processing Table-:Journey_Plan | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:29:11 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-23 16:29:11 | INFO | Start Fetching data for these 857 MIDs
|
|
2026-06-23 16:29:11 | INFO | Start Fetching data for these 857 MIDs or based on this date 2026-06-23
|
|
2026-06-23 16:29:11 | INFO | Fetching data for 857 MIDs
|
|
2026-06-23 16:29:15 | INFO | Fetched 21,737 rows from SQL Server
|
|
2026-06-23 16:29:15 | INFO | Fetched total row -: 21737 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-23 16:29:15 | INFO | Fetched 21737 rows
|
|
2026-06-23 16:29:15 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
|
|
2026-06-23 16:29:15 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:29:16 | INFO | Journey_Plan: inserted 21,737 rows into ClickHouse
|
|
2026-06-23 16:29:16 | INFO | Journey_Plan loaded successfully (21737 rows)
|
|
2026-06-23 16:29:16 | INFO | ================================================================================
|
|
2026-06-23 16:29:16 | INFO | Processing Table-:Web_Logins | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:29:16 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-23 16:29:16 | INFO | Start Fetching data for these 857 MIDs
|
|
2026-06-23 16:29:16 | INFO | Start Fetching data for these 857 MIDs or based on this date 2026-06-23
|
|
2026-06-23 16:29:16 | INFO | Fetching data for 857 MIDs
|
|
2026-06-23 16:29:16 | INFO | Fetched 109 rows from SQL Server
|
|
2026-06-23 16:29:16 | INFO | Fetched total row -: 109 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-23 16:29:16 | INFO | Fetched 109 rows
|
|
2026-06-23 16:29:16 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
|
|
2026-06-23 16:29:16 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:29:16 | INFO | Web_Logins: inserted 109 rows into ClickHouse
|
|
2026-06-23 16:29:16 | INFO | Web_Logins loaded successfully (109 rows)
|
|
2026-06-23 16:29:16 | INFO | ================================================================================
|
|
2026-06-23 16:29:16 | INFO | Processing Table-:Promotion | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:29:16 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-23 16:29:16 | INFO | Fetching data for 857 MIDs
|
|
2026-06-23 16:29:22 | INFO | Fetched 3,938 rows from SQL Server
|
|
2026-06-23 16:29:22 | INFO | Fetched total row -: 3938 from sql server for table-:Promotion ...........!!!
|
|
2026-06-23 16:29:22 | INFO | Fetched 3938 rows
|
|
2026-06-23 16:29:22 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-23 16:29:22 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:29:23 | INFO | Promotion: inserted 3,938 rows into ClickHouse
|
|
2026-06-23 16:29:23 | INFO | Promotion loaded successfully (3938 rows)
|
|
2026-06-23 16:29:23 | INFO | ================================================================================
|
|
2026-06-23 16:29:23 | INFO | Processing Table-:PaidVisibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 16:29:23 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
|
|
2026-06-23 16:29:23 | INFO | Start Fetching data for these 857 MIDs
|
|
2026-06-23 16:29:23 | INFO | Start Fetching data for these 857 MIDs or based on this date 2026-06-23
|
|
2026-06-23 16:29:23 | INFO | Fetching data for 857 MIDs
|
|
2026-06-23 16:29:25 | INFO | Fetched 236 rows from SQL Server
|
|
2026-06-23 16:29:25 | INFO | Fetched total row -: 236 from sql server for table-:PaidVisibility ...........!!!
|
|
2026-06-23 16:29:25 | INFO | Fetched 236 rows
|
|
2026-06-23 16:29:25 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _
|
|
2026-06-23 16:29:25 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:29:25 | INFO | PaidVisibility: inserted 236 rows into ClickHouse
|
|
2026-06-23 16:29:25 | INFO | PaidVisibility loaded successfully (236 rows)
|
|
2026-06-23 16:29:25 | INFO | ================================================================================
|
|
2026-06-23 16:29:25 | INFO | Processing Table-:Store_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:29:25 | INFO | Fetching Data from sql server for table-: Store_Master ..............
|
|
2026-06-23 16:29:25 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:29:25 | INFO | Fetching in progress ....
|
|
2026-06-23 16:29:38 | INFO | Fetched 5,998 rows from SQL Server
|
|
2026-06-23 16:29:38 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
|
|
2026-06-23 16:29:38 | INFO | Fetched 5998 rows
|
|
2026-06-23 16:29:38 | INFO | Truncated table Store_Master
|
|
2026-06-23 16:29:38 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:29:39 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
|
|
2026-06-23 16:29:39 | INFO | Store_Master loaded successfully (5998 rows)
|
|
2026-06-23 16:29:39 | INFO | ================================================================================
|
|
2026-06-23 16:29:39 | INFO | Processing Table-:SKU_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:29:39 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
|
|
2026-06-23 16:29:39 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:29:39 | INFO | Fetching in progress ....
|
|
2026-06-23 16:29:39 | INFO | Fetched 160 rows from SQL Server
|
|
2026-06-23 16:29:39 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
|
|
2026-06-23 16:29:39 | INFO | Fetched 160 rows
|
|
2026-06-23 16:29:39 | INFO | Truncated table SKU_Master
|
|
2026-06-23 16:29:39 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:29:40 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-23 16:29:40 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-23 16:29:40 | INFO | ================================================================================
|
|
2026-06-23 16:29:40 | INFO | Processing Table-:display_master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:29:40 | INFO | Fetching Data from sql server for table-: display_master ..............
|
|
2026-06-23 16:29:40 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:29:40 | INFO | Fetching in progress ....
|
|
2026-06-23 16:29:40 | INFO | Fetched 135 rows from SQL Server
|
|
2026-06-23 16:29:40 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
|
|
2026-06-23 16:29:40 | INFO | Fetched 135 rows
|
|
2026-06-23 16:29:40 | INFO | Truncated table display_master
|
|
2026-06-23 16:29:40 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:29:40 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-23 16:29:40 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-23 16:29:40 | INFO | ================================================================================
|
|
2026-06-23 16:29:40 | INFO | Processing Table-:Employee_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 16:29:40 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
|
|
2026-06-23 16:29:40 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:29:40 | INFO | Fetching in progress ....
|
|
2026-06-23 16:29:42 | INFO | Fetched 2,289 rows from SQL Server
|
|
2026-06-23 16:29:42 | INFO | Fetched total row -: 2289 from sql server for table-:Employee_Master ...........!!!
|
|
2026-06-23 16:29:42 | INFO | Fetched 2289 rows
|
|
2026-06-23 16:29:42 | INFO | Truncated table Employee_Master
|
|
2026-06-23 16:29:42 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:29:43 | INFO | Employee_Master: inserted 2,289 rows into ClickHouse
|
|
2026-06-23 16:29:43 | INFO | Employee_Master loaded successfully (2289 rows)
|
|
2026-06-23 16:29:43 | INFO | ================================================================================
|
|
2026-06-23 16:29:43 | INFO | Processing Table-:coverage_remarks | Table type -:DIMENSION | fetcht by-:reason_id | operation-:DELETE+INSERT
|
|
2026-06-23 16:29:43 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
|
|
2026-06-23 16:29:43 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:29:43 | INFO | Fetching in progress ....
|
|
2026-06-23 16:29:43 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 16:29:43 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 16:29:43 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
|
|
2026-06-23 16:29:43 | WARNING | coverage_remarks returned no rows
|
|
2026-06-23 16:29:43 | INFO | ================================================================================
|
|
2026-06-23 16:29:43 | INFO | Processing Table-:mapping_storevisibility | Table type -:BRIDGE | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 16:29:43 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
|
|
2026-06-23 16:29:43 | INFO | Fetching data from sql server for BRIDGE table......
|
|
2026-06-23 16:29:43 | INFO | Fetching in progress ....
|
|
2026-06-23 16:29:43 | INFO | Fetched 805 rows from SQL Server
|
|
2026-06-23 16:29:43 | INFO | Fetched total row -: 805 from sql server for table-:mapping_storevisibility ...........!!!
|
|
2026-06-23 16:29:43 | INFO | Fetched 805 rows
|
|
2026-06-23 16:29:44 | INFO | Truncated table mapping_storevisibility
|
|
2026-06-23 16:29:44 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:29:44 | INFO | mapping_storevisibility: inserted 805 rows into ClickHouse
|
|
2026-06-23 16:29:44 | INFO | mapping_storevisibility loaded successfully (805 rows)
|
|
2026-06-23 16:29:44 | INFO | ================================================================================
|
|
2026-06-23 16:29:44 | INFO | Processing Table-:Master_VisibilityReason | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 16:29:44 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason ..............
|
|
2026-06-23 16:29:44 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:29:44 | INFO | Fetching in progress ....
|
|
2026-06-23 16:29:44 | INFO | Fetched 17 rows from SQL Server
|
|
2026-06-23 16:29:44 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!!
|
|
2026-06-23 16:29:44 | INFO | Fetched 17 rows
|
|
2026-06-23 16:29:44 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-23 16:29:44 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:29:44 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-23 16:29:44 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-23 16:29:44 | INFO | ================================================================================
|
|
2026-06-23 16:29:44 | INFO | Processing Table-:Master_VisibilityDefinition | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 16:29:44 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition ..............
|
|
2026-06-23 16:29:44 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:29:44 | INFO | Fetching in progress ....
|
|
2026-06-23 16:29:44 | INFO | Fetched 861 rows from SQL Server
|
|
2026-06-23 16:29:44 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!!
|
|
2026-06-23 16:29:44 | INFO | Fetched 861 rows
|
|
2026-06-23 16:29:44 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-23 16:29:44 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:29:45 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-23 16:29:45 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-23 16:29:45 | INFO | ================================================================================
|
|
2026-06-23 16:29:45 | INFO | Processing Table-:Master_Salesterritorylayer | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 16:29:45 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer ..............
|
|
2026-06-23 16:29:45 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 16:29:45 | INFO | Fetching in progress ....
|
|
2026-06-23 16:29:45 | INFO | Fetched 33 rows from SQL Server
|
|
2026-06-23 16:29:45 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!!
|
|
2026-06-23 16:29:45 | INFO | Fetched 33 rows
|
|
2026-06-23 16:29:45 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-23 16:29:45 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 16:29:45 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-23 16:29:45 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-23 16:29:45 | INFO | ================================================================================
|
|
2026-06-23 16:29:45 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 16:29:45 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-23 16:29:45 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-23 16:30:04 | INFO | Fetched 291,600 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-23 16:30:05 | INFO | Fetched 291,165 matched empids fetched for OQAD
|
|
2026-06-23 16:30:05 | INFO | Exists: True
|
|
2026-06-23 16:30:05 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-23 16:30:05 | INFO | Fetching data for 291,165 EMPIDs
|
|
2026-06-23 16:30:05 | INFO | Fetching OQaD data for run_date=2026-06-23
|
|
2026-06-23 16:30:53 | INFO | fn name is fetch_OQad ------Fetched 0 rows
|
|
2026-06-23 16:30:53 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 16:30:53 | INFO | Fetched total row -: 0 from sql server for table-:OQaD ...........!!!
|
|
2026-06-23 16:30:53 | WARNING | OQaD returned no rows
|
|
2026-06-23 16:30:53 | INFO | ================================================================================
|
|
2026-06-23 16:30:53 | INFO | Pipeline Completed Successfully
|
|
2026-06-23 16:30:53 | INFO | ================================================================================
|
|
2026-06-23 16:30:53 | INFO | Pipeline completed successfully. last_successful_run_date=2026-06-23
|
|
2026-06-23 17:07:25 | INFO | Pipeline Start Date: 2026-06-22
|
|
2026-06-23 17:07:25 | INFO | ================================================================================
|
|
2026-06-23 17:07:25 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 17:07:25 | INFO | Pipeline Run Date: 2026-06-22
|
|
2026-06-23 17:07:25 | INFO | Connecting to databases...
|
|
2026-06-23 17:07:25 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000025BFFBB0730>
|
|
2026-06-23 17:07:27 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000025B80DC87D0>
|
|
2026-06-23 17:07:28 | INFO | Database connections established
|
|
2026-06-23 17:07:28 | INFO | Collecting MIDs for: 2026-06-22
|
|
2026-06-23 17:07:28 | INFO | Found 882 MIDs
|
|
2026-06-23 17:07:29 | INFO | ================================================================================
|
|
2026-06-23 17:07:29 | INFO | Processing Table-:SOS_OneApp | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:07:29 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-23 17:07:29 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 17:07:29 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 17:07:29 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 17:07:34 | INFO | Fetched 3,888 rows from SQL Server
|
|
2026-06-23 17:07:34 | INFO | Fetched total row -: 3888 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-23 17:07:34 | INFO | Fetched 3888 rows
|
|
2026-06-23 17:07:34 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-23 17:07:34 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:07:34 | INFO | SOS_OneApp: inserted 3,888 rows into ClickHouse
|
|
2026-06-23 17:07:34 | INFO | SOS_OneApp loaded successfully (3888 rows)
|
|
2026-06-23 17:07:34 | INFO | ================================================================================
|
|
2026-06-23 17:07:34 | INFO | Processing Table-:Survey | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:07:34 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-23 17:07:34 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 17:07:34 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 17:07:35 | INFO | Fetched 152 rows from SQL Server
|
|
2026-06-23 17:07:35 | INFO | Fetched total row -: 152 from sql server for table-:Survey ...........!!!
|
|
2026-06-23 17:07:35 | INFO | Fetched 152 rows
|
|
2026-06-23 17:07:35 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
|
|
2026-06-23 17:07:35 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:07:35 | INFO | Survey: inserted 152 rows into ClickHouse
|
|
2026-06-23 17:07:35 | INFO | Survey loaded successfully (152 rows)
|
|
2026-06-23 17:07:35 | INFO | ================================================================================
|
|
2026-06-23 17:07:35 | INFO | Processing Table-:additional_visibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:07:35 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-23 17:07:35 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 17:07:35 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 17:07:35 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 17:07:36 | INFO | Fetched 2,136 rows from SQL Server
|
|
2026-06-23 17:07:36 | INFO | Fetched total row -: 2136 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-23 17:07:36 | INFO | Fetched 2136 rows
|
|
2026-06-23 17:07:36 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
|
|
2026-06-23 17:07:36 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:07:36 | INFO | additional_visibility: inserted 2,136 rows into ClickHouse
|
|
2026-06-23 17:07:36 | INFO | additional_visibility loaded successfully (2136 rows)
|
|
2026-06-23 17:07:36 | INFO | ================================================================================
|
|
2026-06-23 17:07:36 | INFO | Processing Table-:Coverage | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:07:36 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-23 17:07:36 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 17:07:36 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 17:07:47 | INFO | Fetched 825 rows from SQL Server
|
|
2026-06-23 17:07:47 | INFO | Fetched total row -: 825 from sql server for table-:Coverage ...........!!!
|
|
2026-06-23 17:07:47 | INFO | Fetched 825 rows
|
|
2026-06-23 17:07:47 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
|
|
2026-06-23 17:07:47 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:07:48 | INFO | Coverage: inserted 825 rows into ClickHouse
|
|
2026-06-23 17:07:48 | INFO | Coverage loaded successfully (825 rows)
|
|
2026-06-23 17:07:48 | INFO | ================================================================================
|
|
2026-06-23 17:07:48 | INFO | Processing Table-:Login | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 17:07:48 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-23 17:07:48 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 17:07:48 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 17:07:50 | INFO | Fetched 501 rows from SQL Server
|
|
2026-06-23 17:07:50 | INFO | Fetched total row -: 501 from sql server for table-:Login ...........!!!
|
|
2026-06-23 17:07:50 | INFO | Fetched 501 rows
|
|
2026-06-23 17:07:50 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _
|
|
2026-06-23 17:07:50 | INFO | No delete logic required for Login
|
|
2026-06-23 17:07:50 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:07:50 | INFO | Login: inserted 501 rows into ClickHouse
|
|
2026-06-23 17:07:50 | INFO | Login loaded successfully (501 rows)
|
|
2026-06-23 17:07:50 | INFO | ================================================================================
|
|
2026-06-23 17:07:50 | INFO | Processing Table-:Stock_Details | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:07:50 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-23 17:07:50 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 17:07:50 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 17:07:53 | INFO | Fetched 44,061 rows from SQL Server
|
|
2026-06-23 17:07:53 | INFO | Fetched total row -: 44061 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-23 17:07:53 | INFO | Fetched 44061 rows
|
|
2026-06-23 17:07:54 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _
|
|
2026-06-23 17:07:54 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:07:54 | INFO | Stock_Details: inserted 44,061 rows into ClickHouse
|
|
2026-06-23 17:07:54 | INFO | Stock_Details loaded successfully (44061 rows)
|
|
2026-06-23 17:07:54 | INFO | ================================================================================
|
|
2026-06-23 17:07:54 | INFO | Processing Table-:Attendance | Table type -:FACT | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 17:07:54 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-23 17:07:54 | INFO | Fetching Attendance data from 2026-06-07 to 2026-06-22
|
|
2026-06-23 17:08:03 | INFO | Fetched 121,132 attendance rows for 541 employees
|
|
2026-06-23 17:08:03 | INFO | Fetched total row -: 121132 from sql server for table-:Attendance ...........!!!
|
|
2026-06-23 17:08:03 | INFO | Fetched 121132 rows
|
|
2026-06-23 17:08:03 | INFO | Truncated table Attendance
|
|
2026-06-23 17:08:03 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:08:04 | INFO | Attendance: inserted 121,132 rows into ClickHouse
|
|
2026-06-23 17:08:04 | INFO | Attendance loaded successfully (121132 rows)
|
|
2026-06-23 17:08:04 | INFO | ================================================================================
|
|
2026-06-23 17:08:04 | INFO | Processing Table-:Journey_Plan | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 17:08:04 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-23 17:08:04 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 17:08:04 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 17:08:04 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 17:08:05 | INFO | Fetched 21,739 rows from SQL Server
|
|
2026-06-23 17:08:05 | INFO | Fetched total row -: 21739 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-23 17:08:05 | INFO | Fetched 21739 rows
|
|
2026-06-23 17:08:05 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
|
|
2026-06-23 17:08:05 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:08:06 | INFO | Journey_Plan: inserted 21,739 rows into ClickHouse
|
|
2026-06-23 17:08:06 | INFO | Journey_Plan loaded successfully (21739 rows)
|
|
2026-06-23 17:08:06 | INFO | ================================================================================
|
|
2026-06-23 17:08:06 | INFO | Processing Table-:Web_Logins | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 17:08:06 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-23 17:08:06 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 17:08:06 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 17:08:06 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 17:08:06 | INFO | Fetched 214 rows from SQL Server
|
|
2026-06-23 17:08:06 | INFO | Fetched total row -: 214 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-23 17:08:06 | INFO | Fetched 214 rows
|
|
2026-06-23 17:08:06 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
|
|
2026-06-23 17:08:06 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:08:06 | INFO | Web_Logins: inserted 214 rows into ClickHouse
|
|
2026-06-23 17:08:06 | INFO | Web_Logins loaded successfully (214 rows)
|
|
2026-06-23 17:08:06 | INFO | ================================================================================
|
|
2026-06-23 17:08:06 | INFO | Processing Table-:Promotion | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:08:06 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-23 17:08:06 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 17:08:09 | INFO | Fetched 8,835 rows from SQL Server
|
|
2026-06-23 17:08:09 | INFO | Fetched total row -: 8835 from sql server for table-:Promotion ...........!!!
|
|
2026-06-23 17:08:09 | INFO | Fetched 8835 rows
|
|
2026-06-23 17:08:09 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-23 17:08:09 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:08:10 | INFO | Promotion: inserted 8,835 rows into ClickHouse
|
|
2026-06-23 17:08:10 | INFO | Promotion loaded successfully (8835 rows)
|
|
2026-06-23 17:08:10 | INFO | ================================================================================
|
|
2026-06-23 17:08:10 | INFO | Processing Table-:PaidVisibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:08:10 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
|
|
2026-06-23 17:08:10 | INFO | Start Fetching data for these 882 MIDs
|
|
2026-06-23 17:08:10 | INFO | Start Fetching data for these 882 MIDs or based on this date 2026-06-22
|
|
2026-06-23 17:08:10 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 17:08:11 | INFO | Fetched 1,029 rows from SQL Server
|
|
2026-06-23 17:08:11 | INFO | Fetched total row -: 1029 from sql server for table-:PaidVisibility ...........!!!
|
|
2026-06-23 17:08:11 | INFO | Fetched 1029 rows
|
|
2026-06-23 17:08:11 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _
|
|
2026-06-23 17:08:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:08:11 | INFO | PaidVisibility: inserted 1,029 rows into ClickHouse
|
|
2026-06-23 17:08:11 | INFO | PaidVisibility loaded successfully (1029 rows)
|
|
2026-06-23 17:08:11 | INFO | ================================================================================
|
|
2026-06-23 17:08:11 | INFO | Processing Table-:Store_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 17:08:11 | INFO | Fetching Data from sql server for table-: Store_Master ..............
|
|
2026-06-23 17:08:11 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:08:11 | INFO | Fetching in progress ....
|
|
2026-06-23 17:08:17 | INFO | Fetched 5,998 rows from SQL Server
|
|
2026-06-23 17:08:17 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
|
|
2026-06-23 17:08:17 | INFO | Fetched 5998 rows
|
|
2026-06-23 17:08:17 | INFO | Truncated table Store_Master
|
|
2026-06-23 17:08:17 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:08:18 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
|
|
2026-06-23 17:08:18 | INFO | Store_Master loaded successfully (5998 rows)
|
|
2026-06-23 17:08:18 | INFO | ================================================================================
|
|
2026-06-23 17:08:18 | INFO | Processing Table-:SKU_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 17:08:18 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
|
|
2026-06-23 17:08:18 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:08:18 | INFO | Fetching in progress ....
|
|
2026-06-23 17:08:18 | INFO | Fetched 160 rows from SQL Server
|
|
2026-06-23 17:08:18 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
|
|
2026-06-23 17:08:18 | INFO | Fetched 160 rows
|
|
2026-06-23 17:08:18 | INFO | Truncated table SKU_Master
|
|
2026-06-23 17:08:18 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:08:18 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-23 17:08:18 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-23 17:08:18 | INFO | ================================================================================
|
|
2026-06-23 17:08:18 | INFO | Processing Table-:display_master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 17:08:18 | INFO | Fetching Data from sql server for table-: display_master ..............
|
|
2026-06-23 17:08:18 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:08:18 | INFO | Fetching in progress ....
|
|
2026-06-23 17:08:18 | INFO | Fetched 135 rows from SQL Server
|
|
2026-06-23 17:08:18 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
|
|
2026-06-23 17:08:18 | INFO | Fetched 135 rows
|
|
2026-06-23 17:08:18 | INFO | Truncated table display_master
|
|
2026-06-23 17:08:18 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:08:19 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-23 17:08:19 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-23 17:08:19 | INFO | ================================================================================
|
|
2026-06-23 17:08:19 | INFO | Processing Table-:Employee_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 17:08:19 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
|
|
2026-06-23 17:08:19 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:08:19 | INFO | Fetching in progress ....
|
|
2026-06-23 17:08:20 | INFO | Fetched 2,289 rows from SQL Server
|
|
2026-06-23 17:08:20 | INFO | Fetched total row -: 2289 from sql server for table-:Employee_Master ...........!!!
|
|
2026-06-23 17:08:20 | INFO | Fetched 2289 rows
|
|
2026-06-23 17:08:20 | INFO | Truncated table Employee_Master
|
|
2026-06-23 17:08:20 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:08:20 | INFO | Employee_Master: inserted 2,289 rows into ClickHouse
|
|
2026-06-23 17:08:20 | INFO | Employee_Master loaded successfully (2289 rows)
|
|
2026-06-23 17:08:20 | INFO | ================================================================================
|
|
2026-06-23 17:08:20 | INFO | Processing Table-:coverage_remarks | Table type -:DIMENSION | fetcht by-:reason_id | operation-:DELETE+INSERT
|
|
2026-06-23 17:08:20 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
|
|
2026-06-23 17:08:21 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:08:21 | INFO | Fetching in progress ....
|
|
2026-06-23 17:08:21 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 17:08:21 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 17:08:21 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
|
|
2026-06-23 17:08:21 | WARNING | coverage_remarks returned no rows
|
|
2026-06-23 17:08:21 | INFO | ================================================================================
|
|
2026-06-23 17:08:21 | INFO | Processing Table-:mapping_storevisibility | Table type -:BRIDGE | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 17:08:21 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
|
|
2026-06-23 17:08:21 | INFO | Fetching data from sql server for BRIDGE table......
|
|
2026-06-23 17:08:21 | INFO | Fetching in progress ....
|
|
2026-06-23 17:08:21 | INFO | Fetched 805 rows from SQL Server
|
|
2026-06-23 17:08:21 | INFO | Fetched total row -: 805 from sql server for table-:mapping_storevisibility ...........!!!
|
|
2026-06-23 17:08:21 | INFO | Fetched 805 rows
|
|
2026-06-23 17:08:22 | INFO | Truncated table mapping_storevisibility
|
|
2026-06-23 17:08:22 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:08:22 | INFO | mapping_storevisibility: inserted 805 rows into ClickHouse
|
|
2026-06-23 17:08:22 | INFO | mapping_storevisibility loaded successfully (805 rows)
|
|
2026-06-23 17:08:22 | INFO | ================================================================================
|
|
2026-06-23 17:08:22 | INFO | Processing Table-:Master_VisibilityReason | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 17:08:22 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason ..............
|
|
2026-06-23 17:08:22 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:08:22 | INFO | Fetching in progress ....
|
|
2026-06-23 17:08:22 | INFO | Fetched 17 rows from SQL Server
|
|
2026-06-23 17:08:22 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!!
|
|
2026-06-23 17:08:22 | INFO | Fetched 17 rows
|
|
2026-06-23 17:08:22 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-23 17:08:22 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:08:22 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-23 17:08:22 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-23 17:08:22 | INFO | ================================================================================
|
|
2026-06-23 17:08:22 | INFO | Processing Table-:Master_VisibilityDefinition | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 17:08:22 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition ..............
|
|
2026-06-23 17:08:22 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:08:22 | INFO | Fetching in progress ....
|
|
2026-06-23 17:08:22 | INFO | Fetched 861 rows from SQL Server
|
|
2026-06-23 17:08:22 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!!
|
|
2026-06-23 17:08:22 | INFO | Fetched 861 rows
|
|
2026-06-23 17:08:22 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-23 17:08:22 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:08:22 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-23 17:08:22 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-23 17:08:22 | INFO | ================================================================================
|
|
2026-06-23 17:08:22 | INFO | Processing Table-:Master_Salesterritorylayer | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 17:08:22 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer ..............
|
|
2026-06-23 17:08:22 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:08:22 | INFO | Fetching in progress ....
|
|
2026-06-23 17:08:23 | INFO | Fetched 33 rows from SQL Server
|
|
2026-06-23 17:08:23 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!!
|
|
2026-06-23 17:08:23 | INFO | Fetched 33 rows
|
|
2026-06-23 17:08:23 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-23 17:08:23 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:08:23 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-23 17:08:23 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-23 17:08:23 | INFO | ================================================================================
|
|
2026-06-23 17:08:23 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 17:08:23 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-23 17:08:23 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-23 17:08:35 | INFO | Fetched 291,607 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-23 17:08:36 | INFO | Fetched 291,165 matched empids fetched for OQAD
|
|
2026-06-23 17:08:36 | INFO | Exists: True
|
|
2026-06-23 17:08:36 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-23 17:08:36 | INFO | Fetching data for 291,165 EMPIDs
|
|
2026-06-23 17:08:36 | INFO | Fetching OQaD data for run_date=2026-06-22
|
|
2026-06-23 17:09:14 | INFO | fn name is fetch_OQad ------Fetched 49 rows
|
|
2026-06-23 17:09:14 | INFO | Fetched 49 rows from SQL Server
|
|
2026-06-23 17:09:14 | INFO | Fetched total row -: 49 from sql server for table-:OQaD ...........!!!
|
|
2026-06-23 17:09:14 | INFO | Fetched 49 rows
|
|
2026-06-23 17:09:14 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
|
|
2026-06-23 17:09:15 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:09:15 | INFO | OQaD: inserted 49 rows into ClickHouse
|
|
2026-06-23 17:09:15 | INFO | OQaD loaded successfully (49 rows)
|
|
2026-06-23 17:09:15 | INFO | ================================================================================
|
|
2026-06-23 17:09:15 | INFO | Pipeline Completed Successfully
|
|
2026-06-23 17:09:15 | INFO | ================================================================================
|
|
2026-06-23 17:09:15 | INFO | Pipeline completed successfully. last_successful_run_date=2026-06-23
|
|
2026-06-23 17:09:15 | INFO | ================================================================================
|
|
2026-06-23 17:09:15 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 17:09:15 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-23 17:09:15 | INFO | Connecting to databases...
|
|
2026-06-23 17:09:15 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000025B80DD14F0>
|
|
2026-06-23 17:09:19 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000025B80DD1E50>
|
|
2026-06-23 17:09:20 | INFO | Database connections established
|
|
2026-06-23 17:09:20 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-23 17:09:20 | INFO | Found 860 MIDs
|
|
2026-06-23 17:09:21 | INFO | ================================================================================
|
|
2026-06-23 17:09:21 | INFO | Processing Table-:SOS_OneApp | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:09:21 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-23 17:09:21 | INFO | Start Fetching data for these 860 MIDs
|
|
2026-06-23 17:09:21 | INFO | Start Fetching data for these 860 MIDs or based on this date 2026-06-23
|
|
2026-06-23 17:09:21 | INFO | Fetching data for 860 MIDs
|
|
2026-06-23 17:09:29 | INFO | Fetched 1,965 rows from SQL Server
|
|
2026-06-23 17:09:29 | INFO | Fetched total row -: 1965 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-23 17:09:29 | INFO | Fetched 1965 rows
|
|
2026-06-23 17:09:29 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-23 17:09:29 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:09:29 | INFO | SOS_OneApp: inserted 1,965 rows into ClickHouse
|
|
2026-06-23 17:09:29 | INFO | SOS_OneApp loaded successfully (1965 rows)
|
|
2026-06-23 17:09:29 | INFO | ================================================================================
|
|
2026-06-23 17:09:29 | INFO | Processing Table-:Survey | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:09:29 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-23 17:09:29 | INFO | Start Fetching data for these 860 MIDs or based on this date 2026-06-23
|
|
2026-06-23 17:09:29 | INFO | Fetching data for 860 MIDs
|
|
2026-06-23 17:09:30 | INFO | Fetched 60 rows from SQL Server
|
|
2026-06-23 17:09:30 | INFO | Fetched total row -: 60 from sql server for table-:Survey ...........!!!
|
|
2026-06-23 17:09:30 | INFO | Fetched 60 rows
|
|
2026-06-23 17:09:30 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
|
|
2026-06-23 17:09:30 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:09:30 | INFO | Survey: inserted 60 rows into ClickHouse
|
|
2026-06-23 17:09:30 | INFO | Survey loaded successfully (60 rows)
|
|
2026-06-23 17:09:30 | INFO | ================================================================================
|
|
2026-06-23 17:09:30 | INFO | Processing Table-:additional_visibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:09:30 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-23 17:09:30 | INFO | Start Fetching data for these 860 MIDs
|
|
2026-06-23 17:09:30 | INFO | Start Fetching data for these 860 MIDs or based on this date 2026-06-23
|
|
2026-06-23 17:09:30 | INFO | Fetching data for 860 MIDs
|
|
2026-06-23 17:09:31 | INFO | Fetched 752 rows from SQL Server
|
|
2026-06-23 17:09:31 | INFO | Fetched total row -: 752 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-23 17:09:31 | INFO | Fetched 752 rows
|
|
2026-06-23 17:09:32 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
|
|
2026-06-23 17:09:32 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:09:32 | INFO | additional_visibility: inserted 752 rows into ClickHouse
|
|
2026-06-23 17:09:32 | INFO | additional_visibility loaded successfully (752 rows)
|
|
2026-06-23 17:09:32 | INFO | ================================================================================
|
|
2026-06-23 17:09:32 | INFO | Processing Table-:Coverage | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:09:32 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-23 17:09:32 | INFO | Start Fetching data for these 860 MIDs or based on this date 2026-06-23
|
|
2026-06-23 17:09:32 | INFO | Fetching data for 860 MIDs
|
|
2026-06-23 17:09:33 | INFO | Fetched 803 rows from SQL Server
|
|
2026-06-23 17:09:33 | INFO | Fetched total row -: 803 from sql server for table-:Coverage ...........!!!
|
|
2026-06-23 17:09:33 | INFO | Fetched 803 rows
|
|
2026-06-23 17:09:33 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
|
|
2026-06-23 17:09:33 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:09:33 | INFO | Coverage: inserted 803 rows into ClickHouse
|
|
2026-06-23 17:09:33 | INFO | Coverage loaded successfully (803 rows)
|
|
2026-06-23 17:09:33 | INFO | ================================================================================
|
|
2026-06-23 17:09:33 | INFO | Processing Table-:Login | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 17:09:33 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-23 17:09:33 | INFO | Start Fetching data for these 860 MIDs or based on this date 2026-06-23
|
|
2026-06-23 17:09:33 | INFO | Fetching data for 860 MIDs
|
|
2026-06-23 17:09:35 | INFO | Fetched 457 rows from SQL Server
|
|
2026-06-23 17:09:35 | INFO | Fetched total row -: 457 from sql server for table-:Login ...........!!!
|
|
2026-06-23 17:09:35 | INFO | Fetched 457 rows
|
|
2026-06-23 17:09:35 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _
|
|
2026-06-23 17:09:35 | INFO | No delete logic required for Login
|
|
2026-06-23 17:09:35 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:09:35 | INFO | Login: inserted 457 rows into ClickHouse
|
|
2026-06-23 17:09:35 | INFO | Login loaded successfully (457 rows)
|
|
2026-06-23 17:09:35 | INFO | ================================================================================
|
|
2026-06-23 17:09:35 | INFO | Processing Table-:Stock_Details | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:09:35 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-23 17:09:35 | INFO | Start Fetching data for these 860 MIDs or based on this date 2026-06-23
|
|
2026-06-23 17:09:35 | INFO | Fetching data for 860 MIDs
|
|
2026-06-23 17:09:59 | INFO | Fetched 21,429 rows from SQL Server
|
|
2026-06-23 17:09:59 | INFO | Fetched total row -: 21429 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-23 17:09:59 | INFO | Fetched 21429 rows
|
|
2026-06-23 17:09:59 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _
|
|
2026-06-23 17:09:59 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:10:00 | INFO | Stock_Details: inserted 21,429 rows into ClickHouse
|
|
2026-06-23 17:10:00 | INFO | Stock_Details loaded successfully (21429 rows)
|
|
2026-06-23 17:10:00 | INFO | ================================================================================
|
|
2026-06-23 17:10:00 | INFO | Processing Table-:Attendance | Table type -:FACT | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 17:10:00 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-23 17:10:00 | INFO | Fetching Attendance data from 2026-06-08 to 2026-06-23
|
|
2026-06-23 17:12:41 | INFO | Fetched 121,974 attendance rows for 541 employees
|
|
2026-06-23 17:12:41 | INFO | Fetched total row -: 121974 from sql server for table-:Attendance ...........!!!
|
|
2026-06-23 17:12:41 | INFO | Fetched 121974 rows
|
|
2026-06-23 17:12:42 | INFO | Truncated table Attendance
|
|
2026-06-23 17:12:42 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:12:43 | INFO | Attendance: inserted 121,974 rows into ClickHouse
|
|
2026-06-23 17:12:43 | INFO | Attendance loaded successfully (121974 rows)
|
|
2026-06-23 17:12:43 | INFO | ================================================================================
|
|
2026-06-23 17:12:43 | INFO | Processing Table-:Journey_Plan | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 17:12:43 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-23 17:12:43 | INFO | Start Fetching data for these 860 MIDs
|
|
2026-06-23 17:12:43 | INFO | Start Fetching data for these 860 MIDs or based on this date 2026-06-23
|
|
2026-06-23 17:12:43 | INFO | Fetching data for 860 MIDs
|
|
2026-06-23 17:12:46 | INFO | Fetched 21,739 rows from SQL Server
|
|
2026-06-23 17:12:46 | INFO | Fetched total row -: 21739 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-23 17:12:46 | INFO | Fetched 21739 rows
|
|
2026-06-23 17:12:46 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
|
|
2026-06-23 17:12:46 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:12:46 | INFO | Journey_Plan: inserted 21,739 rows into ClickHouse
|
|
2026-06-23 17:12:46 | INFO | Journey_Plan loaded successfully (21739 rows)
|
|
2026-06-23 17:12:46 | INFO | ================================================================================
|
|
2026-06-23 17:12:46 | INFO | Processing Table-:Web_Logins | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 17:12:46 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-23 17:12:46 | INFO | Start Fetching data for these 860 MIDs
|
|
2026-06-23 17:12:46 | INFO | Start Fetching data for these 860 MIDs or based on this date 2026-06-23
|
|
2026-06-23 17:12:46 | INFO | Fetching data for 860 MIDs
|
|
2026-06-23 17:12:47 | INFO | Fetched 109 rows from SQL Server
|
|
2026-06-23 17:12:47 | INFO | Fetched total row -: 109 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-23 17:12:47 | INFO | Fetched 109 rows
|
|
2026-06-23 17:12:47 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
|
|
2026-06-23 17:12:47 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:12:47 | INFO | Web_Logins: inserted 109 rows into ClickHouse
|
|
2026-06-23 17:12:47 | INFO | Web_Logins loaded successfully (109 rows)
|
|
2026-06-23 17:12:47 | INFO | ================================================================================
|
|
2026-06-23 17:12:47 | INFO | Processing Table-:Promotion | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:12:47 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-23 17:12:47 | INFO | Fetching data for 860 MIDs
|
|
2026-06-23 17:12:58 | INFO | Fetched 4,595 rows from SQL Server
|
|
2026-06-23 17:12:58 | INFO | Fetched total row -: 4595 from sql server for table-:Promotion ...........!!!
|
|
2026-06-23 17:12:58 | INFO | Fetched 4595 rows
|
|
2026-06-23 17:12:58 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-23 17:12:58 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:12:58 | INFO | Promotion: inserted 4,595 rows into ClickHouse
|
|
2026-06-23 17:12:58 | INFO | Promotion loaded successfully (4595 rows)
|
|
2026-06-23 17:12:58 | INFO | ================================================================================
|
|
2026-06-23 17:12:58 | INFO | Processing Table-:PaidVisibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 17:12:58 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
|
|
2026-06-23 17:12:58 | INFO | Start Fetching data for these 860 MIDs
|
|
2026-06-23 17:12:58 | INFO | Start Fetching data for these 860 MIDs or based on this date 2026-06-23
|
|
2026-06-23 17:12:58 | INFO | Fetching data for 860 MIDs
|
|
2026-06-23 17:13:01 | INFO | Fetched 284 rows from SQL Server
|
|
2026-06-23 17:13:01 | INFO | Fetched total row -: 284 from sql server for table-:PaidVisibility ...........!!!
|
|
2026-06-23 17:13:01 | INFO | Fetched 284 rows
|
|
2026-06-23 17:13:01 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _
|
|
2026-06-23 17:13:01 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:13:01 | INFO | PaidVisibility: inserted 284 rows into ClickHouse
|
|
2026-06-23 17:13:01 | INFO | PaidVisibility loaded successfully (284 rows)
|
|
2026-06-23 17:13:01 | INFO | ================================================================================
|
|
2026-06-23 17:13:01 | INFO | Processing Table-:Store_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 17:13:01 | INFO | Fetching Data from sql server for table-: Store_Master ..............
|
|
2026-06-23 17:13:01 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:13:01 | INFO | Fetching in progress ....
|
|
2026-06-23 17:13:13 | INFO | Fetched 5,998 rows from SQL Server
|
|
2026-06-23 17:13:13 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
|
|
2026-06-23 17:13:13 | INFO | Fetched 5998 rows
|
|
2026-06-23 17:13:13 | INFO | Truncated table Store_Master
|
|
2026-06-23 17:13:13 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:13:14 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
|
|
2026-06-23 17:13:14 | INFO | Store_Master loaded successfully (5998 rows)
|
|
2026-06-23 17:13:14 | INFO | ================================================================================
|
|
2026-06-23 17:13:14 | INFO | Processing Table-:SKU_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 17:13:14 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
|
|
2026-06-23 17:13:14 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:13:14 | INFO | Fetching in progress ....
|
|
2026-06-23 17:13:14 | INFO | Fetched 160 rows from SQL Server
|
|
2026-06-23 17:13:14 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
|
|
2026-06-23 17:13:14 | INFO | Fetched 160 rows
|
|
2026-06-23 17:13:14 | INFO | Truncated table SKU_Master
|
|
2026-06-23 17:13:14 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:13:15 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-23 17:13:15 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-23 17:13:15 | INFO | ================================================================================
|
|
2026-06-23 17:13:15 | INFO | Processing Table-:display_master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 17:13:15 | INFO | Fetching Data from sql server for table-: display_master ..............
|
|
2026-06-23 17:13:15 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:13:15 | INFO | Fetching in progress ....
|
|
2026-06-23 17:13:15 | INFO | Fetched 135 rows from SQL Server
|
|
2026-06-23 17:13:15 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
|
|
2026-06-23 17:13:15 | INFO | Fetched 135 rows
|
|
2026-06-23 17:13:15 | INFO | Truncated table display_master
|
|
2026-06-23 17:13:15 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:13:16 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-23 17:13:16 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-23 17:13:16 | INFO | ================================================================================
|
|
2026-06-23 17:13:16 | INFO | Processing Table-:Employee_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-23 17:13:16 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
|
|
2026-06-23 17:13:16 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:13:16 | INFO | Fetching in progress ....
|
|
2026-06-23 17:13:22 | INFO | Fetched 2,289 rows from SQL Server
|
|
2026-06-23 17:13:22 | INFO | Fetched total row -: 2289 from sql server for table-:Employee_Master ...........!!!
|
|
2026-06-23 17:13:22 | INFO | Fetched 2289 rows
|
|
2026-06-23 17:13:22 | INFO | Truncated table Employee_Master
|
|
2026-06-23 17:13:22 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:13:22 | INFO | Employee_Master: inserted 2,289 rows into ClickHouse
|
|
2026-06-23 17:13:22 | INFO | Employee_Master loaded successfully (2289 rows)
|
|
2026-06-23 17:13:22 | INFO | ================================================================================
|
|
2026-06-23 17:13:22 | INFO | Processing Table-:coverage_remarks | Table type -:DIMENSION | fetcht by-:reason_id | operation-:DELETE+INSERT
|
|
2026-06-23 17:13:22 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
|
|
2026-06-23 17:13:23 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:13:23 | INFO | Fetching in progress ....
|
|
2026-06-23 17:13:23 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 17:13:23 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-23 17:13:23 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
|
|
2026-06-23 17:13:23 | WARNING | coverage_remarks returned no rows
|
|
2026-06-23 17:13:23 | INFO | ================================================================================
|
|
2026-06-23 17:13:23 | INFO | Processing Table-:mapping_storevisibility | Table type -:BRIDGE | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-23 17:13:23 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
|
|
2026-06-23 17:13:23 | INFO | Fetching data from sql server for BRIDGE table......
|
|
2026-06-23 17:13:23 | INFO | Fetching in progress ....
|
|
2026-06-23 17:13:24 | INFO | Fetched 805 rows from SQL Server
|
|
2026-06-23 17:13:24 | INFO | Fetched total row -: 805 from sql server for table-:mapping_storevisibility ...........!!!
|
|
2026-06-23 17:13:24 | INFO | Fetched 805 rows
|
|
2026-06-23 17:13:24 | INFO | Truncated table mapping_storevisibility
|
|
2026-06-23 17:13:24 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:13:24 | INFO | mapping_storevisibility: inserted 805 rows into ClickHouse
|
|
2026-06-23 17:13:24 | INFO | mapping_storevisibility loaded successfully (805 rows)
|
|
2026-06-23 17:13:24 | INFO | ================================================================================
|
|
2026-06-23 17:13:24 | INFO | Processing Table-:Master_VisibilityReason | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 17:13:24 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason ..............
|
|
2026-06-23 17:13:24 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:13:24 | INFO | Fetching in progress ....
|
|
2026-06-23 17:13:24 | INFO | Fetched 17 rows from SQL Server
|
|
2026-06-23 17:13:24 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!!
|
|
2026-06-23 17:13:24 | INFO | Fetched 17 rows
|
|
2026-06-23 17:13:24 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-23 17:13:24 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:13:24 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-23 17:13:24 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-23 17:13:24 | INFO | ================================================================================
|
|
2026-06-23 17:13:24 | INFO | Processing Table-:Master_VisibilityDefinition | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 17:13:24 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition ..............
|
|
2026-06-23 17:13:24 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:13:24 | INFO | Fetching in progress ....
|
|
2026-06-23 17:13:25 | INFO | Fetched 861 rows from SQL Server
|
|
2026-06-23 17:13:25 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!!
|
|
2026-06-23 17:13:25 | INFO | Fetched 861 rows
|
|
2026-06-23 17:13:25 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-23 17:13:25 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:13:25 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-23 17:13:25 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-23 17:13:25 | INFO | ================================================================================
|
|
2026-06-23 17:13:25 | INFO | Processing Table-:Master_Salesterritorylayer | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-23 17:13:25 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer ..............
|
|
2026-06-23 17:13:25 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-23 17:13:25 | INFO | Fetching in progress ....
|
|
2026-06-23 17:13:26 | INFO | Fetched 33 rows from SQL Server
|
|
2026-06-23 17:13:26 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!!
|
|
2026-06-23 17:13:26 | INFO | Fetched 33 rows
|
|
2026-06-23 17:13:26 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-23 17:13:26 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:13:26 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-23 17:13:26 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-23 17:13:26 | INFO | ================================================================================
|
|
2026-06-23 17:13:26 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 17:13:26 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-23 17:13:26 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-23 17:13:56 | INFO | Fetched 291,607 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-23 17:13:57 | INFO | Fetched 290,674 matched empids fetched for OQAD
|
|
2026-06-23 17:13:58 | INFO | Exists: True
|
|
2026-06-23 17:13:58 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-23 17:13:58 | INFO | Fetching data for 290,674 EMPIDs
|
|
2026-06-23 17:13:58 | INFO | Fetching OQaD data for run_date=2026-06-23
|
|
2026-06-23 17:14:34 | INFO | fn name is fetch_OQad ------Fetched 2 rows
|
|
2026-06-23 17:14:34 | INFO | Fetched 2 rows from SQL Server
|
|
2026-06-23 17:14:34 | INFO | Fetched total row -: 2 from sql server for table-:OQaD ...........!!!
|
|
2026-06-23 17:14:34 | INFO | Fetched 2 rows
|
|
2026-06-23 17:14:34 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
|
|
2026-06-23 17:14:35 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 17:14:35 | INFO | OQaD: inserted 2 rows into ClickHouse
|
|
2026-06-23 17:14:35 | INFO | OQaD loaded successfully (2 rows)
|
|
2026-06-23 17:14:35 | INFO | ================================================================================
|
|
2026-06-23 17:14:35 | INFO | Pipeline Completed Successfully
|
|
2026-06-23 17:14:35 | INFO | ================================================================================
|
|
2026-06-23 17:14:35 | INFO | Pipeline completed successfully. last_successful_run_date=2026-06-23
|
|
2026-06-23 18:06:23 | INFO | Pipeline Start Date: 2026-06-22
|
|
2026-06-23 18:06:23 | INFO | ================================================================================
|
|
2026-06-23 18:06:23 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 18:06:23 | INFO | Pipeline Run Date: 2026-06-22
|
|
2026-06-23 18:06:23 | INFO | Connecting to databases...
|
|
2026-06-23 18:06:24 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001BF971185D0>
|
|
2026-06-23 18:06:26 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001BF981BC7D0>
|
|
2026-06-23 18:06:26 | INFO | Database connections established
|
|
2026-06-23 18:06:26 | INFO | Collecting MIDs for: 2026-06-22
|
|
2026-06-23 18:06:27 | INFO | Found 882 MIDs
|
|
2026-06-23 18:06:27 | INFO | ================================================================================
|
|
2026-06-23 18:06:27 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 18:06:27 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-23 18:06:28 | INFO | Fetching quiz empids for run_date=2026-06-22
|
|
2026-06-23 18:06:28 | INFO | Fetched 491 (EmpId, VisitDate) pairs from SQL Server
|
|
2026-06-23 18:06:29 | INFO | Fetched 290725 existing (EmpId, VisitDate) pairs from ClickHouse
|
|
2026-06-23 18:06:29 | INFO | Found 489 NEW empids to fetch for OQaD
|
|
2026-06-23 18:06:29 | INFO | Loading SQL from: D:\z\airflow-project\src\sql\fact\OQaD.sql (exists=True)
|
|
2026-06-23 18:06:29 | INFO | Fetching full OQaD data for 489 empids, run_date=2026-06-22
|
|
2026-06-23 18:06:33 | INFO | Fetched 2 rows from SQL Server for table=OQaD
|
|
2026-06-23 18:06:33 | INFO | fetch_OQaD complete — returning 2 rows
|
|
2026-06-23 18:06:33 | INFO | Fetched total row -: 2 from sql server for table-:OQaD ...........!!!
|
|
2026-06-23 18:06:33 | INFO | Fetched 2 rows
|
|
2026-06-23 18:06:33 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
|
|
2026-06-23 18:06:33 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 18:06:33 | INFO | OQaD: inserted 2 rows into ClickHouse
|
|
2026-06-23 18:06:33 | INFO | OQaD loaded successfully (2 rows)
|
|
2026-06-23 18:06:33 | INFO | ================================================================================
|
|
2026-06-23 18:06:33 | INFO | Processing Table-:Promotion | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 18:06:33 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-23 18:06:33 | INFO | Fetching data for 882 MIDs
|
|
2026-06-23 18:06:37 | INFO | Fetched 8,835 rows from SQL Server
|
|
2026-06-23 18:06:37 | INFO | Fetched total row -: 8835 from sql server for table-:Promotion ...........!!!
|
|
2026-06-23 18:06:37 | INFO | Fetched 8835 rows
|
|
2026-06-23 18:06:37 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-23 18:06:37 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 18:06:37 | INFO | Promotion: inserted 8,835 rows into ClickHouse
|
|
2026-06-23 18:06:37 | INFO | Promotion loaded successfully (8835 rows)
|
|
2026-06-23 18:06:37 | INFO | ================================================================================
|
|
2026-06-23 18:06:37 | INFO | Pipeline Completed Successfully
|
|
2026-06-23 18:06:37 | INFO | ================================================================================
|
|
2026-06-23 18:06:37 | INFO | Pipeline completed successfully. last_successful_run_date=2026-06-23
|
|
2026-06-23 18:06:37 | INFO | ================================================================================
|
|
2026-06-23 18:06:37 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 18:06:37 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-23 18:06:37 | INFO | Connecting to databases...
|
|
2026-06-23 18:06:38 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001BF981C0F50>
|
|
2026-06-23 18:06:38 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001BF981C1A90>
|
|
2026-06-23 18:06:38 | INFO | Database connections established
|
|
2026-06-23 18:06:38 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-23 18:06:39 | INFO | Found 860 MIDs
|
|
2026-06-23 18:06:39 | INFO | ================================================================================
|
|
2026-06-23 18:06:39 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 18:06:39 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-23 18:06:39 | INFO | Fetching quiz empids for run_date=2026-06-23
|
|
2026-06-23 18:06:40 | INFO | Fetched 447 (EmpId, VisitDate) pairs from SQL Server
|
|
2026-06-23 18:06:41 | INFO | Fetched 290725 existing (EmpId, VisitDate) pairs from ClickHouse
|
|
2026-06-23 18:06:41 | INFO | Found 446 NEW empids to fetch for OQaD
|
|
2026-06-23 18:06:41 | INFO | Loading SQL from: D:\z\airflow-project\src\sql\fact\OQaD.sql (exists=True)
|
|
2026-06-23 18:06:41 | INFO | Fetching full OQaD data for 446 empids, run_date=2026-06-23
|
|
2026-06-23 18:06:44 | INFO | Fetched 1 rows from SQL Server for table=OQaD
|
|
2026-06-23 18:06:44 | INFO | fetch_OQaD complete — returning 1 rows
|
|
2026-06-23 18:06:44 | INFO | Fetched total row -: 1 from sql server for table-:OQaD ...........!!!
|
|
2026-06-23 18:06:44 | INFO | Fetched 1 rows
|
|
2026-06-23 18:06:44 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
|
|
2026-06-23 18:06:44 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 18:06:44 | INFO | OQaD: inserted 1 rows into ClickHouse
|
|
2026-06-23 18:06:44 | INFO | OQaD loaded successfully (1 rows)
|
|
2026-06-23 18:06:44 | INFO | ================================================================================
|
|
2026-06-23 18:06:44 | INFO | Processing Table-:Promotion | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-23 18:06:44 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-23 18:06:44 | INFO | Fetching data for 860 MIDs
|
|
2026-06-23 18:06:46 | INFO | Fetched 5,668 rows from SQL Server
|
|
2026-06-23 18:06:46 | INFO | Fetched total row -: 5668 from sql server for table-:Promotion ...........!!!
|
|
2026-06-23 18:06:46 | INFO | Fetched 5668 rows
|
|
2026-06-23 18:06:46 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-23 18:06:46 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 18:06:47 | INFO | Promotion: inserted 5,668 rows into ClickHouse
|
|
2026-06-23 18:06:47 | INFO | Promotion loaded successfully (5668 rows)
|
|
2026-06-23 18:06:47 | INFO | ================================================================================
|
|
2026-06-23 18:06:47 | INFO | Pipeline Completed Successfully
|
|
2026-06-23 18:06:47 | INFO | ================================================================================
|
|
2026-06-23 18:06:47 | INFO | Pipeline completed successfully. last_successful_run_date=2026-06-23
|
|
2026-06-23 18:14:23 | INFO | Pipeline Start Date: 2026-06-22
|
|
2026-06-23 18:14:23 | INFO | ================================================================================
|
|
2026-06-23 18:14:23 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 18:14:23 | INFO | Pipeline Run Date: 2026-06-22
|
|
2026-06-23 18:14:23 | INFO | Connecting to databases...
|
|
2026-06-23 18:14:24 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000239F2CF0730>
|
|
2026-06-23 18:14:25 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000239F3DA47D0>
|
|
2026-06-23 18:14:26 | INFO | Database connections established
|
|
2026-06-23 18:14:26 | INFO | Collecting MIDs for: 2026-06-22
|
|
2026-06-23 18:14:26 | INFO | Found 882 MIDs
|
|
2026-06-23 18:14:27 | INFO | ================================================================================
|
|
2026-06-23 18:14:27 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 18:14:27 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-23 18:14:27 | INFO | Fetching quiz empids for run_date=2026-06-22
|
|
2026-06-23 18:14:28 | INFO | Fetched 491 (EmpId, VisitDate) pairs from SQL Server
|
|
2026-06-23 18:14:29 | INFO | Fetched 290725 existing (EmpId, VisitDate) pairs from ClickHouse
|
|
2026-06-23 18:14:29 | INFO | Found 489 NEW empids to fetch for OQaD
|
|
2026-06-23 18:14:29 | INFO | Loading SQL from: D:\z\airflow-project\src\sql\fact\OQaD.sql (exists=True)
|
|
2026-06-23 18:14:29 | INFO | Fetching full OQaD data for 489 empids, run_date=2026-06-22
|
|
2026-06-23 18:14:32 | INFO | Fetched 2 rows from SQL Server for table=OQaD
|
|
2026-06-23 18:14:32 | INFO | fetch_OQaD complete — returning 2 rows
|
|
2026-06-23 18:14:32 | INFO | Fetched total row -: 2 from sql server for table-:OQaD ...........!!!
|
|
2026-06-23 18:14:32 | INFO | Fetched 2 rows
|
|
2026-06-23 18:14:32 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
|
|
2026-06-23 18:14:32 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 18:14:32 | INFO | OQaD: inserted 2 rows into ClickHouse
|
|
2026-06-23 18:14:32 | INFO | OQaD loaded successfully (2 rows)
|
|
2026-06-23 18:14:32 | INFO | ================================================================================
|
|
2026-06-23 18:14:32 | INFO | Pipeline Completed Successfully
|
|
2026-06-23 18:14:32 | INFO | ================================================================================
|
|
2026-06-23 18:14:32 | INFO | Pipeline completed successfully. last_successful_run_date=2026-06-23
|
|
2026-06-23 18:14:32 | INFO | ================================================================================
|
|
2026-06-23 18:14:32 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 18:14:32 | INFO | Pipeline Run Date: 2026-06-23
|
|
2026-06-23 18:14:32 | INFO | Connecting to databases...
|
|
2026-06-23 18:14:33 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000239F3DA9310>
|
|
2026-06-23 18:14:33 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000239F3DA9D60>
|
|
2026-06-23 18:14:34 | INFO | Database connections established
|
|
2026-06-23 18:14:34 | INFO | Collecting MIDs for: 2026-06-23
|
|
2026-06-23 18:14:34 | INFO | Found 860 MIDs
|
|
2026-06-23 18:14:34 | INFO | ================================================================================
|
|
2026-06-23 18:14:34 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 18:14:34 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-23 18:14:34 | INFO | Fetching quiz empids for run_date=2026-06-23
|
|
2026-06-23 18:14:35 | INFO | Fetched 447 (EmpId, VisitDate) pairs from SQL Server
|
|
2026-06-23 18:14:36 | INFO | Fetched 290725 existing (EmpId, VisitDate) pairs from ClickHouse
|
|
2026-06-23 18:14:36 | INFO | Found 446 NEW empids to fetch for OQaD
|
|
2026-06-23 18:14:36 | INFO | Loading SQL from: D:\z\airflow-project\src\sql\fact\OQaD.sql (exists=True)
|
|
2026-06-23 18:14:36 | INFO | Fetching full OQaD data for 446 empids, run_date=2026-06-23
|
|
2026-06-23 18:14:39 | INFO | Fetched 1 rows from SQL Server for table=OQaD
|
|
2026-06-23 18:14:39 | INFO | fetch_OQaD complete — returning 1 rows
|
|
2026-06-23 18:14:39 | INFO | Fetched total row -: 1 from sql server for table-:OQaD ...........!!!
|
|
2026-06-23 18:14:39 | INFO | Fetched 1 rows
|
|
2026-06-23 18:14:39 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
|
|
2026-06-23 18:14:39 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 18:14:39 | INFO | OQaD: inserted 1 rows into ClickHouse
|
|
2026-06-23 18:14:39 | INFO | OQaD loaded successfully (1 rows)
|
|
2026-06-23 18:14:39 | INFO | ================================================================================
|
|
2026-06-23 18:14:39 | INFO | Pipeline Completed Successfully
|
|
2026-06-23 18:14:39 | INFO | ================================================================================
|
|
2026-06-23 18:14:39 | INFO | Pipeline completed successfully. last_successful_run_date=2026-06-23
|
|
2026-06-23 18:18:24 | INFO | Pipeline Start Date: 2026-06-22
|
|
2026-06-23 18:18:24 | INFO | ================================================================================
|
|
2026-06-23 18:18:24 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-23 18:18:24 | INFO | Pipeline Run Date: 2026-06-22
|
|
2026-06-23 18:18:24 | INFO | Connecting to databases...
|
|
2026-06-23 18:18:25 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000210A0000730>
|
|
2026-06-23 18:18:26 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000210A11787D0>
|
|
2026-06-23 18:18:26 | INFO | Database connections established
|
|
2026-06-23 18:18:26 | INFO | Collecting MIDs for: 2026-06-22
|
|
2026-06-23 18:18:27 | INFO | Found 882 MIDs
|
|
2026-06-23 18:18:27 | INFO | ================================================================================
|
|
2026-06-23 18:18:27 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-23 18:18:27 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-23 18:18:27 | INFO | Fetching quiz empids for run_date=2026-06-22
|
|
2026-06-23 18:18:28 | INFO | Fetched 491 (EmpId, VisitDate) pairs from SQL Server
|
|
2026-06-23 18:18:29 | INFO | Fetched 290725 existing (EmpId, VisitDate) pairs from ClickHouse
|
|
2026-06-23 18:18:29 | INFO | Found 489 NEW empids to fetch for OQaD
|
|
2026-06-23 18:18:29 | INFO | Loading SQL from: D:\z\airflow-project\src\sql\fact\OQaD.sql (exists=True)
|
|
2026-06-23 18:18:29 | INFO | Fetching full OQaD data for 489 empids, run_date=2026-06-22
|
|
2026-06-23 18:18:30 | INFO | Fetched 2 rows from SQL Server for table=OQaD
|
|
2026-06-23 18:18:30 | INFO | fetch_OQaD complete — returning 2 rows
|
|
2026-06-23 18:18:30 | INFO | Fetched total row -: 2 from sql server for table-:OQaD ...........!!!
|
|
2026-06-23 18:18:30 | INFO | Fetched 2 rows
|
|
2026-06-23 18:18:30 | INFO | _ _ _ _ Deleting Data from ClickHouse for OQaD _ _ _ _
|
|
2026-06-23 18:18:31 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-23 18:18:31 | INFO | OQaD: inserted 2 rows into ClickHouse
|
|
2026-06-23 18:18:31 | INFO | OQaD loaded successfully (2 rows)
|
|
2026-06-23 18:18:31 | INFO | ================================================================================
|
|
2026-06-23 18:18:31 | INFO | Pipeline Completed Successfully
|
|
2026-06-23 18:18:31 | INFO | ================================================================================
|
|
2026-06-23 18:18:31 | INFO | Pipeline completed successfully. pipeline_trigeered_on_date=2026-06-23last_successful_run_date=2026-06-22
|