2026-06-22 12:12:58 | INFO | ================================================================================ 2026-06-22 12:12:58 | INFO | Hello from data-move Python data pipeline! 2026-06-22 12:12:58 | INFO | Pipeline Run Date: 2026-06-21 2026-06-22 12:12:58 | INFO | Connecting to databases... 2026-06-22 12:12:59 | INFO | 2026-06-22 12:13:03 | INFO | 2026-06-22 12:13:03 | INFO | Database connections established 2026-06-22 12:13:03 | INFO | Collecting MIDs for: 2026-06-21 2026-06-22 12:13:04 | INFO | Found 569 MIDs 2026-06-22 12:13:05 | INFO | ================================================================================ 2026-06-22 12:13:05 | INFO | Processing Table: Master_VisibilityReason | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT 2026-06-22 12:13:05 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason .............. 2026-06-22 12:13:05 | INFO | Fetching data from sql server for Master table...... 2026-06-22 12:13:05 | INFO | Fetching in progress .... 2026-06-22 12:13:05 | INFO | Fetched 17 rows from SQL Server 2026-06-22 12:13:05 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!! 2026-06-22 12:13:05 | INFO | Fetched 17 rows 2026-06-22 12:13:05 | INFO | Truncated table Master_VisibilityReason 2026-06-22 12:13:05 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:13:05 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse 2026-06-22 12:13:05 | INFO | Master_VisibilityReason loaded successfully (17 rows) 2026-06-22 12:13:05 | INFO | ================================================================================ 2026-06-22 12:13:05 | INFO | Processing Table: Master_VisibilityDefinition | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT 2026-06-22 12:13:05 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition .............. 2026-06-22 12:13:05 | INFO | Fetching data from sql server for Master table...... 2026-06-22 12:13:05 | INFO | Fetching in progress .... 2026-06-22 12:13:05 | INFO | Fetched 861 rows from SQL Server 2026-06-22 12:13:05 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!! 2026-06-22 12:13:05 | INFO | Fetched 861 rows 2026-06-22 12:13:05 | INFO | Truncated table Master_VisibilityDefinition 2026-06-22 12:13:05 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:13:06 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse 2026-06-22 12:13:06 | INFO | Master_VisibilityDefinition loaded successfully (861 rows) 2026-06-22 12:13:06 | INFO | ================================================================================ 2026-06-22 12:13:06 | INFO | Processing Table: Master_Salesterritorylayer | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT 2026-06-22 12:13:06 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer .............. 2026-06-22 12:13:06 | INFO | Fetching data from sql server for Master table...... 2026-06-22 12:13:06 | INFO | Fetching in progress .... 2026-06-22 12:13:06 | INFO | Fetched 33 rows from SQL Server 2026-06-22 12:13:06 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!! 2026-06-22 12:13:06 | INFO | Fetched 33 rows 2026-06-22 12:13:06 | INFO | Truncated table Master_Salesterritorylayer 2026-06-22 12:13:06 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:13:06 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse 2026-06-22 12:13:06 | INFO | Master_Salesterritorylayer loaded successfully (33 rows) 2026-06-22 12:13:06 | INFO | ================================================================================ 2026-06-22 12:13:06 | INFO | Pipeline Completed Successfully 2026-06-22 12:13:06 | INFO | ================================================================================ 2026-06-22 12:16:47 | INFO | ================================================================================ 2026-06-22 12:16:47 | INFO | Hello from data-move Python data pipeline! 2026-06-22 12:16:47 | INFO | Pipeline Run Date: 2026-06-21 2026-06-22 12:16:47 | INFO | Connecting to databases... 2026-06-22 12:16:48 | INFO | 2026-06-22 12:16:50 | INFO | 2026-06-22 12:16:52 | INFO | Database connections established 2026-06-22 12:16:52 | INFO | Collecting MIDs for: 2026-06-21 2026-06-22 12:16:52 | INFO | Found 569 MIDs 2026-06-22 12:16:53 | INFO | ================================================================================ 2026-06-22 12:16:53 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:16:53 | INFO | Fetching Data from sql server for table-: SOS_OneApp .............. 2026-06-22 12:16:53 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 12:16:53 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:16:53 | INFO | Fetching data for 569 MIDs 2026-06-22 12:16:57 | INFO | Fetched 1,907 rows from SQL Server 2026-06-22 12:16:57 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!! 2026-06-22 12:16:57 | INFO | Fetched 1907 rows 2026-06-22 12:16:57 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _ 2026-06-22 12:16:57 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:16:58 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse 2026-06-22 12:16:58 | INFO | SOS_OneApp loaded successfully (1907 rows) 2026-06-22 12:16:58 | INFO | ================================================================================ 2026-06-22 12:16:58 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 12:16:58 | INFO | Fetching Data from sql server for table-: OQaD .............. 2026-06-22 12:16:58 | INFO | Fetching quiz_empids data for EMPID and Visitid 2026-06-22 12:17:14 | INFO | Fetched 291,134 total empid and visitdate fetched for OQAD from SQL Server 2026-06-22 12:17:14 | WARNING | Table 'OQaD' does not exist. 2026-06-22 12:17:14 | INFO | Fetched 0 matched empids fetched for OQAD 2026-06-22 12:17:14 | INFO | Exists: True 2026-06-22 12:17:14 | INFO | Path: D:\z\data_move\src\sql\fact\OQaD.sql 2026-06-22 12:17:14 | INFO | Fetching data for 0 EMPIDs 2026-06-22 12:17:14 | INFO | Fetching OQaD data for run_date=2026-06-21 2026-06-22 12:17:14 | ERROR | Failed processing table OQaD Traceback (most recent call last): File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 1969, in _exec_single_context self.dialect.do_execute( ~~~~~~~~~~~~~~~~~~~~~~~^ cursor, str_statement, effective_parameters, context ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\default.py", line 952, in do_execute cursor.execute(statement, parameters) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Incorrect syntax near ')'. (102) (SQLExecDirectW)") The above exception was the direct cause of the following exception: Traceback (most recent call last): File "D:\z\data_move\main2.py", line 159, in main df=fn(sql_engine, table_name , table_type, mids, run_date) File "D:\z\data_move\src\fact.py", line 285, in fetch_OQaD df=fetch_data( engine=sql_engine, table_name=table_name, ...<2 lines>... run_date=run_date ) File "D:\z\data_move\src\fact.py", line 277, in fetch_data df = pl.read_database( query=sql, connection=engine, ) File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\polars\io\database\functions.py", line 284, in read_database return cx.execute( ~~~~~~~~~~^ query=query, ^^^^^^^^^^^^ options=execute_options, ^^^^^^^^^^^^^^^^^^^^^^^^ ).to_polars( ^ File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\polars\io\database\_executor.py", line 546, in execute result = cursor_execute(query, **options) File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 1421, in execute return meth( self, distilled_parameters, execution_options or NO_OPTIONS, ) File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\sql\elements.py", line 526, in _execute_on_connection return connection._execute_clauseelement( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ self, distilled_params, execution_options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 1643, in _execute_clauseelement ret = self._execute_context( dialect, ...<8 lines>... cache_hit=cache_hit, ) File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 1848, in _execute_context return self._exec_single_context( ~~~~~~~~~~~~~~~~~~~~~~~~~^ dialect, context, statement, parameters ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 1988, in _exec_single_context self._handle_dbapi_exception( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ e, str_statement, effective_parameters, cursor, context ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 2365, in _handle_dbapi_exception raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 1969, in _exec_single_context self.dialect.do_execute( ~~~~~~~~~~~~~~~~~~~~~~~^ cursor, str_statement, effective_parameters, context ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\default.py", line 952, in do_execute cursor.execute(statement, parameters) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Incorrect syntax near ')'. (102) (SQLExecDirectW)") [SQL: WITH MID_TABLE_COV1 AS ( SELECT EmpId, VisitDate FROM OneApp_KelloggsMT.dbo.T_OQAD WHERE CreateDate >= 2026-06-21 AND CreateDate < DATEADD(DAY,1,'2026-06-21') UNION ALL SELECT EmpId, VisitDate FROM OneApp_KelloggsMT.dbo.T_OQAD WHERE UpdateDate >= 2026-06-21 AND UpdateDate < DATEADD(DAY,1, '2026-06-21') ), QUIZ AS ( SELECT DISTINCT E.EmpId, E.EmpName, E.SupervisorId, E.SupervisorName, E.DesignationName, E.CityName, E.StateName, E.RegionName, CAST(DQ.VisitDate AS DATE) AS VisitDate, DQ.QuestionId, DQ.AnswerId, QC.QuestionCategoryId, QC.QuestionCategory FROM OneApp_KelloggsMT.dbo.T_OQAD DQ INNER JOIN OneApp_KelloggsMT.dbo.vw_Employee_Detail E ON DQ.EmpId = E.EmpId INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Question QU ON DQ.QuestionId = QU.QuestionId INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Category QC ON QU.QuestionCategoryId = QC.QuestionCategoryId WHERE E.EmpName NOT LIKE '%TEST%' AND E.RightId = 6 AND ( E.ResignDate IS NULL OR CAST(E.ResignDate AS DATE) >= '2026-06-21' ) AND EXISTS ( SELECT 1 FROM MID_TABLE_COV1 A WHERE A.EmpId = DQ.EmpId AND CAST(A.VisitDate AS DATE) = CAST(DQ.VisitDate AS DATE) ) ) SELECT 40148 AS project_id, Q.EmpId AS employee_id, 0 AS process_id, Q.VisitDate AS visit_date, Q.QuestionCategoryId AS question_category_id, Q.QuestionCategory AS question_category, QM.QuestionId AS question_id, QM.Question AS question, ISNULL(QA.AnswerId,0) AS answer_id, ISNULL(QA.Answer,'') AS answer, CASE WHEN QA.AnswerId IS NULL THEN 'Not Answer' WHEN QA.RightAnswer = 1 THEN 'Y' WHEN QA.RightAnswer IS NULL THEN 'Not Answer' ELSE 'N' END AS correct_answer, GETDATE() AS update_date, 'SP-Pius' AS update_by FROM QUIZ Q INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Question QM ON Q.QuestionId = QM.QuestionId LEFT JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Answer QA ON Q.AnswerId = QA.AnswerId where Q.EmpId not in () ] (Background on this error at: https://sqlalche.me/e/20/f405) 2026-06-22 12:27:42 | INFO | ================================================================================ 2026-06-22 12:27:42 | INFO | Hello from data-move Python data pipeline! 2026-06-22 12:27:42 | INFO | Pipeline Run Date: 2026-06-21 2026-06-22 12:27:42 | INFO | Connecting to databases... 2026-06-22 12:27:43 | INFO | 2026-06-22 12:27:44 | INFO | 2026-06-22 12:27:44 | INFO | Database connections established 2026-06-22 12:27:44 | INFO | Collecting MIDs for: 2026-06-21 2026-06-22 12:27:45 | INFO | Found 569 MIDs 2026-06-22 12:27:45 | INFO | ================================================================================ 2026-06-22 12:27:45 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:27:45 | INFO | Fetching Data from sql server for table-: SOS_OneApp .............. 2026-06-22 12:27:45 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 12:27:45 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:27:45 | INFO | Fetching data for 569 MIDs 2026-06-22 12:27:50 | INFO | Fetched 1,907 rows from SQL Server 2026-06-22 12:27:50 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!! 2026-06-22 12:27:50 | INFO | Fetched 1907 rows 2026-06-22 12:27:50 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _ 2026-06-22 12:27:51 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:27:52 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse 2026-06-22 12:27:52 | INFO | SOS_OneApp loaded successfully (1907 rows) 2026-06-22 12:27:52 | INFO | ================================================================================ 2026-06-22 12:27:52 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 12:27:52 | INFO | Fetching Data from sql server for table-: OQaD .............. 2026-06-22 12:27:53 | INFO | Fetching quiz_empids data for EMPID and Visitid 2026-06-22 12:28:12 | INFO | Fetched 291,145 total empid and visitdate fetched for OQAD from SQL Server 2026-06-22 12:28:12 | WARNING | Table 'OQaD' does not exist. 2026-06-22 12:28:12 | WARNING | OQaD Matched df in OQaD returned no rows 2026-06-22 12:28:12 | ERROR | Failed processing table OQaD Traceback (most recent call last): File "D:\z\data_move\main2.py", line 159, in main df=fn(sql_engine, table_name , table_type, mids, run_date) File "D:\z\data_move\src\fact.py", line 257, in fetch_OQaD log.info(f"Fetched {len(empids):,} matched empids fetched for OQAD ") ^^^^^^ UnboundLocalError: cannot access local variable 'empids' where it is not associated with a value 2026-06-22 12:28:18 | INFO | ================================================================================ 2026-06-22 12:28:18 | INFO | Hello from data-move Python data pipeline! 2026-06-22 12:28:18 | INFO | Pipeline Run Date: 2026-06-21 2026-06-22 12:28:18 | INFO | Connecting to databases... 2026-06-22 12:28:19 | INFO | 2026-06-22 12:28:19 | INFO | 2026-06-22 12:28:20 | INFO | Database connections established 2026-06-22 12:28:20 | INFO | Collecting MIDs for: 2026-06-21 2026-06-22 12:28:20 | INFO | Found 569 MIDs 2026-06-22 12:28:20 | INFO | ================================================================================ 2026-06-22 12:28:20 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:28:20 | INFO | Fetching Data from sql server for table-: SOS_OneApp .............. 2026-06-22 12:28:20 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 12:28:20 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:28:20 | INFO | Fetching data for 569 MIDs 2026-06-22 12:28:25 | INFO | Fetched 1,907 rows from SQL Server 2026-06-22 12:28:25 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!! 2026-06-22 12:28:25 | INFO | Fetched 1907 rows 2026-06-22 12:28:25 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _ 2026-06-22 12:28:25 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:28:25 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse 2026-06-22 12:28:25 | INFO | SOS_OneApp loaded successfully (1907 rows) 2026-06-22 12:28:25 | INFO | ================================================================================ 2026-06-22 12:28:25 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 12:28:25 | INFO | Fetching Data from sql server for table-: OQaD .............. 2026-06-22 12:28:25 | INFO | Fetching quiz_empids data for EMPID and Visitid 2026-06-22 12:28:47 | INFO | Fetched 291,145 total empid and visitdate fetched for OQAD from SQL Server 2026-06-22 12:28:47 | WARNING | Table 'OQaD' does not exist. 2026-06-22 12:28:47 | ERROR | Failed processing table OQaD Traceback (most recent call last): File "D:\z\data_move\main2.py", line 159, in main df=fn(sql_engine, table_name , table_type, mids, run_date) File "D:\z\data_move\src\fact.py", line 248, in fetch_OQaD log.warning( ~~~~~~~~~~~^ "%s Matched df in OQaD returned no rows", ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ empids=[0] ^^^^^^^^^^ ) ^ File "C:\Users\ankitm\AppData\Roaming\uv\python\cpython-3.14-windows-x86_64-none\Lib\logging\__init__.py", line 1532, in warning self._log(WARNING, msg, args, **kwargs) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: Logger._log() got an unexpected keyword argument 'empids' 2026-06-22 12:29:11 | INFO | ================================================================================ 2026-06-22 12:29:11 | INFO | Hello from data-move Python data pipeline! 2026-06-22 12:29:11 | INFO | Pipeline Run Date: 2026-06-21 2026-06-22 12:29:11 | INFO | Connecting to databases... 2026-06-22 12:29:12 | INFO | 2026-06-22 12:29:13 | INFO | 2026-06-22 12:29:13 | INFO | Database connections established 2026-06-22 12:29:13 | INFO | Collecting MIDs for: 2026-06-21 2026-06-22 12:29:14 | INFO | Found 569 MIDs 2026-06-22 12:29:14 | INFO | ================================================================================ 2026-06-22 12:29:14 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:29:14 | INFO | Fetching Data from sql server for table-: SOS_OneApp .............. 2026-06-22 12:29:14 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 12:29:14 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:29:14 | INFO | Fetching data for 569 MIDs 2026-06-22 12:29:15 | INFO | Fetched 1,907 rows from SQL Server 2026-06-22 12:29:15 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!! 2026-06-22 12:29:15 | INFO | Fetched 1907 rows 2026-06-22 12:29:15 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _ 2026-06-22 12:29:15 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:29:15 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse 2026-06-22 12:29:15 | INFO | SOS_OneApp loaded successfully (1907 rows) 2026-06-22 12:29:15 | INFO | ================================================================================ 2026-06-22 12:29:15 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 12:29:15 | INFO | Fetching Data from sql server for table-: OQaD .............. 2026-06-22 12:29:16 | INFO | Fetching quiz_empids data for EMPID and Visitid 2026-06-22 12:29:27 | INFO | Fetched 291,145 total empid and visitdate fetched for OQAD from SQL Server 2026-06-22 12:29:27 | WARNING | Table 'OQaD' does not exist. 2026-06-22 12:29:27 | ERROR | Failed processing table OQaD Traceback (most recent call last): File "D:\z\data_move\main2.py", line 159, in main df=fn(sql_engine, table_name , table_type, mids, run_date) File "D:\z\data_move\src\fact.py", line 248, in fetch_OQaD log.warning( ~~~~~~~~~~~^ "%s Matched df in OQaD returned no rows", ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ empids=[0] ^^^^^^^^^^ ) ^ File "C:\Users\ankitm\AppData\Roaming\uv\python\cpython-3.14-windows-x86_64-none\Lib\logging\__init__.py", line 1532, in warning self._log(WARNING, msg, args, **kwargs) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: Logger._log() got an unexpected keyword argument 'empids' 2026-06-22 12:30:00 | INFO | ================================================================================ 2026-06-22 12:30:00 | INFO | Hello from data-move Python data pipeline! 2026-06-22 12:30:00 | INFO | Pipeline Run Date: 2026-06-21 2026-06-22 12:30:00 | INFO | Connecting to databases... 2026-06-22 12:30:02 | INFO | 2026-06-22 12:30:03 | INFO | 2026-06-22 12:30:03 | INFO | Database connections established 2026-06-22 12:30:03 | INFO | Collecting MIDs for: 2026-06-21 2026-06-22 12:30:04 | INFO | Found 569 MIDs 2026-06-22 12:30:04 | INFO | ================================================================================ 2026-06-22 12:30:04 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:30:04 | INFO | Fetching Data from sql server for table-: SOS_OneApp .............. 2026-06-22 12:30:04 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 12:30:04 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:30:04 | INFO | Fetching data for 569 MIDs 2026-06-22 12:30:06 | INFO | Fetched 1,907 rows from SQL Server 2026-06-22 12:30:06 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!! 2026-06-22 12:30:06 | INFO | Fetched 1907 rows 2026-06-22 12:30:06 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _ 2026-06-22 12:30:06 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:30:06 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse 2026-06-22 12:30:06 | INFO | SOS_OneApp loaded successfully (1907 rows) 2026-06-22 12:30:06 | INFO | ================================================================================ 2026-06-22 12:30:06 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 12:30:06 | INFO | Fetching Data from sql server for table-: OQaD .............. 2026-06-22 12:30:07 | INFO | Fetching quiz_empids data for EMPID and Visitid 2026-06-22 12:30:24 | INFO | Fetched 291,147 total empid and visitdate fetched for OQAD from SQL Server 2026-06-22 12:30:24 | WARNING | Table 'OQaD' does not exist. 2026-06-22 12:30:24 | WARNING | OQaD Matched df in OQaD returned no rows 2026-06-22 12:30:24 | INFO | Fetched 1 matched empids fetched for OQAD 2026-06-22 12:30:24 | INFO | Exists: True 2026-06-22 12:30:24 | INFO | Path: D:\z\data_move\src\sql\fact\OQaD.sql 2026-06-22 12:30:24 | INFO | Fetching data for 1 EMPIDs 2026-06-22 12:30:24 | INFO | Fetching OQaD data for run_date=2026-06-21 2026-06-22 12:32:08 | INFO | fn name is fetch_OQad ------Fetched 290730 rows 2026-06-22 12:32:08 | INFO | Fetched 290,730 rows from SQL Server 2026-06-22 12:32:08 | INFO | Fetched total row -: 290730 from sql server for table-:OQaD ...........!!! 2026-06-22 12:32:08 | INFO | Fetched 290730 rows 2026-06-22 12:32:08 | INFO | Creating table OQaD 2026-06-22 12:32:08 | INFO | Table ready: OQaD 2026-06-22 12:32:08 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:32:12 | INFO | OQaD: inserted 290,730 rows into ClickHouse 2026-06-22 12:32:12 | INFO | OQaD loaded successfully (290730 rows) 2026-06-22 12:32:12 | INFO | ================================================================================ 2026-06-22 12:32:12 | INFO | Processing Table: Survey | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:32:12 | INFO | Fetching Data from sql server for table-: Survey .............. 2026-06-22 12:32:12 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:32:12 | INFO | Fetching data for 569 MIDs 2026-06-22 12:32:13 | INFO | Fetched 165 rows from SQL Server 2026-06-22 12:32:13 | INFO | Fetched total row -: 165 from sql server for table-:Survey ...........!!! 2026-06-22 12:32:13 | INFO | Fetched 165 rows 2026-06-22 12:32:13 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _ 2026-06-22 12:32:13 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:32:13 | INFO | Survey: inserted 165 rows into ClickHouse 2026-06-22 12:32:13 | INFO | Survey loaded successfully (165 rows) 2026-06-22 12:32:13 | INFO | ================================================================================ 2026-06-22 12:32:13 | INFO | Processing Table: additional_visibility | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:32:13 | INFO | Fetching Data from sql server for table-: additional_visibility .............. 2026-06-22 12:32:13 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 12:32:13 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:32:13 | INFO | Fetching data for 569 MIDs 2026-06-22 12:32:14 | INFO | Fetched 1,579 rows from SQL Server 2026-06-22 12:32:14 | INFO | Fetched total row -: 1579 from sql server for table-:additional_visibility ...........!!! 2026-06-22 12:32:14 | INFO | Fetched 1579 rows 2026-06-22 12:32:14 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _ 2026-06-22 12:32:14 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:32:14 | INFO | additional_visibility: inserted 1,579 rows into ClickHouse 2026-06-22 12:32:14 | INFO | additional_visibility loaded successfully (1579 rows) 2026-06-22 12:32:14 | INFO | ================================================================================ 2026-06-22 12:32:14 | INFO | Processing Table: Coverage | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:32:14 | INFO | Fetching Data from sql server for table-: Coverage .............. 2026-06-22 12:32:14 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:32:14 | INFO | Fetching data for 569 MIDs 2026-06-22 12:32:33 | INFO | Fetched 512 rows from SQL Server 2026-06-22 12:32:33 | INFO | Fetched total row -: 512 from sql server for table-:Coverage ...........!!! 2026-06-22 12:32:33 | INFO | Fetched 512 rows 2026-06-22 12:32:33 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _ 2026-06-22 12:32:33 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:32:33 | INFO | Coverage: inserted 512 rows into ClickHouse 2026-06-22 12:32:33 | INFO | Coverage loaded successfully (512 rows) 2026-06-22 12:32:33 | INFO | ================================================================================ 2026-06-22 12:32:33 | INFO | Processing Table: Login | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 12:32:33 | INFO | Fetching Data from sql server for table-: Login .............. 2026-06-22 12:32:33 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:32:33 | ERROR | Failed processing table Login Traceback (most recent call last): File "D:\z\data_move\main2.py", line 159, in main df=fn(sql_engine, table_name , table_type, mids, run_date) File "D:\z\data_move\src\fact.py", line 443, in fetch_Login sql = sql_template.format( mid_list=mid_list, run_date=run_date.strftime("%Y-%m-%d") ) KeyError: 'start_date' 2026-06-22 12:43:20 | INFO | ================================================================================ 2026-06-22 12:43:20 | INFO | Hello from data-move Python data pipeline! 2026-06-22 12:43:20 | INFO | Pipeline Run Date: 2026-06-21 2026-06-22 12:43:20 | INFO | Connecting to databases... 2026-06-22 12:43:21 | INFO | 2026-06-22 12:43:22 | INFO | 2026-06-22 12:43:23 | INFO | Database connections established 2026-06-22 12:43:23 | INFO | Collecting MIDs for: 2026-06-21 2026-06-22 12:43:23 | INFO | Found 569 MIDs 2026-06-22 12:43:23 | INFO | ================================================================================ 2026-06-22 12:43:23 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:43:23 | INFO | Fetching Data from sql server for table-: SOS_OneApp .............. 2026-06-22 12:43:23 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 12:43:23 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:43:23 | INFO | Fetching data for 569 MIDs 2026-06-22 12:43:25 | INFO | Fetched 1,907 rows from SQL Server 2026-06-22 12:43:25 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!! 2026-06-22 12:43:25 | INFO | Fetched 1907 rows 2026-06-22 12:43:25 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _ 2026-06-22 12:43:25 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:43:26 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse 2026-06-22 12:43:26 | INFO | SOS_OneApp loaded successfully (1907 rows) 2026-06-22 12:43:26 | INFO | ================================================================================ 2026-06-22 12:43:26 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 12:43:26 | INFO | Fetching Data from sql server for table-: OQaD .............. 2026-06-22 12:43:26 | INFO | Fetching quiz_empids data for EMPID and Visitid 2026-06-22 12:43:57 | INFO | Fetched 291,158 total empid and visitdate fetched for OQAD from SQL Server 2026-06-22 12:43:59 | INFO | Fetched 290,722 matched empids fetched for OQAD 2026-06-22 12:43:59 | INFO | Exists: True 2026-06-22 12:43:59 | INFO | Path: D:\z\data_move\src\sql\fact\OQaD.sql 2026-06-22 12:43:59 | INFO | Fetching data for 290,722 EMPIDs 2026-06-22 12:43:59 | INFO | Fetching OQaD data for run_date=2026-06-21 2026-06-22 12:44:40 | INFO | fn name is fetch_OQad ------Fetched 0 rows 2026-06-22 12:44:40 | INFO | Fetched 0 rows from SQL Server 2026-06-22 12:44:40 | INFO | Fetched total row -: 0 from sql server for table-:OQaD ...........!!! 2026-06-22 12:44:40 | WARNING | OQaD returned no rows 2026-06-22 12:44:40 | INFO | ================================================================================ 2026-06-22 12:44:40 | INFO | Processing Table: Survey | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:44:40 | INFO | Fetching Data from sql server for table-: Survey .............. 2026-06-22 12:44:40 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:44:40 | INFO | Fetching data for 569 MIDs 2026-06-22 12:44:40 | INFO | Fetched 165 rows from SQL Server 2026-06-22 12:44:40 | INFO | Fetched total row -: 165 from sql server for table-:Survey ...........!!! 2026-06-22 12:44:40 | INFO | Fetched 165 rows 2026-06-22 12:44:40 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _ 2026-06-22 12:44:40 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:44:41 | INFO | Survey: inserted 165 rows into ClickHouse 2026-06-22 12:44:41 | INFO | Survey loaded successfully (165 rows) 2026-06-22 12:44:41 | INFO | ================================================================================ 2026-06-22 12:44:41 | INFO | Processing Table: additional_visibility | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:44:41 | INFO | Fetching Data from sql server for table-: additional_visibility .............. 2026-06-22 12:44:41 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 12:44:41 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:44:41 | INFO | Fetching data for 569 MIDs 2026-06-22 12:44:41 | INFO | Fetched 1,579 rows from SQL Server 2026-06-22 12:44:41 | INFO | Fetched total row -: 1579 from sql server for table-:additional_visibility ...........!!! 2026-06-22 12:44:41 | INFO | Fetched 1579 rows 2026-06-22 12:44:41 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _ 2026-06-22 12:44:41 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:44:42 | INFO | additional_visibility: inserted 1,579 rows into ClickHouse 2026-06-22 12:44:42 | INFO | additional_visibility loaded successfully (1579 rows) 2026-06-22 12:44:42 | INFO | ================================================================================ 2026-06-22 12:44:42 | INFO | Processing Table: Coverage | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:44:42 | INFO | Fetching Data from sql server for table-: Coverage .............. 2026-06-22 12:44:42 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:44:42 | INFO | Fetching data for 569 MIDs 2026-06-22 12:44:58 | INFO | Fetched 512 rows from SQL Server 2026-06-22 12:44:58 | INFO | Fetched total row -: 512 from sql server for table-:Coverage ...........!!! 2026-06-22 12:44:58 | INFO | Fetched 512 rows 2026-06-22 12:44:58 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _ 2026-06-22 12:44:58 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:44:59 | INFO | Coverage: inserted 512 rows into ClickHouse 2026-06-22 12:44:59 | INFO | Coverage loaded successfully (512 rows) 2026-06-22 12:44:59 | INFO | ================================================================================ 2026-06-22 12:44:59 | INFO | Processing Table: Login | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 12:44:59 | INFO | Fetching Data from sql server for table-: Login .............. 2026-06-22 12:44:59 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:44:59 | INFO | Fetching data for 569 MIDs 2026-06-22 12:45:02 | INFO | Fetched 454 rows from SQL Server 2026-06-22 12:45:02 | INFO | Fetched total row -: 454 from sql server for table-:Login ...........!!! 2026-06-22 12:45:02 | INFO | Fetched 454 rows 2026-06-22 12:45:02 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _ 2026-06-22 12:45:02 | INFO | No delete logic required for Login 2026-06-22 12:45:02 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:45:03 | INFO | Login: inserted 454 rows into ClickHouse 2026-06-22 12:45:03 | INFO | Login loaded successfully (454 rows) 2026-06-22 12:45:03 | INFO | ================================================================================ 2026-06-22 12:45:03 | INFO | Processing Table: Stock_Details | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:45:03 | INFO | Fetching Data from sql server for table-: Stock_Details .............. 2026-06-22 12:45:03 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:45:03 | INFO | Fetching data for 569 MIDs 2026-06-22 12:45:19 | INFO | Fetched 27,359 rows from SQL Server 2026-06-22 12:45:19 | INFO | Fetched total row -: 27359 from sql server for table-:Stock_Details ...........!!! 2026-06-22 12:45:19 | INFO | Fetched 27359 rows 2026-06-22 12:45:19 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _ 2026-06-22 12:45:19 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:45:19 | INFO | Stock_Details: inserted 27,359 rows into ClickHouse 2026-06-22 12:45:19 | INFO | Stock_Details loaded successfully (27359 rows) 2026-06-22 12:45:19 | INFO | ================================================================================ 2026-06-22 12:45:19 | INFO | Processing Table: Attendance | Table type is -: FACT | Based on -run_date and operation is used -DELETE+INSERT 2026-06-22 12:45:19 | INFO | Fetching Data from sql server for table-: Attendance .............. 2026-06-22 12:45:19 | INFO | Fetching Attendance data from 2026-06-06 to 2026-06-21 2026-06-22 12:47:52 | INFO | Fetched 121,639 attendance rows for 544 employees 2026-06-22 12:47:52 | INFO | Fetched total row -: 121639 from sql server for table-:Attendance ...........!!! 2026-06-22 12:47:52 | INFO | Fetched 121639 rows 2026-06-22 12:47:52 | INFO | Truncated table Attendance 2026-06-22 12:47:52 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:47:54 | INFO | Attendance: inserted 121,639 rows into ClickHouse 2026-06-22 12:47:54 | INFO | Attendance loaded successfully (121639 rows) 2026-06-22 12:47:54 | INFO | ================================================================================ 2026-06-22 12:47:54 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 12:47:54 | INFO | Fetching Data from sql server for table-: Journey_Plan .............. 2026-06-22 12:47:54 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 12:47:54 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:47:54 | INFO | Fetching data for 569 MIDs 2026-06-22 12:48:07 | INFO | Fetched 21,621 rows from SQL Server 2026-06-22 12:48:07 | INFO | Fetched total row -: 21621 from sql server for table-:Journey_Plan ...........!!! 2026-06-22 12:48:07 | INFO | Fetched 21621 rows 2026-06-22 12:48:07 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _ 2026-06-22 12:48:07 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:48:08 | INFO | Journey_Plan: inserted 21,621 rows into ClickHouse 2026-06-22 12:48:08 | INFO | Journey_Plan loaded successfully (21621 rows) 2026-06-22 12:48:08 | INFO | ================================================================================ 2026-06-22 12:48:08 | INFO | Processing Table: Web_Logins | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 12:48:08 | INFO | Fetching Data from sql server for table-: Web_Logins .............. 2026-06-22 12:48:08 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 12:48:08 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 12:48:08 | INFO | Fetching data for 569 MIDs 2026-06-22 12:48:08 | INFO | Fetched 12 rows from SQL Server 2026-06-22 12:48:08 | INFO | Fetched total row -: 12 from sql server for table-:Web_Logins ...........!!! 2026-06-22 12:48:08 | INFO | Fetched 12 rows 2026-06-22 12:48:08 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _ 2026-06-22 12:48:09 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 12:48:09 | INFO | Web_Logins: inserted 12 rows into ClickHouse 2026-06-22 12:48:09 | INFO | Web_Logins loaded successfully (12 rows) 2026-06-22 12:48:09 | INFO | ================================================================================ 2026-06-22 12:48:09 | INFO | Processing Table: Promotion | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 12:48:09 | INFO | Fetching Data from sql server for table-: Promotion .............. 2026-06-22 12:48:09 | ERROR | Failed processing table Promotion Traceback (most recent call last): File "D:\z\data_move\main2.py", line 158, in main fn = globals()[fn_name] ~~~~~~~~~^^^^^^^^^ KeyError: 'fetch_Promotion' 2026-06-22 13:04:30 | INFO | ================================================================================ 2026-06-22 13:04:30 | INFO | Hello from data-move Python data pipeline! 2026-06-22 13:04:30 | INFO | Pipeline Run Date: 2026-06-21 2026-06-22 13:04:30 | INFO | Connecting to databases... 2026-06-22 13:04:30 | INFO | 2026-06-22 13:04:31 | INFO | 2026-06-22 13:04:32 | INFO | Database connections established 2026-06-22 13:04:32 | INFO | Collecting MIDs for: 2026-06-21 2026-06-22 13:04:32 | INFO | Found 569 MIDs 2026-06-22 13:04:32 | INFO | ================================================================================ 2026-06-22 13:04:32 | INFO | Processing Table: Promotion | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:04:32 | INFO | Fetching Data from sql server for table-: Promotion .............. 2026-06-22 13:04:32 | INFO | Fetching data for 569 MIDs 2026-06-22 13:04:36 | INFO | Fetched 4,223 rows from SQL Server 2026-06-22 13:04:36 | INFO | Fetched total row -: 4223 from sql server for table-:Promotion ...........!!! 2026-06-22 13:04:36 | INFO | Fetched 4223 rows 2026-06-22 13:04:36 | INFO | Creating table Promotion 2026-06-22 13:04:36 | INFO | Table ready: Promotion 2026-06-22 13:04:36 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:04:36 | INFO | Promotion: inserted 4,223 rows into ClickHouse 2026-06-22 13:04:36 | INFO | Promotion loaded successfully (4223 rows) 2026-06-22 13:04:36 | INFO | ================================================================================ 2026-06-22 13:04:36 | INFO | Pipeline Completed Successfully 2026-06-22 13:04:36 | INFO | ================================================================================ 2026-06-22 13:04:41 | INFO | ================================================================================ 2026-06-22 13:04:41 | INFO | Hello from data-move Python data pipeline! 2026-06-22 13:04:41 | INFO | Pipeline Run Date: 2026-06-21 2026-06-22 13:04:41 | INFO | Connecting to databases... 2026-06-22 13:04:42 | INFO | 2026-06-22 13:04:43 | INFO | 2026-06-22 13:04:44 | INFO | Database connections established 2026-06-22 13:04:44 | INFO | Collecting MIDs for: 2026-06-21 2026-06-22 13:04:44 | INFO | Found 569 MIDs 2026-06-22 13:04:44 | INFO | ================================================================================ 2026-06-22 13:04:44 | INFO | Processing Table: Promotion | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:04:44 | INFO | Fetching Data from sql server for table-: Promotion .............. 2026-06-22 13:04:44 | INFO | Fetching data for 569 MIDs 2026-06-22 13:04:54 | INFO | Fetched 4,223 rows from SQL Server 2026-06-22 13:04:54 | INFO | Fetched total row -: 4223 from sql server for table-:Promotion ...........!!! 2026-06-22 13:04:54 | INFO | Fetched 4223 rows 2026-06-22 13:04:54 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _ 2026-06-22 13:04:54 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:04:55 | INFO | Promotion: inserted 4,223 rows into ClickHouse 2026-06-22 13:04:55 | INFO | Promotion loaded successfully (4223 rows) 2026-06-22 13:04:55 | INFO | ================================================================================ 2026-06-22 13:04:55 | INFO | Pipeline Completed Successfully 2026-06-22 13:04:55 | INFO | ================================================================================ 2026-06-22 13:05:14 | INFO | ================================================================================ 2026-06-22 13:05:14 | INFO | Hello from data-move Python data pipeline! 2026-06-22 13:05:14 | INFO | Pipeline Run Date: 2026-06-21 2026-06-22 13:05:14 | INFO | Connecting to databases... 2026-06-22 13:05:15 | INFO | 2026-06-22 13:05:16 | INFO | 2026-06-22 13:05:17 | INFO | Database connections established 2026-06-22 13:05:17 | INFO | Collecting MIDs for: 2026-06-21 2026-06-22 13:05:17 | INFO | Found 569 MIDs 2026-06-22 13:05:17 | INFO | ================================================================================ 2026-06-22 13:05:17 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:05:17 | INFO | Fetching Data from sql server for table-: SOS_OneApp .............. 2026-06-22 13:05:17 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 13:05:17 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:05:17 | INFO | Fetching data for 569 MIDs 2026-06-22 13:05:19 | INFO | Fetched 1,907 rows from SQL Server 2026-06-22 13:05:19 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!! 2026-06-22 13:05:19 | INFO | Fetched 1907 rows 2026-06-22 13:05:19 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _ 2026-06-22 13:05:20 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:05:20 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse 2026-06-22 13:05:20 | INFO | SOS_OneApp loaded successfully (1907 rows) 2026-06-22 13:05:20 | INFO | ================================================================================ 2026-06-22 13:05:20 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 13:05:20 | INFO | Fetching Data from sql server for table-: OQaD .............. 2026-06-22 13:05:20 | INFO | Fetching quiz_empids data for EMPID and Visitid 2026-06-22 13:05:32 | INFO | Fetched 291,165 total empid and visitdate fetched for OQAD from SQL Server 2026-06-22 13:05:34 | INFO | Fetched 290,722 matched empids fetched for OQAD 2026-06-22 13:05:34 | INFO | Exists: True 2026-06-22 13:05:34 | INFO | Path: D:\z\data_move\src\sql\fact\OQaD.sql 2026-06-22 13:05:34 | INFO | Fetching data for 290,722 EMPIDs 2026-06-22 13:05:34 | INFO | Fetching OQaD data for run_date=2026-06-21 2026-06-22 13:06:10 | INFO | fn name is fetch_OQad ------Fetched 0 rows 2026-06-22 13:06:10 | INFO | Fetched 0 rows from SQL Server 2026-06-22 13:06:10 | INFO | Fetched total row -: 0 from sql server for table-:OQaD ...........!!! 2026-06-22 13:06:10 | WARNING | OQaD returned no rows 2026-06-22 13:06:10 | INFO | ================================================================================ 2026-06-22 13:06:10 | INFO | Processing Table: Survey | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:06:10 | INFO | Fetching Data from sql server for table-: Survey .............. 2026-06-22 13:06:10 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:06:10 | INFO | Fetching data for 569 MIDs 2026-06-22 13:06:10 | INFO | Fetched 165 rows from SQL Server 2026-06-22 13:06:10 | INFO | Fetched total row -: 165 from sql server for table-:Survey ...........!!! 2026-06-22 13:06:10 | INFO | Fetched 165 rows 2026-06-22 13:06:11 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _ 2026-06-22 13:06:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:06:11 | INFO | Survey: inserted 165 rows into ClickHouse 2026-06-22 13:06:11 | INFO | Survey loaded successfully (165 rows) 2026-06-22 13:06:11 | INFO | ================================================================================ 2026-06-22 13:06:11 | INFO | Processing Table: additional_visibility | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:06:11 | INFO | Fetching Data from sql server for table-: additional_visibility .............. 2026-06-22 13:06:11 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 13:06:11 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:06:11 | INFO | Fetching data for 569 MIDs 2026-06-22 13:06:11 | INFO | Fetched 1,579 rows from SQL Server 2026-06-22 13:06:11 | INFO | Fetched total row -: 1579 from sql server for table-:additional_visibility ...........!!! 2026-06-22 13:06:11 | INFO | Fetched 1579 rows 2026-06-22 13:06:11 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _ 2026-06-22 13:06:12 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:06:12 | INFO | additional_visibility: inserted 1,579 rows into ClickHouse 2026-06-22 13:06:12 | INFO | additional_visibility loaded successfully (1579 rows) 2026-06-22 13:06:12 | INFO | ================================================================================ 2026-06-22 13:06:12 | INFO | Processing Table: Coverage | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:06:12 | INFO | Fetching Data from sql server for table-: Coverage .............. 2026-06-22 13:06:12 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:06:12 | INFO | Fetching data for 569 MIDs 2026-06-22 13:06:27 | INFO | Fetched 512 rows from SQL Server 2026-06-22 13:06:27 | INFO | Fetched total row -: 512 from sql server for table-:Coverage ...........!!! 2026-06-22 13:06:27 | INFO | Fetched 512 rows 2026-06-22 13:06:27 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _ 2026-06-22 13:06:27 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:06:28 | INFO | Coverage: inserted 512 rows into ClickHouse 2026-06-22 13:06:28 | INFO | Coverage loaded successfully (512 rows) 2026-06-22 13:06:28 | INFO | ================================================================================ 2026-06-22 13:06:28 | INFO | Processing Table: Login | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 13:06:28 | INFO | Fetching Data from sql server for table-: Login .............. 2026-06-22 13:06:28 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:06:28 | INFO | Fetching data for 569 MIDs 2026-06-22 13:06:29 | INFO | Fetched 454 rows from SQL Server 2026-06-22 13:06:29 | INFO | Fetched total row -: 454 from sql server for table-:Login ...........!!! 2026-06-22 13:06:29 | INFO | Fetched 454 rows 2026-06-22 13:06:29 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _ 2026-06-22 13:06:29 | INFO | No delete logic required for Login 2026-06-22 13:06:29 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:06:30 | INFO | Login: inserted 454 rows into ClickHouse 2026-06-22 13:06:30 | INFO | Login loaded successfully (454 rows) 2026-06-22 13:06:30 | INFO | ================================================================================ 2026-06-22 13:06:30 | INFO | Processing Table: Stock_Details | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:06:30 | INFO | Fetching Data from sql server for table-: Stock_Details .............. 2026-06-22 13:06:30 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:06:30 | INFO | Fetching data for 569 MIDs 2026-06-22 13:06:35 | INFO | Fetched 27,359 rows from SQL Server 2026-06-22 13:06:35 | INFO | Fetched total row -: 27359 from sql server for table-:Stock_Details ...........!!! 2026-06-22 13:06:35 | INFO | Fetched 27359 rows 2026-06-22 13:06:35 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _ 2026-06-22 13:06:35 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:06:35 | INFO | Stock_Details: inserted 27,359 rows into ClickHouse 2026-06-22 13:06:35 | INFO | Stock_Details loaded successfully (27359 rows) 2026-06-22 13:06:35 | INFO | ================================================================================ 2026-06-22 13:06:35 | INFO | Processing Table: Attendance | Table type is -: FACT | Based on -run_date and operation is used -DELETE+INSERT 2026-06-22 13:06:35 | INFO | Fetching Data from sql server for table-: Attendance .............. 2026-06-22 13:06:35 | INFO | Fetching Attendance data from 2026-06-06 to 2026-06-21 2026-06-22 13:06:51 | INFO | Fetched 121,639 attendance rows for 544 employees 2026-06-22 13:06:51 | INFO | Fetched total row -: 121639 from sql server for table-:Attendance ...........!!! 2026-06-22 13:06:51 | INFO | Fetched 121639 rows 2026-06-22 13:06:51 | INFO | Truncated table Attendance 2026-06-22 13:06:51 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:06:56 | INFO | Attendance: inserted 121,639 rows into ClickHouse 2026-06-22 13:06:56 | INFO | Attendance loaded successfully (121639 rows) 2026-06-22 13:06:56 | INFO | ================================================================================ 2026-06-22 13:06:56 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 13:06:56 | INFO | Fetching Data from sql server for table-: Journey_Plan .............. 2026-06-22 13:06:56 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 13:06:56 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:06:56 | INFO | Fetching data for 569 MIDs 2026-06-22 13:06:57 | INFO | Fetched 21,625 rows from SQL Server 2026-06-22 13:06:57 | INFO | Fetched total row -: 21625 from sql server for table-:Journey_Plan ...........!!! 2026-06-22 13:06:57 | INFO | Fetched 21625 rows 2026-06-22 13:06:57 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _ 2026-06-22 13:06:57 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:06:57 | INFO | Journey_Plan: inserted 21,625 rows into ClickHouse 2026-06-22 13:06:57 | INFO | Journey_Plan loaded successfully (21625 rows) 2026-06-22 13:06:57 | INFO | ================================================================================ 2026-06-22 13:06:57 | INFO | Processing Table: Web_Logins | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 13:06:57 | INFO | Fetching Data from sql server for table-: Web_Logins .............. 2026-06-22 13:06:57 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 13:06:57 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:06:57 | INFO | Fetching data for 569 MIDs 2026-06-22 13:06:58 | INFO | Fetched 12 rows from SQL Server 2026-06-22 13:06:58 | INFO | Fetched total row -: 12 from sql server for table-:Web_Logins ...........!!! 2026-06-22 13:06:58 | INFO | Fetched 12 rows 2026-06-22 13:06:58 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _ 2026-06-22 13:06:58 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:06:58 | INFO | Web_Logins: inserted 12 rows into ClickHouse 2026-06-22 13:06:58 | INFO | Web_Logins loaded successfully (12 rows) 2026-06-22 13:06:58 | INFO | ================================================================================ 2026-06-22 13:06:58 | INFO | Processing Table: Promotion | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:06:58 | INFO | Fetching Data from sql server for table-: Promotion .............. 2026-06-22 13:06:58 | INFO | Fetching data for 569 MIDs 2026-06-22 13:07:00 | INFO | Fetched 4,223 rows from SQL Server 2026-06-22 13:07:00 | INFO | Fetched total row -: 4223 from sql server for table-:Promotion ...........!!! 2026-06-22 13:07:00 | INFO | Fetched 4223 rows 2026-06-22 13:07:00 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _ 2026-06-22 13:07:01 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:07:01 | INFO | Promotion: inserted 4,223 rows into ClickHouse 2026-06-22 13:07:01 | INFO | Promotion loaded successfully (4223 rows) 2026-06-22 13:07:01 | INFO | ================================================================================ 2026-06-22 13:07:01 | INFO | Processing Table: PaidVisibility | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:07:01 | INFO | Fetching Data from sql server for table-: PaidVisibility .............. 2026-06-22 13:07:01 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 13:07:01 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:07:01 | INFO | Fetching data for 569 MIDs 2026-06-22 13:07:02 | INFO | Fetched 1,284 rows from SQL Server 2026-06-22 13:07:02 | INFO | Fetched total row -: 1284 from sql server for table-:PaidVisibility ...........!!! 2026-06-22 13:07:02 | INFO | Fetched 1284 rows 2026-06-22 13:07:02 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _ 2026-06-22 13:07:02 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:07:03 | INFO | PaidVisibility: inserted 1,284 rows into ClickHouse 2026-06-22 13:07:03 | INFO | PaidVisibility loaded successfully (1284 rows) 2026-06-22 13:07:03 | INFO | ================================================================================ 2026-06-22 13:07:03 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT 2026-06-22 13:07:03 | INFO | Fetching Data from sql server for table-: Store_Master .............. 2026-06-22 13:07:03 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:07:03 | INFO | Fetching in progress .... 2026-06-22 13:07:05 | INFO | Fetched 5,998 rows from SQL Server 2026-06-22 13:07:05 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!! 2026-06-22 13:07:05 | INFO | Fetched 5998 rows 2026-06-22 13:07:05 | INFO | Truncated table Store_Master 2026-06-22 13:07:05 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:07:07 | INFO | Store_Master: inserted 5,998 rows into ClickHouse 2026-06-22 13:07:07 | INFO | Store_Master loaded successfully (5998 rows) 2026-06-22 13:07:07 | INFO | ================================================================================ 2026-06-22 13:07:07 | INFO | Processing Table: SKU_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT 2026-06-22 13:07:07 | INFO | Fetching Data from sql server for table-: SKU_Master .............. 2026-06-22 13:07:07 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:07:07 | INFO | Fetching in progress .... 2026-06-22 13:07:07 | INFO | Fetched 160 rows from SQL Server 2026-06-22 13:07:07 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!! 2026-06-22 13:07:07 | INFO | Fetched 160 rows 2026-06-22 13:07:07 | INFO | Truncated table SKU_Master 2026-06-22 13:07:07 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:07:07 | INFO | SKU_Master: inserted 160 rows into ClickHouse 2026-06-22 13:07:07 | INFO | SKU_Master loaded successfully (160 rows) 2026-06-22 13:07:07 | INFO | ================================================================================ 2026-06-22 13:07:07 | INFO | Processing Table: display_master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT 2026-06-22 13:07:07 | INFO | Fetching Data from sql server for table-: display_master .............. 2026-06-22 13:07:07 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:07:07 | INFO | Fetching in progress .... 2026-06-22 13:07:07 | INFO | Fetched 135 rows from SQL Server 2026-06-22 13:07:07 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!! 2026-06-22 13:07:07 | INFO | Fetched 135 rows 2026-06-22 13:07:07 | INFO | Truncated table display_master 2026-06-22 13:07:07 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:07:07 | INFO | display_master: inserted 135 rows into ClickHouse 2026-06-22 13:07:07 | INFO | display_master loaded successfully (135 rows) 2026-06-22 13:07:07 | INFO | ================================================================================ 2026-06-22 13:07:07 | INFO | Processing Table: Employee_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT 2026-06-22 13:07:07 | INFO | Fetching Data from sql server for table-: Employee_Master .............. 2026-06-22 13:07:07 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:07:07 | INFO | Fetching in progress .... 2026-06-22 13:07:08 | INFO | Fetched 2,288 rows from SQL Server 2026-06-22 13:07:08 | INFO | Fetched total row -: 2288 from sql server for table-:Employee_Master ...........!!! 2026-06-22 13:07:08 | INFO | Fetched 2288 rows 2026-06-22 13:07:08 | INFO | Truncated table Employee_Master 2026-06-22 13:07:08 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:07:08 | INFO | Employee_Master: inserted 2,288 rows into ClickHouse 2026-06-22 13:07:08 | INFO | Employee_Master loaded successfully (2288 rows) 2026-06-22 13:07:08 | INFO | ================================================================================ 2026-06-22 13:07:08 | INFO | Processing Table: coverage_remarks | Table type is -: DIMENSION | Based on -reason_id and operation is used -DELETE+INSERT 2026-06-22 13:07:08 | INFO | Fetching Data from sql server for table-: coverage_remarks .............. 2026-06-22 13:07:09 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:07:09 | INFO | Fetching in progress .... 2026-06-22 13:07:09 | INFO | Fetched 0 rows from SQL Server 2026-06-22 13:07:09 | INFO | Fetched 0 rows from SQL Server 2026-06-22 13:07:09 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!! 2026-06-22 13:07:09 | WARNING | coverage_remarks returned no rows 2026-06-22 13:07:09 | INFO | ================================================================================ 2026-06-22 13:07:09 | INFO | Processing Table: mapping_storevisibility | Table type is -: BRIDGE | Based on -run_date and operation is used -ONLY_INSERT 2026-06-22 13:07:09 | INFO | Fetching Data from sql server for table-: mapping_storevisibility .............. 2026-06-22 13:07:10 | WARNING | Table 'mapping_storevisibility' does not exist. During collecting store_ids 2026-06-22 13:07:10 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:07:10 | INFO | Fetching in progress .... 2026-06-22 13:07:10 | INFO | Fetched 0 rows from SQL Server 2026-06-22 13:07:10 | INFO | Fetched 0 rows from SQL Server 2026-06-22 13:07:10 | INFO | Fetched total row -: 0 from sql server for table-:mapping_storevisibility ...........!!! 2026-06-22 13:07:10 | WARNING | mapping_storevisibility returned no rows 2026-06-22 13:07:10 | INFO | ================================================================================ 2026-06-22 13:07:10 | INFO | Processing Table: Master_VisibilityReason | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT 2026-06-22 13:07:10 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason .............. 2026-06-22 13:07:10 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:07:10 | INFO | Fetching in progress .... 2026-06-22 13:07:10 | INFO | Fetched 17 rows from SQL Server 2026-06-22 13:07:10 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!! 2026-06-22 13:07:10 | INFO | Fetched 17 rows 2026-06-22 13:07:11 | INFO | Truncated table Master_VisibilityReason 2026-06-22 13:07:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:07:11 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse 2026-06-22 13:07:11 | INFO | Master_VisibilityReason loaded successfully (17 rows) 2026-06-22 13:07:11 | INFO | ================================================================================ 2026-06-22 13:07:11 | INFO | Processing Table: Master_VisibilityDefinition | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT 2026-06-22 13:07:11 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition .............. 2026-06-22 13:07:11 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:07:11 | INFO | Fetching in progress .... 2026-06-22 13:07:11 | INFO | Fetched 861 rows from SQL Server 2026-06-22 13:07:11 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!! 2026-06-22 13:07:11 | INFO | Fetched 861 rows 2026-06-22 13:07:11 | INFO | Truncated table Master_VisibilityDefinition 2026-06-22 13:07:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:07:11 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse 2026-06-22 13:07:11 | INFO | Master_VisibilityDefinition loaded successfully (861 rows) 2026-06-22 13:07:11 | INFO | ================================================================================ 2026-06-22 13:07:11 | INFO | Processing Table: Master_Salesterritorylayer | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT 2026-06-22 13:07:11 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer .............. 2026-06-22 13:07:11 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:07:11 | INFO | Fetching in progress .... 2026-06-22 13:07:11 | INFO | Fetched 33 rows from SQL Server 2026-06-22 13:07:11 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!! 2026-06-22 13:07:11 | INFO | Fetched 33 rows 2026-06-22 13:07:11 | INFO | Truncated table Master_Salesterritorylayer 2026-06-22 13:07:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:07:12 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse 2026-06-22 13:07:12 | INFO | Master_Salesterritorylayer loaded successfully (33 rows) 2026-06-22 13:07:12 | INFO | ================================================================================ 2026-06-22 13:07:12 | INFO | Pipeline Completed Successfully 2026-06-22 13:07:12 | INFO | ================================================================================ 2026-06-22 13:15:45 | INFO | ================================================================================ 2026-06-22 13:15:45 | INFO | Hello from data-move Python data pipeline! 2026-06-22 13:15:45 | INFO | Pipeline Run Date: 2026-06-21 2026-06-22 13:15:45 | INFO | Connecting to databases... 2026-06-22 13:15:46 | INFO | 2026-06-22 13:15:47 | INFO | 2026-06-22 13:15:48 | INFO | Database connections established 2026-06-22 13:15:48 | INFO | Collecting MIDs for: 2026-06-21 2026-06-22 13:15:49 | INFO | Found 569 MIDs 2026-06-22 13:15:49 | INFO | ================================================================================ 2026-06-22 13:15:49 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:15:49 | INFO | Fetching Data from sql server for table-: SOS_OneApp .............. 2026-06-22 13:15:49 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 13:15:49 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:15:49 | INFO | Fetching data for 569 MIDs 2026-06-22 13:15:51 | INFO | Fetched 1,907 rows from SQL Server 2026-06-22 13:15:51 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!! 2026-06-22 13:15:51 | INFO | Fetched 1907 rows 2026-06-22 13:15:51 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _ 2026-06-22 13:15:51 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:15:52 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse 2026-06-22 13:15:52 | INFO | SOS_OneApp loaded successfully (1907 rows) 2026-06-22 13:15:52 | INFO | ================================================================================ 2026-06-22 13:15:52 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 13:15:52 | INFO | Fetching Data from sql server for table-: OQaD .............. 2026-06-22 13:15:52 | INFO | Fetching quiz_empids data for EMPID and Visitid 2026-06-22 13:16:09 | INFO | Fetched 291,166 total empid and visitdate fetched for OQAD from SQL Server 2026-06-22 13:16:10 | INFO | Fetched 290,722 matched empids fetched for OQAD 2026-06-22 13:16:10 | INFO | Exists: True 2026-06-22 13:16:10 | INFO | Path: D:\z\data_move\src\sql\fact\OQaD.sql 2026-06-22 13:16:10 | INFO | Fetching data for 290,722 EMPIDs 2026-06-22 13:16:10 | INFO | Fetching OQaD data for run_date=2026-06-21 2026-06-22 13:16:46 | INFO | fn name is fetch_OQad ------Fetched 0 rows 2026-06-22 13:16:46 | INFO | Fetched 0 rows from SQL Server 2026-06-22 13:16:46 | INFO | Fetched total row -: 0 from sql server for table-:OQaD ...........!!! 2026-06-22 13:16:46 | WARNING | OQaD returned no rows 2026-06-22 13:16:46 | INFO | ================================================================================ 2026-06-22 13:16:46 | INFO | Processing Table: Survey | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:16:46 | INFO | Fetching Data from sql server for table-: Survey .............. 2026-06-22 13:16:46 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:16:46 | INFO | Fetching data for 569 MIDs 2026-06-22 13:16:47 | INFO | Fetched 165 rows from SQL Server 2026-06-22 13:16:47 | INFO | Fetched total row -: 165 from sql server for table-:Survey ...........!!! 2026-06-22 13:16:47 | INFO | Fetched 165 rows 2026-06-22 13:16:47 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _ 2026-06-22 13:16:47 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:16:47 | INFO | Survey: inserted 165 rows into ClickHouse 2026-06-22 13:16:47 | INFO | Survey loaded successfully (165 rows) 2026-06-22 13:16:47 | INFO | ================================================================================ 2026-06-22 13:16:47 | INFO | Processing Table: additional_visibility | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:16:47 | INFO | Fetching Data from sql server for table-: additional_visibility .............. 2026-06-22 13:16:47 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 13:16:47 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:16:47 | INFO | Fetching data for 569 MIDs 2026-06-22 13:16:47 | INFO | Fetched 1,579 rows from SQL Server 2026-06-22 13:16:47 | INFO | Fetched total row -: 1579 from sql server for table-:additional_visibility ...........!!! 2026-06-22 13:16:47 | INFO | Fetched 1579 rows 2026-06-22 13:16:48 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _ 2026-06-22 13:16:48 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:16:48 | INFO | additional_visibility: inserted 1,579 rows into ClickHouse 2026-06-22 13:16:48 | INFO | additional_visibility loaded successfully (1579 rows) 2026-06-22 13:16:48 | INFO | ================================================================================ 2026-06-22 13:16:48 | INFO | Processing Table: Coverage | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:16:48 | INFO | Fetching Data from sql server for table-: Coverage .............. 2026-06-22 13:16:48 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:16:48 | INFO | Fetching data for 569 MIDs 2026-06-22 13:17:03 | INFO | Fetched 512 rows from SQL Server 2026-06-22 13:17:03 | INFO | Fetched total row -: 512 from sql server for table-:Coverage ...........!!! 2026-06-22 13:17:03 | INFO | Fetched 512 rows 2026-06-22 13:17:03 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _ 2026-06-22 13:17:03 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:03 | INFO | Coverage: inserted 512 rows into ClickHouse 2026-06-22 13:17:03 | INFO | Coverage loaded successfully (512 rows) 2026-06-22 13:17:03 | INFO | ================================================================================ 2026-06-22 13:17:03 | INFO | Processing Table: Login | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 13:17:03 | INFO | Fetching Data from sql server for table-: Login .............. 2026-06-22 13:17:03 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:17:03 | INFO | Fetching data for 569 MIDs 2026-06-22 13:17:05 | INFO | Fetched 454 rows from SQL Server 2026-06-22 13:17:05 | INFO | Fetched total row -: 454 from sql server for table-:Login ...........!!! 2026-06-22 13:17:05 | INFO | Fetched 454 rows 2026-06-22 13:17:05 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _ 2026-06-22 13:17:05 | INFO | No delete logic required for Login 2026-06-22 13:17:05 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:05 | INFO | Login: inserted 454 rows into ClickHouse 2026-06-22 13:17:05 | INFO | Login loaded successfully (454 rows) 2026-06-22 13:17:05 | INFO | ================================================================================ 2026-06-22 13:17:05 | INFO | Processing Table: Stock_Details | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:17:05 | INFO | Fetching Data from sql server for table-: Stock_Details .............. 2026-06-22 13:17:05 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:17:05 | INFO | Fetching data for 569 MIDs 2026-06-22 13:17:10 | INFO | Fetched 27,359 rows from SQL Server 2026-06-22 13:17:10 | INFO | Fetched total row -: 27359 from sql server for table-:Stock_Details ...........!!! 2026-06-22 13:17:10 | INFO | Fetched 27359 rows 2026-06-22 13:17:10 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _ 2026-06-22 13:17:10 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:11 | INFO | Stock_Details: inserted 27,359 rows into ClickHouse 2026-06-22 13:17:11 | INFO | Stock_Details loaded successfully (27359 rows) 2026-06-22 13:17:11 | INFO | ================================================================================ 2026-06-22 13:17:11 | INFO | Processing Table: Attendance | Table type is -: FACT | Based on -run_date and operation is used -DELETE+INSERT 2026-06-22 13:17:11 | INFO | Fetching Data from sql server for table-: Attendance .............. 2026-06-22 13:17:11 | INFO | Fetching Attendance data from 2026-06-06 to 2026-06-21 2026-06-22 13:17:27 | INFO | Fetched 121,639 attendance rows for 544 employees 2026-06-22 13:17:27 | INFO | Fetched total row -: 121639 from sql server for table-:Attendance ...........!!! 2026-06-22 13:17:27 | INFO | Fetched 121639 rows 2026-06-22 13:17:27 | INFO | Truncated table Attendance 2026-06-22 13:17:27 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:28 | INFO | Attendance: inserted 121,639 rows into ClickHouse 2026-06-22 13:17:28 | INFO | Attendance loaded successfully (121639 rows) 2026-06-22 13:17:28 | INFO | ================================================================================ 2026-06-22 13:17:28 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 13:17:28 | INFO | Fetching Data from sql server for table-: Journey_Plan .............. 2026-06-22 13:17:28 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 13:17:28 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:17:28 | INFO | Fetching data for 569 MIDs 2026-06-22 13:17:29 | INFO | Fetched 21,625 rows from SQL Server 2026-06-22 13:17:29 | INFO | Fetched total row -: 21625 from sql server for table-:Journey_Plan ...........!!! 2026-06-22 13:17:29 | INFO | Fetched 21625 rows 2026-06-22 13:17:30 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _ 2026-06-22 13:17:30 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:30 | INFO | Journey_Plan: inserted 21,625 rows into ClickHouse 2026-06-22 13:17:30 | INFO | Journey_Plan loaded successfully (21625 rows) 2026-06-22 13:17:30 | INFO | ================================================================================ 2026-06-22 13:17:30 | INFO | Processing Table: Web_Logins | Table type is -: FACT | Based on -run_date and operation is used -INSERT 2026-06-22 13:17:30 | INFO | Fetching Data from sql server for table-: Web_Logins .............. 2026-06-22 13:17:30 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 13:17:30 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:17:30 | INFO | Fetching data for 569 MIDs 2026-06-22 13:17:31 | INFO | Fetched 12 rows from SQL Server 2026-06-22 13:17:31 | INFO | Fetched total row -: 12 from sql server for table-:Web_Logins ...........!!! 2026-06-22 13:17:31 | INFO | Fetched 12 rows 2026-06-22 13:17:31 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _ 2026-06-22 13:17:31 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:31 | INFO | Web_Logins: inserted 12 rows into ClickHouse 2026-06-22 13:17:31 | INFO | Web_Logins loaded successfully (12 rows) 2026-06-22 13:17:31 | INFO | ================================================================================ 2026-06-22 13:17:31 | INFO | Processing Table: Promotion | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:17:31 | INFO | Fetching Data from sql server for table-: Promotion .............. 2026-06-22 13:17:31 | INFO | Fetching data for 569 MIDs 2026-06-22 13:17:32 | INFO | Fetched 4,223 rows from SQL Server 2026-06-22 13:17:32 | INFO | Fetched total row -: 4223 from sql server for table-:Promotion ...........!!! 2026-06-22 13:17:32 | INFO | Fetched 4223 rows 2026-06-22 13:17:32 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _ 2026-06-22 13:17:32 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:33 | INFO | Promotion: inserted 4,223 rows into ClickHouse 2026-06-22 13:17:33 | INFO | Promotion loaded successfully (4223 rows) 2026-06-22 13:17:33 | INFO | ================================================================================ 2026-06-22 13:17:33 | INFO | Processing Table: PaidVisibility | Table type is -: FACT | Based on -mids and operation is used -INSERT 2026-06-22 13:17:33 | INFO | Fetching Data from sql server for table-: PaidVisibility .............. 2026-06-22 13:17:33 | INFO | Start Fetching data for these 569 MIDs 2026-06-22 13:17:33 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21 2026-06-22 13:17:33 | INFO | Fetching data for 569 MIDs 2026-06-22 13:17:34 | INFO | Fetched 1,284 rows from SQL Server 2026-06-22 13:17:34 | INFO | Fetched total row -: 1284 from sql server for table-:PaidVisibility ...........!!! 2026-06-22 13:17:34 | INFO | Fetched 1284 rows 2026-06-22 13:17:34 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _ 2026-06-22 13:17:34 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:34 | INFO | PaidVisibility: inserted 1,284 rows into ClickHouse 2026-06-22 13:17:34 | INFO | PaidVisibility loaded successfully (1284 rows) 2026-06-22 13:17:34 | INFO | ================================================================================ 2026-06-22 13:17:34 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT 2026-06-22 13:17:34 | INFO | Fetching Data from sql server for table-: Store_Master .............. 2026-06-22 13:17:34 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:17:34 | INFO | Fetching in progress .... 2026-06-22 13:17:38 | INFO | Fetched 5,998 rows from SQL Server 2026-06-22 13:17:38 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!! 2026-06-22 13:17:38 | INFO | Fetched 5998 rows 2026-06-22 13:17:38 | INFO | Truncated table Store_Master 2026-06-22 13:17:38 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:39 | INFO | Store_Master: inserted 5,998 rows into ClickHouse 2026-06-22 13:17:39 | INFO | Store_Master loaded successfully (5998 rows) 2026-06-22 13:17:39 | INFO | ================================================================================ 2026-06-22 13:17:39 | INFO | Processing Table: SKU_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT 2026-06-22 13:17:39 | INFO | Fetching Data from sql server for table-: SKU_Master .............. 2026-06-22 13:17:39 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:17:39 | INFO | Fetching in progress .... 2026-06-22 13:17:39 | INFO | Fetched 160 rows from SQL Server 2026-06-22 13:17:39 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!! 2026-06-22 13:17:39 | INFO | Fetched 160 rows 2026-06-22 13:17:39 | INFO | Truncated table SKU_Master 2026-06-22 13:17:39 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:40 | INFO | SKU_Master: inserted 160 rows into ClickHouse 2026-06-22 13:17:40 | INFO | SKU_Master loaded successfully (160 rows) 2026-06-22 13:17:40 | INFO | ================================================================================ 2026-06-22 13:17:40 | INFO | Processing Table: display_master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT 2026-06-22 13:17:40 | INFO | Fetching Data from sql server for table-: display_master .............. 2026-06-22 13:17:40 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:17:40 | INFO | Fetching in progress .... 2026-06-22 13:17:40 | INFO | Fetched 135 rows from SQL Server 2026-06-22 13:17:40 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!! 2026-06-22 13:17:40 | INFO | Fetched 135 rows 2026-06-22 13:17:40 | INFO | Truncated table display_master 2026-06-22 13:17:40 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:40 | INFO | display_master: inserted 135 rows into ClickHouse 2026-06-22 13:17:40 | INFO | display_master loaded successfully (135 rows) 2026-06-22 13:17:40 | INFO | ================================================================================ 2026-06-22 13:17:40 | INFO | Processing Table: Employee_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT 2026-06-22 13:17:40 | INFO | Fetching Data from sql server for table-: Employee_Master .............. 2026-06-22 13:17:40 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:17:40 | INFO | Fetching in progress .... 2026-06-22 13:17:41 | INFO | Fetched 2,288 rows from SQL Server 2026-06-22 13:17:41 | INFO | Fetched total row -: 2288 from sql server for table-:Employee_Master ...........!!! 2026-06-22 13:17:41 | INFO | Fetched 2288 rows 2026-06-22 13:17:41 | INFO | Truncated table Employee_Master 2026-06-22 13:17:41 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:41 | INFO | Employee_Master: inserted 2,288 rows into ClickHouse 2026-06-22 13:17:41 | INFO | Employee_Master loaded successfully (2288 rows) 2026-06-22 13:17:41 | INFO | ================================================================================ 2026-06-22 13:17:41 | INFO | Processing Table: coverage_remarks | Table type is -: DIMENSION | Based on -reason_id and operation is used -DELETE+INSERT 2026-06-22 13:17:41 | INFO | Fetching Data from sql server for table-: coverage_remarks .............. 2026-06-22 13:17:42 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:17:42 | INFO | Fetching in progress .... 2026-06-22 13:17:42 | INFO | Fetched 0 rows from SQL Server 2026-06-22 13:17:42 | INFO | Fetched 0 rows from SQL Server 2026-06-22 13:17:42 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!! 2026-06-22 13:17:42 | WARNING | coverage_remarks returned no rows 2026-06-22 13:17:42 | INFO | ================================================================================ 2026-06-22 13:17:42 | INFO | Processing Table: mapping_storevisibility | Table type is -: BRIDGE | Based on -run_date and operation is used -ONLY_INSERT 2026-06-22 13:17:42 | INFO | Fetching Data from sql server for table-: mapping_storevisibility .............. 2026-06-22 13:17:42 | WARNING | Table 'mapping_storevisibility' does not exist. During collecting store_ids 2026-06-22 13:17:42 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:17:42 | INFO | Fetching in progress .... 2026-06-22 13:17:42 | INFO | Fetched 0 rows from SQL Server 2026-06-22 13:17:42 | INFO | Fetched 0 rows from SQL Server 2026-06-22 13:17:42 | INFO | Fetched total row -: 0 from sql server for table-:mapping_storevisibility ...........!!! 2026-06-22 13:17:42 | WARNING | mapping_storevisibility returned no rows 2026-06-22 13:17:42 | INFO | ================================================================================ 2026-06-22 13:17:42 | INFO | Processing Table: Master_VisibilityReason | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT 2026-06-22 13:17:42 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason .............. 2026-06-22 13:17:42 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:17:42 | INFO | Fetching in progress .... 2026-06-22 13:17:42 | INFO | Fetched 17 rows from SQL Server 2026-06-22 13:17:42 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!! 2026-06-22 13:17:42 | INFO | Fetched 17 rows 2026-06-22 13:17:43 | INFO | Truncated table Master_VisibilityReason 2026-06-22 13:17:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:43 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse 2026-06-22 13:17:43 | INFO | Master_VisibilityReason loaded successfully (17 rows) 2026-06-22 13:17:43 | INFO | ================================================================================ 2026-06-22 13:17:43 | INFO | Processing Table: Master_VisibilityDefinition | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT 2026-06-22 13:17:43 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition .............. 2026-06-22 13:17:43 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:17:43 | INFO | Fetching in progress .... 2026-06-22 13:17:43 | INFO | Fetched 861 rows from SQL Server 2026-06-22 13:17:43 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!! 2026-06-22 13:17:43 | INFO | Fetched 861 rows 2026-06-22 13:17:43 | INFO | Truncated table Master_VisibilityDefinition 2026-06-22 13:17:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:43 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse 2026-06-22 13:17:43 | INFO | Master_VisibilityDefinition loaded successfully (861 rows) 2026-06-22 13:17:43 | INFO | ================================================================================ 2026-06-22 13:17:43 | INFO | Processing Table: Master_Salesterritorylayer | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT 2026-06-22 13:17:43 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer .............. 2026-06-22 13:17:43 | INFO | Fetching data from sql server for Master table...... 2026-06-22 13:17:43 | INFO | Fetching in progress .... 2026-06-22 13:17:43 | INFO | Fetched 33 rows from SQL Server 2026-06-22 13:17:43 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!! 2026-06-22 13:17:43 | INFO | Fetched 33 rows 2026-06-22 13:17:43 | INFO | Truncated table Master_Salesterritorylayer 2026-06-22 13:17:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _ 2026-06-22 13:17:44 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse 2026-06-22 13:17:44 | INFO | Master_Salesterritorylayer loaded successfully (33 rows) 2026-06-22 13:17:44 | INFO | ================================================================================ 2026-06-22 13:17:44 | INFO | Pipeline Completed Successfully 2026-06-22 13:17:44 | INFO | ================================================================================