Ankit Malik
1990 lines
164 KiB
Plaintext
1990 lines
164 KiB
Plaintext
2026-06-22 12:12:58 | INFO | ================================================================================
|
|
2026-06-22 12:12:58 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 12:12:58 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 12:12:58 | INFO | Connecting to databases...
|
|
2026-06-22 12:12:59 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001C383F050D0>
|
|
2026-06-22 12:13:03 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001C3850E60D0>
|
|
2026-06-22 12:13:03 | INFO | Database connections established
|
|
2026-06-22 12:13:03 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 12:13:04 | INFO | Found 569 MIDs
|
|
2026-06-22 12:13:05 | INFO | ================================================================================
|
|
2026-06-22 12:13:05 | INFO | Processing Table: Master_VisibilityReason | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
|
|
2026-06-22 12:13:05 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason ..............
|
|
2026-06-22 12:13:05 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 12:13:05 | INFO | Fetching in progress ....
|
|
2026-06-22 12:13:05 | INFO | Fetched 17 rows from SQL Server
|
|
2026-06-22 12:13:05 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!!
|
|
2026-06-22 12:13:05 | INFO | Fetched 17 rows
|
|
2026-06-22 12:13:05 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-22 12:13:05 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:13:05 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-22 12:13:05 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-22 12:13:05 | INFO | ================================================================================
|
|
2026-06-22 12:13:05 | INFO | Processing Table: Master_VisibilityDefinition | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
|
|
2026-06-22 12:13:05 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition ..............
|
|
2026-06-22 12:13:05 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 12:13:05 | INFO | Fetching in progress ....
|
|
2026-06-22 12:13:05 | INFO | Fetched 861 rows from SQL Server
|
|
2026-06-22 12:13:05 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!!
|
|
2026-06-22 12:13:05 | INFO | Fetched 861 rows
|
|
2026-06-22 12:13:05 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-22 12:13:05 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:13:06 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-22 12:13:06 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-22 12:13:06 | INFO | ================================================================================
|
|
2026-06-22 12:13:06 | INFO | Processing Table: Master_Salesterritorylayer | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
|
|
2026-06-22 12:13:06 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer ..............
|
|
2026-06-22 12:13:06 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 12:13:06 | INFO | Fetching in progress ....
|
|
2026-06-22 12:13:06 | INFO | Fetched 33 rows from SQL Server
|
|
2026-06-22 12:13:06 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!!
|
|
2026-06-22 12:13:06 | INFO | Fetched 33 rows
|
|
2026-06-22 12:13:06 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-22 12:13:06 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:13:06 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-22 12:13:06 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-22 12:13:06 | INFO | ================================================================================
|
|
2026-06-22 12:13:06 | INFO | Pipeline Completed Successfully
|
|
2026-06-22 12:13:06 | INFO | ================================================================================
|
|
2026-06-22 12:16:47 | INFO | ================================================================================
|
|
2026-06-22 12:16:47 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 12:16:47 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 12:16:47 | INFO | Connecting to databases...
|
|
2026-06-22 12:16:48 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002292C618730>
|
|
2026-06-22 12:16:50 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002292D6DC7D0>
|
|
2026-06-22 12:16:52 | INFO | Database connections established
|
|
2026-06-22 12:16:52 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 12:16:52 | INFO | Found 569 MIDs
|
|
2026-06-22 12:16:53 | INFO | ================================================================================
|
|
2026-06-22 12:16:53 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:16:53 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-22 12:16:53 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 12:16:53 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:16:53 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:16:57 | INFO | Fetched 1,907 rows from SQL Server
|
|
2026-06-22 12:16:57 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-22 12:16:57 | INFO | Fetched 1907 rows
|
|
2026-06-22 12:16:57 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-22 12:16:57 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:16:58 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse
|
|
2026-06-22 12:16:58 | INFO | SOS_OneApp loaded successfully (1907 rows)
|
|
2026-06-22 12:16:58 | INFO | ================================================================================
|
|
2026-06-22 12:16:58 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 12:16:58 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-22 12:16:58 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-22 12:17:14 | INFO | Fetched 291,134 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-22 12:17:14 | WARNING | Table 'OQaD' does not exist.
|
|
2026-06-22 12:17:14 | INFO | Fetched 0 matched empids fetched for OQAD
|
|
2026-06-22 12:17:14 | INFO | Exists: True
|
|
2026-06-22 12:17:14 | INFO | Path: D:\z\data_move\src\sql\fact\OQaD.sql
|
|
2026-06-22 12:17:14 | INFO | Fetching data for 0 EMPIDs
|
|
2026-06-22 12:17:14 | INFO | Fetching OQaD data for run_date=2026-06-21
|
|
2026-06-22 12:17:14 | ERROR | Failed processing table OQaD
|
|
Traceback (most recent call last):
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 1969, in _exec_single_context
|
|
self.dialect.do_execute(
|
|
~~~~~~~~~~~~~~~~~~~~~~~^
|
|
cursor, str_statement, effective_parameters, context
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\default.py", line 952, in do_execute
|
|
cursor.execute(statement, parameters)
|
|
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
|
|
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Incorrect syntax near ')'. (102) (SQLExecDirectW)")
|
|
|
|
The above exception was the direct cause of the following exception:
|
|
|
|
Traceback (most recent call last):
|
|
File "D:\z\data_move\main2.py", line 159, in main
|
|
df=fn(sql_engine, table_name , table_type, mids, run_date)
|
|
File "D:\z\data_move\src\fact.py", line 285, in fetch_OQaD
|
|
df=fetch_data( engine=sql_engine,
|
|
table_name=table_name,
|
|
...<2 lines>...
|
|
run_date=run_date
|
|
)
|
|
File "D:\z\data_move\src\fact.py", line 277, in fetch_data
|
|
df = pl.read_database(
|
|
query=sql,
|
|
connection=engine,
|
|
)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\polars\io\database\functions.py", line 284, in read_database
|
|
return cx.execute(
|
|
~~~~~~~~~~^
|
|
query=query,
|
|
^^^^^^^^^^^^
|
|
options=execute_options,
|
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
|
).to_polars(
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\polars\io\database\_executor.py", line 546, in execute
|
|
result = cursor_execute(query, **options)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 1421, in execute
|
|
return meth(
|
|
self,
|
|
distilled_parameters,
|
|
execution_options or NO_OPTIONS,
|
|
)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\sql\elements.py", line 526, in _execute_on_connection
|
|
return connection._execute_clauseelement(
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
|
|
self, distilled_params, execution_options
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 1643, in _execute_clauseelement
|
|
ret = self._execute_context(
|
|
dialect,
|
|
...<8 lines>...
|
|
cache_hit=cache_hit,
|
|
)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 1848, in _execute_context
|
|
return self._exec_single_context(
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~^
|
|
dialect, context, statement, parameters
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 1988, in _exec_single_context
|
|
self._handle_dbapi_exception(
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
|
|
e, str_statement, effective_parameters, cursor, context
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 2365, in _handle_dbapi_exception
|
|
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\base.py", line 1969, in _exec_single_context
|
|
self.dialect.do_execute(
|
|
~~~~~~~~~~~~~~~~~~~~~~~^
|
|
cursor, str_statement, effective_parameters, context
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-bb41b3d8f97c187a\Lib\site-packages\sqlalchemy\engine\default.py", line 952, in do_execute
|
|
cursor.execute(statement, parameters)
|
|
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
|
|
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Incorrect syntax near ')'. (102) (SQLExecDirectW)")
|
|
[SQL: WITH MID_TABLE_COV1 AS
|
|
(
|
|
SELECT EmpId, VisitDate
|
|
FROM OneApp_KelloggsMT.dbo.T_OQAD
|
|
WHERE CreateDate >= 2026-06-21
|
|
AND CreateDate < DATEADD(DAY,1,'2026-06-21')
|
|
|
|
UNION ALL
|
|
|
|
SELECT EmpId, VisitDate
|
|
FROM OneApp_KelloggsMT.dbo.T_OQAD
|
|
WHERE UpdateDate >= 2026-06-21
|
|
AND UpdateDate < DATEADD(DAY,1, '2026-06-21')
|
|
),
|
|
|
|
|
|
QUIZ AS
|
|
(
|
|
SELECT DISTINCT
|
|
E.EmpId,
|
|
E.EmpName,
|
|
E.SupervisorId,
|
|
E.SupervisorName,
|
|
E.DesignationName,
|
|
E.CityName,
|
|
E.StateName,
|
|
E.RegionName,
|
|
CAST(DQ.VisitDate AS DATE) AS VisitDate,
|
|
DQ.QuestionId,
|
|
DQ.AnswerId,
|
|
QC.QuestionCategoryId,
|
|
QC.QuestionCategory
|
|
FROM OneApp_KelloggsMT.dbo.T_OQAD DQ
|
|
INNER JOIN OneApp_KelloggsMT.dbo.vw_Employee_Detail E
|
|
ON DQ.EmpId = E.EmpId
|
|
INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Question QU
|
|
ON DQ.QuestionId = QU.QuestionId
|
|
INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Category QC
|
|
ON QU.QuestionCategoryId = QC.QuestionCategoryId
|
|
WHERE E.EmpName NOT LIKE '%TEST%'
|
|
AND E.RightId = 6
|
|
AND (
|
|
E.ResignDate IS NULL
|
|
OR CAST(E.ResignDate AS DATE) >= '2026-06-21'
|
|
)
|
|
AND EXISTS
|
|
(
|
|
SELECT 1
|
|
FROM MID_TABLE_COV1 A
|
|
WHERE A.EmpId = DQ.EmpId
|
|
AND CAST(A.VisitDate AS DATE) = CAST(DQ.VisitDate AS DATE)
|
|
)
|
|
)
|
|
|
|
SELECT
|
|
40148 AS project_id,
|
|
Q.EmpId AS employee_id,
|
|
0 AS process_id,
|
|
Q.VisitDate AS visit_date,
|
|
Q.QuestionCategoryId AS question_category_id,
|
|
Q.QuestionCategory AS question_category,
|
|
QM.QuestionId AS question_id,
|
|
QM.Question AS question,
|
|
ISNULL(QA.AnswerId,0) AS answer_id,
|
|
ISNULL(QA.Answer,'') AS answer,
|
|
CASE
|
|
WHEN QA.AnswerId IS NULL THEN 'Not Answer'
|
|
WHEN QA.RightAnswer = 1 THEN 'Y'
|
|
WHEN QA.RightAnswer IS NULL THEN 'Not Answer'
|
|
ELSE 'N'
|
|
END AS correct_answer,
|
|
GETDATE() AS update_date,
|
|
'SP-Pius' AS update_by
|
|
FROM QUIZ Q
|
|
INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Question QM
|
|
ON Q.QuestionId = QM.QuestionId
|
|
LEFT JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Answer QA
|
|
ON Q.AnswerId = QA.AnswerId
|
|
where Q.EmpId not in ()
|
|
|
|
|
|
]
|
|
(Background on this error at: https://sqlalche.me/e/20/f405)
|
|
2026-06-22 12:27:42 | INFO | ================================================================================
|
|
2026-06-22 12:27:42 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 12:27:42 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 12:27:42 | INFO | Connecting to databases...
|
|
2026-06-22 12:27:43 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000028171BB45D0>
|
|
2026-06-22 12:27:44 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000028172C50A50>
|
|
2026-06-22 12:27:44 | INFO | Database connections established
|
|
2026-06-22 12:27:44 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 12:27:45 | INFO | Found 569 MIDs
|
|
2026-06-22 12:27:45 | INFO | ================================================================================
|
|
2026-06-22 12:27:45 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:27:45 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-22 12:27:45 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 12:27:45 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:27:45 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:27:50 | INFO | Fetched 1,907 rows from SQL Server
|
|
2026-06-22 12:27:50 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-22 12:27:50 | INFO | Fetched 1907 rows
|
|
2026-06-22 12:27:50 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-22 12:27:51 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:27:52 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse
|
|
2026-06-22 12:27:52 | INFO | SOS_OneApp loaded successfully (1907 rows)
|
|
2026-06-22 12:27:52 | INFO | ================================================================================
|
|
2026-06-22 12:27:52 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 12:27:52 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-22 12:27:53 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-22 12:28:12 | INFO | Fetched 291,145 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-22 12:28:12 | WARNING | Table 'OQaD' does not exist.
|
|
2026-06-22 12:28:12 | WARNING | OQaD Matched df in OQaD returned no rows
|
|
2026-06-22 12:28:12 | ERROR | Failed processing table OQaD
|
|
Traceback (most recent call last):
|
|
File "D:\z\data_move\main2.py", line 159, in main
|
|
df=fn(sql_engine, table_name , table_type, mids, run_date)
|
|
File "D:\z\data_move\src\fact.py", line 257, in fetch_OQaD
|
|
log.info(f"Fetched {len(empids):,} matched empids fetched for OQAD ")
|
|
^^^^^^
|
|
UnboundLocalError: cannot access local variable 'empids' where it is not associated with a value
|
|
2026-06-22 12:28:18 | INFO | ================================================================================
|
|
2026-06-22 12:28:18 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 12:28:18 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 12:28:18 | INFO | Connecting to databases...
|
|
2026-06-22 12:28:19 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000015822E845D0>
|
|
2026-06-22 12:28:19 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000015823F20A50>
|
|
2026-06-22 12:28:20 | INFO | Database connections established
|
|
2026-06-22 12:28:20 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 12:28:20 | INFO | Found 569 MIDs
|
|
2026-06-22 12:28:20 | INFO | ================================================================================
|
|
2026-06-22 12:28:20 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:28:20 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-22 12:28:20 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 12:28:20 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:28:20 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:28:25 | INFO | Fetched 1,907 rows from SQL Server
|
|
2026-06-22 12:28:25 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-22 12:28:25 | INFO | Fetched 1907 rows
|
|
2026-06-22 12:28:25 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-22 12:28:25 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:28:25 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse
|
|
2026-06-22 12:28:25 | INFO | SOS_OneApp loaded successfully (1907 rows)
|
|
2026-06-22 12:28:25 | INFO | ================================================================================
|
|
2026-06-22 12:28:25 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 12:28:25 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-22 12:28:25 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-22 12:28:47 | INFO | Fetched 291,145 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-22 12:28:47 | WARNING | Table 'OQaD' does not exist.
|
|
2026-06-22 12:28:47 | ERROR | Failed processing table OQaD
|
|
Traceback (most recent call last):
|
|
File "D:\z\data_move\main2.py", line 159, in main
|
|
df=fn(sql_engine, table_name , table_type, mids, run_date)
|
|
File "D:\z\data_move\src\fact.py", line 248, in fetch_OQaD
|
|
log.warning(
|
|
~~~~~~~~~~~^
|
|
"%s Matched df in OQaD returned no rows",
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
empids=[0]
|
|
^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Roaming\uv\python\cpython-3.14-windows-x86_64-none\Lib\logging\__init__.py", line 1532, in warning
|
|
self._log(WARNING, msg, args, **kwargs)
|
|
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
TypeError: Logger._log() got an unexpected keyword argument 'empids'
|
|
2026-06-22 12:29:11 | INFO | ================================================================================
|
|
2026-06-22 12:29:11 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 12:29:11 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 12:29:11 | INFO | Connecting to databases...
|
|
2026-06-22 12:29:12 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000023B04EE45D0>
|
|
2026-06-22 12:29:13 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000023B05F80A50>
|
|
2026-06-22 12:29:13 | INFO | Database connections established
|
|
2026-06-22 12:29:13 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 12:29:14 | INFO | Found 569 MIDs
|
|
2026-06-22 12:29:14 | INFO | ================================================================================
|
|
2026-06-22 12:29:14 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:29:14 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-22 12:29:14 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 12:29:14 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:29:14 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:29:15 | INFO | Fetched 1,907 rows from SQL Server
|
|
2026-06-22 12:29:15 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-22 12:29:15 | INFO | Fetched 1907 rows
|
|
2026-06-22 12:29:15 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-22 12:29:15 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:29:15 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse
|
|
2026-06-22 12:29:15 | INFO | SOS_OneApp loaded successfully (1907 rows)
|
|
2026-06-22 12:29:15 | INFO | ================================================================================
|
|
2026-06-22 12:29:15 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 12:29:15 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-22 12:29:16 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-22 12:29:27 | INFO | Fetched 291,145 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-22 12:29:27 | WARNING | Table 'OQaD' does not exist.
|
|
2026-06-22 12:29:27 | ERROR | Failed processing table OQaD
|
|
Traceback (most recent call last):
|
|
File "D:\z\data_move\main2.py", line 159, in main
|
|
df=fn(sql_engine, table_name , table_type, mids, run_date)
|
|
File "D:\z\data_move\src\fact.py", line 248, in fetch_OQaD
|
|
log.warning(
|
|
~~~~~~~~~~~^
|
|
"%s Matched df in OQaD returned no rows",
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
empids=[0]
|
|
^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Roaming\uv\python\cpython-3.14-windows-x86_64-none\Lib\logging\__init__.py", line 1532, in warning
|
|
self._log(WARNING, msg, args, **kwargs)
|
|
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
TypeError: Logger._log() got an unexpected keyword argument 'empids'
|
|
2026-06-22 12:30:00 | INFO | ================================================================================
|
|
2026-06-22 12:30:00 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 12:30:00 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 12:30:00 | INFO | Connecting to databases...
|
|
2026-06-22 12:30:02 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001C9CBD745D0>
|
|
2026-06-22 12:30:03 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001C9CCDC0A50>
|
|
2026-06-22 12:30:03 | INFO | Database connections established
|
|
2026-06-22 12:30:03 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 12:30:04 | INFO | Found 569 MIDs
|
|
2026-06-22 12:30:04 | INFO | ================================================================================
|
|
2026-06-22 12:30:04 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:30:04 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-22 12:30:04 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 12:30:04 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:30:04 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:30:06 | INFO | Fetched 1,907 rows from SQL Server
|
|
2026-06-22 12:30:06 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-22 12:30:06 | INFO | Fetched 1907 rows
|
|
2026-06-22 12:30:06 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-22 12:30:06 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:30:06 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse
|
|
2026-06-22 12:30:06 | INFO | SOS_OneApp loaded successfully (1907 rows)
|
|
2026-06-22 12:30:06 | INFO | ================================================================================
|
|
2026-06-22 12:30:06 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 12:30:06 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-22 12:30:07 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-22 12:30:24 | INFO | Fetched 291,147 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-22 12:30:24 | WARNING | Table 'OQaD' does not exist.
|
|
2026-06-22 12:30:24 | WARNING | OQaD Matched df in OQaD returned no rows
|
|
2026-06-22 12:30:24 | INFO | Fetched 1 matched empids fetched for OQAD
|
|
2026-06-22 12:30:24 | INFO | Exists: True
|
|
2026-06-22 12:30:24 | INFO | Path: D:\z\data_move\src\sql\fact\OQaD.sql
|
|
2026-06-22 12:30:24 | INFO | Fetching data for 1 EMPIDs
|
|
2026-06-22 12:30:24 | INFO | Fetching OQaD data for run_date=2026-06-21
|
|
2026-06-22 12:32:08 | INFO | fn name is fetch_OQad ------Fetched 290730 rows
|
|
2026-06-22 12:32:08 | INFO | Fetched 290,730 rows from SQL Server
|
|
2026-06-22 12:32:08 | INFO | Fetched total row -: 290730 from sql server for table-:OQaD ...........!!!
|
|
2026-06-22 12:32:08 | INFO | Fetched 290730 rows
|
|
2026-06-22 12:32:08 | INFO | Creating table OQaD
|
|
2026-06-22 12:32:08 | INFO | Table ready: OQaD
|
|
2026-06-22 12:32:08 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:32:12 | INFO | OQaD: inserted 290,730 rows into ClickHouse
|
|
2026-06-22 12:32:12 | INFO | OQaD loaded successfully (290730 rows)
|
|
2026-06-22 12:32:12 | INFO | ================================================================================
|
|
2026-06-22 12:32:12 | INFO | Processing Table: Survey | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:32:12 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-22 12:32:12 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:32:12 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:32:13 | INFO | Fetched 165 rows from SQL Server
|
|
2026-06-22 12:32:13 | INFO | Fetched total row -: 165 from sql server for table-:Survey ...........!!!
|
|
2026-06-22 12:32:13 | INFO | Fetched 165 rows
|
|
2026-06-22 12:32:13 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
|
|
2026-06-22 12:32:13 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:32:13 | INFO | Survey: inserted 165 rows into ClickHouse
|
|
2026-06-22 12:32:13 | INFO | Survey loaded successfully (165 rows)
|
|
2026-06-22 12:32:13 | INFO | ================================================================================
|
|
2026-06-22 12:32:13 | INFO | Processing Table: additional_visibility | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:32:13 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-22 12:32:13 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 12:32:13 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:32:13 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:32:14 | INFO | Fetched 1,579 rows from SQL Server
|
|
2026-06-22 12:32:14 | INFO | Fetched total row -: 1579 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-22 12:32:14 | INFO | Fetched 1579 rows
|
|
2026-06-22 12:32:14 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
|
|
2026-06-22 12:32:14 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:32:14 | INFO | additional_visibility: inserted 1,579 rows into ClickHouse
|
|
2026-06-22 12:32:14 | INFO | additional_visibility loaded successfully (1579 rows)
|
|
2026-06-22 12:32:14 | INFO | ================================================================================
|
|
2026-06-22 12:32:14 | INFO | Processing Table: Coverage | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:32:14 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-22 12:32:14 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:32:14 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:32:33 | INFO | Fetched 512 rows from SQL Server
|
|
2026-06-22 12:32:33 | INFO | Fetched total row -: 512 from sql server for table-:Coverage ...........!!!
|
|
2026-06-22 12:32:33 | INFO | Fetched 512 rows
|
|
2026-06-22 12:32:33 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
|
|
2026-06-22 12:32:33 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:32:33 | INFO | Coverage: inserted 512 rows into ClickHouse
|
|
2026-06-22 12:32:33 | INFO | Coverage loaded successfully (512 rows)
|
|
2026-06-22 12:32:33 | INFO | ================================================================================
|
|
2026-06-22 12:32:33 | INFO | Processing Table: Login | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 12:32:33 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-22 12:32:33 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:32:33 | ERROR | Failed processing table Login
|
|
Traceback (most recent call last):
|
|
File "D:\z\data_move\main2.py", line 159, in main
|
|
df=fn(sql_engine, table_name , table_type, mids, run_date)
|
|
File "D:\z\data_move\src\fact.py", line 443, in fetch_Login
|
|
sql = sql_template.format(
|
|
mid_list=mid_list,
|
|
run_date=run_date.strftime("%Y-%m-%d")
|
|
)
|
|
KeyError: 'start_date'
|
|
2026-06-22 12:43:20 | INFO | ================================================================================
|
|
2026-06-22 12:43:20 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 12:43:20 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 12:43:20 | INFO | Connecting to databases...
|
|
2026-06-22 12:43:21 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001F8F72945D0>
|
|
2026-06-22 12:43:22 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001F8F8370A50>
|
|
2026-06-22 12:43:23 | INFO | Database connections established
|
|
2026-06-22 12:43:23 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 12:43:23 | INFO | Found 569 MIDs
|
|
2026-06-22 12:43:23 | INFO | ================================================================================
|
|
2026-06-22 12:43:23 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:43:23 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-22 12:43:23 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 12:43:23 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:43:23 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:43:25 | INFO | Fetched 1,907 rows from SQL Server
|
|
2026-06-22 12:43:25 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-22 12:43:25 | INFO | Fetched 1907 rows
|
|
2026-06-22 12:43:25 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-22 12:43:25 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:43:26 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse
|
|
2026-06-22 12:43:26 | INFO | SOS_OneApp loaded successfully (1907 rows)
|
|
2026-06-22 12:43:26 | INFO | ================================================================================
|
|
2026-06-22 12:43:26 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 12:43:26 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-22 12:43:26 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-22 12:43:57 | INFO | Fetched 291,158 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-22 12:43:59 | INFO | Fetched 290,722 matched empids fetched for OQAD
|
|
2026-06-22 12:43:59 | INFO | Exists: True
|
|
2026-06-22 12:43:59 | INFO | Path: D:\z\data_move\src\sql\fact\OQaD.sql
|
|
2026-06-22 12:43:59 | INFO | Fetching data for 290,722 EMPIDs
|
|
2026-06-22 12:43:59 | INFO | Fetching OQaD data for run_date=2026-06-21
|
|
2026-06-22 12:44:40 | INFO | fn name is fetch_OQad ------Fetched 0 rows
|
|
2026-06-22 12:44:40 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 12:44:40 | INFO | Fetched total row -: 0 from sql server for table-:OQaD ...........!!!
|
|
2026-06-22 12:44:40 | WARNING | OQaD returned no rows
|
|
2026-06-22 12:44:40 | INFO | ================================================================================
|
|
2026-06-22 12:44:40 | INFO | Processing Table: Survey | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:44:40 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-22 12:44:40 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:44:40 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:44:40 | INFO | Fetched 165 rows from SQL Server
|
|
2026-06-22 12:44:40 | INFO | Fetched total row -: 165 from sql server for table-:Survey ...........!!!
|
|
2026-06-22 12:44:40 | INFO | Fetched 165 rows
|
|
2026-06-22 12:44:40 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
|
|
2026-06-22 12:44:40 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:44:41 | INFO | Survey: inserted 165 rows into ClickHouse
|
|
2026-06-22 12:44:41 | INFO | Survey loaded successfully (165 rows)
|
|
2026-06-22 12:44:41 | INFO | ================================================================================
|
|
2026-06-22 12:44:41 | INFO | Processing Table: additional_visibility | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:44:41 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-22 12:44:41 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 12:44:41 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:44:41 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:44:41 | INFO | Fetched 1,579 rows from SQL Server
|
|
2026-06-22 12:44:41 | INFO | Fetched total row -: 1579 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-22 12:44:41 | INFO | Fetched 1579 rows
|
|
2026-06-22 12:44:41 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
|
|
2026-06-22 12:44:41 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:44:42 | INFO | additional_visibility: inserted 1,579 rows into ClickHouse
|
|
2026-06-22 12:44:42 | INFO | additional_visibility loaded successfully (1579 rows)
|
|
2026-06-22 12:44:42 | INFO | ================================================================================
|
|
2026-06-22 12:44:42 | INFO | Processing Table: Coverage | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:44:42 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-22 12:44:42 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:44:42 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:44:58 | INFO | Fetched 512 rows from SQL Server
|
|
2026-06-22 12:44:58 | INFO | Fetched total row -: 512 from sql server for table-:Coverage ...........!!!
|
|
2026-06-22 12:44:58 | INFO | Fetched 512 rows
|
|
2026-06-22 12:44:58 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
|
|
2026-06-22 12:44:58 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:44:59 | INFO | Coverage: inserted 512 rows into ClickHouse
|
|
2026-06-22 12:44:59 | INFO | Coverage loaded successfully (512 rows)
|
|
2026-06-22 12:44:59 | INFO | ================================================================================
|
|
2026-06-22 12:44:59 | INFO | Processing Table: Login | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 12:44:59 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-22 12:44:59 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:44:59 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:45:02 | INFO | Fetched 454 rows from SQL Server
|
|
2026-06-22 12:45:02 | INFO | Fetched total row -: 454 from sql server for table-:Login ...........!!!
|
|
2026-06-22 12:45:02 | INFO | Fetched 454 rows
|
|
2026-06-22 12:45:02 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _
|
|
2026-06-22 12:45:02 | INFO | No delete logic required for Login
|
|
2026-06-22 12:45:02 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:45:03 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-22 12:45:03 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-22 12:45:03 | INFO | ================================================================================
|
|
2026-06-22 12:45:03 | INFO | Processing Table: Stock_Details | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:45:03 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-22 12:45:03 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:45:03 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:45:19 | INFO | Fetched 27,359 rows from SQL Server
|
|
2026-06-22 12:45:19 | INFO | Fetched total row -: 27359 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-22 12:45:19 | INFO | Fetched 27359 rows
|
|
2026-06-22 12:45:19 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _
|
|
2026-06-22 12:45:19 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:45:19 | INFO | Stock_Details: inserted 27,359 rows into ClickHouse
|
|
2026-06-22 12:45:19 | INFO | Stock_Details loaded successfully (27359 rows)
|
|
2026-06-22 12:45:19 | INFO | ================================================================================
|
|
2026-06-22 12:45:19 | INFO | Processing Table: Attendance | Table type is -: FACT | Based on -run_date and operation is used -DELETE+INSERT
|
|
2026-06-22 12:45:19 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-22 12:45:19 | INFO | Fetching Attendance data from 2026-06-06 to 2026-06-21
|
|
2026-06-22 12:47:52 | INFO | Fetched 121,639 attendance rows for 544 employees
|
|
2026-06-22 12:47:52 | INFO | Fetched total row -: 121639 from sql server for table-:Attendance ...........!!!
|
|
2026-06-22 12:47:52 | INFO | Fetched 121639 rows
|
|
2026-06-22 12:47:52 | INFO | Truncated table Attendance
|
|
2026-06-22 12:47:52 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:47:54 | INFO | Attendance: inserted 121,639 rows into ClickHouse
|
|
2026-06-22 12:47:54 | INFO | Attendance loaded successfully (121639 rows)
|
|
2026-06-22 12:47:54 | INFO | ================================================================================
|
|
2026-06-22 12:47:54 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 12:47:54 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-22 12:47:54 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 12:47:54 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:47:54 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:48:07 | INFO | Fetched 21,621 rows from SQL Server
|
|
2026-06-22 12:48:07 | INFO | Fetched total row -: 21621 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-22 12:48:07 | INFO | Fetched 21621 rows
|
|
2026-06-22 12:48:07 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
|
|
2026-06-22 12:48:07 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:48:08 | INFO | Journey_Plan: inserted 21,621 rows into ClickHouse
|
|
2026-06-22 12:48:08 | INFO | Journey_Plan loaded successfully (21621 rows)
|
|
2026-06-22 12:48:08 | INFO | ================================================================================
|
|
2026-06-22 12:48:08 | INFO | Processing Table: Web_Logins | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 12:48:08 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-22 12:48:08 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 12:48:08 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 12:48:08 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 12:48:08 | INFO | Fetched 12 rows from SQL Server
|
|
2026-06-22 12:48:08 | INFO | Fetched total row -: 12 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-22 12:48:08 | INFO | Fetched 12 rows
|
|
2026-06-22 12:48:08 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
|
|
2026-06-22 12:48:09 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 12:48:09 | INFO | Web_Logins: inserted 12 rows into ClickHouse
|
|
2026-06-22 12:48:09 | INFO | Web_Logins loaded successfully (12 rows)
|
|
2026-06-22 12:48:09 | INFO | ================================================================================
|
|
2026-06-22 12:48:09 | INFO | Processing Table: Promotion | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 12:48:09 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-22 12:48:09 | ERROR | Failed processing table Promotion
|
|
Traceback (most recent call last):
|
|
File "D:\z\data_move\main2.py", line 158, in main
|
|
fn = globals()[fn_name]
|
|
~~~~~~~~~^^^^^^^^^
|
|
KeyError: 'fetch_Promotion'
|
|
2026-06-22 13:04:30 | INFO | ================================================================================
|
|
2026-06-22 13:04:30 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 13:04:30 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 13:04:30 | INFO | Connecting to databases...
|
|
2026-06-22 13:04:30 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000248650E0730>
|
|
2026-06-22 13:04:31 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000024866190A50>
|
|
2026-06-22 13:04:32 | INFO | Database connections established
|
|
2026-06-22 13:04:32 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 13:04:32 | INFO | Found 569 MIDs
|
|
2026-06-22 13:04:32 | INFO | ================================================================================
|
|
2026-06-22 13:04:32 | INFO | Processing Table: Promotion | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:04:32 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-22 13:04:32 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:04:36 | INFO | Fetched 4,223 rows from SQL Server
|
|
2026-06-22 13:04:36 | INFO | Fetched total row -: 4223 from sql server for table-:Promotion ...........!!!
|
|
2026-06-22 13:04:36 | INFO | Fetched 4223 rows
|
|
2026-06-22 13:04:36 | INFO | Creating table Promotion
|
|
2026-06-22 13:04:36 | INFO | Table ready: Promotion
|
|
2026-06-22 13:04:36 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:04:36 | INFO | Promotion: inserted 4,223 rows into ClickHouse
|
|
2026-06-22 13:04:36 | INFO | Promotion loaded successfully (4223 rows)
|
|
2026-06-22 13:04:36 | INFO | ================================================================================
|
|
2026-06-22 13:04:36 | INFO | Pipeline Completed Successfully
|
|
2026-06-22 13:04:36 | INFO | ================================================================================
|
|
2026-06-22 13:04:41 | INFO | ================================================================================
|
|
2026-06-22 13:04:41 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 13:04:41 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 13:04:41 | INFO | Connecting to databases...
|
|
2026-06-22 13:04:42 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002BA5BE88890>
|
|
2026-06-22 13:04:43 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002BA5CEAC7D0>
|
|
2026-06-22 13:04:44 | INFO | Database connections established
|
|
2026-06-22 13:04:44 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 13:04:44 | INFO | Found 569 MIDs
|
|
2026-06-22 13:04:44 | INFO | ================================================================================
|
|
2026-06-22 13:04:44 | INFO | Processing Table: Promotion | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:04:44 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-22 13:04:44 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:04:54 | INFO | Fetched 4,223 rows from SQL Server
|
|
2026-06-22 13:04:54 | INFO | Fetched total row -: 4223 from sql server for table-:Promotion ...........!!!
|
|
2026-06-22 13:04:54 | INFO | Fetched 4223 rows
|
|
2026-06-22 13:04:54 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-22 13:04:54 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:04:55 | INFO | Promotion: inserted 4,223 rows into ClickHouse
|
|
2026-06-22 13:04:55 | INFO | Promotion loaded successfully (4223 rows)
|
|
2026-06-22 13:04:55 | INFO | ================================================================================
|
|
2026-06-22 13:04:55 | INFO | Pipeline Completed Successfully
|
|
2026-06-22 13:04:55 | INFO | ================================================================================
|
|
2026-06-22 13:05:14 | INFO | ================================================================================
|
|
2026-06-22 13:05:14 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 13:05:14 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 13:05:14 | INFO | Connecting to databases...
|
|
2026-06-22 13:05:15 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D8C9D98890>
|
|
2026-06-22 13:05:16 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D8CAE5C7D0>
|
|
2026-06-22 13:05:17 | INFO | Database connections established
|
|
2026-06-22 13:05:17 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 13:05:17 | INFO | Found 569 MIDs
|
|
2026-06-22 13:05:17 | INFO | ================================================================================
|
|
2026-06-22 13:05:17 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:05:17 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-22 13:05:17 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 13:05:17 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:05:17 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:05:19 | INFO | Fetched 1,907 rows from SQL Server
|
|
2026-06-22 13:05:19 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-22 13:05:19 | INFO | Fetched 1907 rows
|
|
2026-06-22 13:05:19 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-22 13:05:20 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:05:20 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse
|
|
2026-06-22 13:05:20 | INFO | SOS_OneApp loaded successfully (1907 rows)
|
|
2026-06-22 13:05:20 | INFO | ================================================================================
|
|
2026-06-22 13:05:20 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 13:05:20 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-22 13:05:20 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-22 13:05:32 | INFO | Fetched 291,165 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-22 13:05:34 | INFO | Fetched 290,722 matched empids fetched for OQAD
|
|
2026-06-22 13:05:34 | INFO | Exists: True
|
|
2026-06-22 13:05:34 | INFO | Path: D:\z\data_move\src\sql\fact\OQaD.sql
|
|
2026-06-22 13:05:34 | INFO | Fetching data for 290,722 EMPIDs
|
|
2026-06-22 13:05:34 | INFO | Fetching OQaD data for run_date=2026-06-21
|
|
2026-06-22 13:06:10 | INFO | fn name is fetch_OQad ------Fetched 0 rows
|
|
2026-06-22 13:06:10 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 13:06:10 | INFO | Fetched total row -: 0 from sql server for table-:OQaD ...........!!!
|
|
2026-06-22 13:06:10 | WARNING | OQaD returned no rows
|
|
2026-06-22 13:06:10 | INFO | ================================================================================
|
|
2026-06-22 13:06:10 | INFO | Processing Table: Survey | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:06:10 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-22 13:06:10 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:06:10 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:06:10 | INFO | Fetched 165 rows from SQL Server
|
|
2026-06-22 13:06:10 | INFO | Fetched total row -: 165 from sql server for table-:Survey ...........!!!
|
|
2026-06-22 13:06:10 | INFO | Fetched 165 rows
|
|
2026-06-22 13:06:11 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
|
|
2026-06-22 13:06:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:06:11 | INFO | Survey: inserted 165 rows into ClickHouse
|
|
2026-06-22 13:06:11 | INFO | Survey loaded successfully (165 rows)
|
|
2026-06-22 13:06:11 | INFO | ================================================================================
|
|
2026-06-22 13:06:11 | INFO | Processing Table: additional_visibility | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:06:11 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-22 13:06:11 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 13:06:11 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:06:11 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:06:11 | INFO | Fetched 1,579 rows from SQL Server
|
|
2026-06-22 13:06:11 | INFO | Fetched total row -: 1579 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-22 13:06:11 | INFO | Fetched 1579 rows
|
|
2026-06-22 13:06:11 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
|
|
2026-06-22 13:06:12 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:06:12 | INFO | additional_visibility: inserted 1,579 rows into ClickHouse
|
|
2026-06-22 13:06:12 | INFO | additional_visibility loaded successfully (1579 rows)
|
|
2026-06-22 13:06:12 | INFO | ================================================================================
|
|
2026-06-22 13:06:12 | INFO | Processing Table: Coverage | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:06:12 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-22 13:06:12 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:06:12 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:06:27 | INFO | Fetched 512 rows from SQL Server
|
|
2026-06-22 13:06:27 | INFO | Fetched total row -: 512 from sql server for table-:Coverage ...........!!!
|
|
2026-06-22 13:06:27 | INFO | Fetched 512 rows
|
|
2026-06-22 13:06:27 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
|
|
2026-06-22 13:06:27 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:06:28 | INFO | Coverage: inserted 512 rows into ClickHouse
|
|
2026-06-22 13:06:28 | INFO | Coverage loaded successfully (512 rows)
|
|
2026-06-22 13:06:28 | INFO | ================================================================================
|
|
2026-06-22 13:06:28 | INFO | Processing Table: Login | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 13:06:28 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-22 13:06:28 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:06:28 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:06:29 | INFO | Fetched 454 rows from SQL Server
|
|
2026-06-22 13:06:29 | INFO | Fetched total row -: 454 from sql server for table-:Login ...........!!!
|
|
2026-06-22 13:06:29 | INFO | Fetched 454 rows
|
|
2026-06-22 13:06:29 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _
|
|
2026-06-22 13:06:29 | INFO | No delete logic required for Login
|
|
2026-06-22 13:06:29 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:06:30 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-22 13:06:30 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-22 13:06:30 | INFO | ================================================================================
|
|
2026-06-22 13:06:30 | INFO | Processing Table: Stock_Details | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:06:30 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-22 13:06:30 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:06:30 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:06:35 | INFO | Fetched 27,359 rows from SQL Server
|
|
2026-06-22 13:06:35 | INFO | Fetched total row -: 27359 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-22 13:06:35 | INFO | Fetched 27359 rows
|
|
2026-06-22 13:06:35 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _
|
|
2026-06-22 13:06:35 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:06:35 | INFO | Stock_Details: inserted 27,359 rows into ClickHouse
|
|
2026-06-22 13:06:35 | INFO | Stock_Details loaded successfully (27359 rows)
|
|
2026-06-22 13:06:35 | INFO | ================================================================================
|
|
2026-06-22 13:06:35 | INFO | Processing Table: Attendance | Table type is -: FACT | Based on -run_date and operation is used -DELETE+INSERT
|
|
2026-06-22 13:06:35 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-22 13:06:35 | INFO | Fetching Attendance data from 2026-06-06 to 2026-06-21
|
|
2026-06-22 13:06:51 | INFO | Fetched 121,639 attendance rows for 544 employees
|
|
2026-06-22 13:06:51 | INFO | Fetched total row -: 121639 from sql server for table-:Attendance ...........!!!
|
|
2026-06-22 13:06:51 | INFO | Fetched 121639 rows
|
|
2026-06-22 13:06:51 | INFO | Truncated table Attendance
|
|
2026-06-22 13:06:51 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:06:56 | INFO | Attendance: inserted 121,639 rows into ClickHouse
|
|
2026-06-22 13:06:56 | INFO | Attendance loaded successfully (121639 rows)
|
|
2026-06-22 13:06:56 | INFO | ================================================================================
|
|
2026-06-22 13:06:56 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 13:06:56 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-22 13:06:56 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 13:06:56 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:06:56 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:06:57 | INFO | Fetched 21,625 rows from SQL Server
|
|
2026-06-22 13:06:57 | INFO | Fetched total row -: 21625 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-22 13:06:57 | INFO | Fetched 21625 rows
|
|
2026-06-22 13:06:57 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
|
|
2026-06-22 13:06:57 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:06:57 | INFO | Journey_Plan: inserted 21,625 rows into ClickHouse
|
|
2026-06-22 13:06:57 | INFO | Journey_Plan loaded successfully (21625 rows)
|
|
2026-06-22 13:06:57 | INFO | ================================================================================
|
|
2026-06-22 13:06:57 | INFO | Processing Table: Web_Logins | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 13:06:57 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-22 13:06:57 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 13:06:57 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:06:57 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:06:58 | INFO | Fetched 12 rows from SQL Server
|
|
2026-06-22 13:06:58 | INFO | Fetched total row -: 12 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-22 13:06:58 | INFO | Fetched 12 rows
|
|
2026-06-22 13:06:58 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
|
|
2026-06-22 13:06:58 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:06:58 | INFO | Web_Logins: inserted 12 rows into ClickHouse
|
|
2026-06-22 13:06:58 | INFO | Web_Logins loaded successfully (12 rows)
|
|
2026-06-22 13:06:58 | INFO | ================================================================================
|
|
2026-06-22 13:06:58 | INFO | Processing Table: Promotion | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:06:58 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-22 13:06:58 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:07:00 | INFO | Fetched 4,223 rows from SQL Server
|
|
2026-06-22 13:07:00 | INFO | Fetched total row -: 4223 from sql server for table-:Promotion ...........!!!
|
|
2026-06-22 13:07:00 | INFO | Fetched 4223 rows
|
|
2026-06-22 13:07:00 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-22 13:07:01 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:07:01 | INFO | Promotion: inserted 4,223 rows into ClickHouse
|
|
2026-06-22 13:07:01 | INFO | Promotion loaded successfully (4223 rows)
|
|
2026-06-22 13:07:01 | INFO | ================================================================================
|
|
2026-06-22 13:07:01 | INFO | Processing Table: PaidVisibility | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:07:01 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
|
|
2026-06-22 13:07:01 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 13:07:01 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:07:01 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:07:02 | INFO | Fetched 1,284 rows from SQL Server
|
|
2026-06-22 13:07:02 | INFO | Fetched total row -: 1284 from sql server for table-:PaidVisibility ...........!!!
|
|
2026-06-22 13:07:02 | INFO | Fetched 1284 rows
|
|
2026-06-22 13:07:02 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _
|
|
2026-06-22 13:07:02 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:07:03 | INFO | PaidVisibility: inserted 1,284 rows into ClickHouse
|
|
2026-06-22 13:07:03 | INFO | PaidVisibility loaded successfully (1284 rows)
|
|
2026-06-22 13:07:03 | INFO | ================================================================================
|
|
2026-06-22 13:07:03 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
|
|
2026-06-22 13:07:03 | INFO | Fetching Data from sql server for table-: Store_Master ..............
|
|
2026-06-22 13:07:03 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:07:03 | INFO | Fetching in progress ....
|
|
2026-06-22 13:07:05 | INFO | Fetched 5,998 rows from SQL Server
|
|
2026-06-22 13:07:05 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
|
|
2026-06-22 13:07:05 | INFO | Fetched 5998 rows
|
|
2026-06-22 13:07:05 | INFO | Truncated table Store_Master
|
|
2026-06-22 13:07:05 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:07:07 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
|
|
2026-06-22 13:07:07 | INFO | Store_Master loaded successfully (5998 rows)
|
|
2026-06-22 13:07:07 | INFO | ================================================================================
|
|
2026-06-22 13:07:07 | INFO | Processing Table: SKU_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
|
|
2026-06-22 13:07:07 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
|
|
2026-06-22 13:07:07 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:07:07 | INFO | Fetching in progress ....
|
|
2026-06-22 13:07:07 | INFO | Fetched 160 rows from SQL Server
|
|
2026-06-22 13:07:07 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
|
|
2026-06-22 13:07:07 | INFO | Fetched 160 rows
|
|
2026-06-22 13:07:07 | INFO | Truncated table SKU_Master
|
|
2026-06-22 13:07:07 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:07:07 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-22 13:07:07 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-22 13:07:07 | INFO | ================================================================================
|
|
2026-06-22 13:07:07 | INFO | Processing Table: display_master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
|
|
2026-06-22 13:07:07 | INFO | Fetching Data from sql server for table-: display_master ..............
|
|
2026-06-22 13:07:07 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:07:07 | INFO | Fetching in progress ....
|
|
2026-06-22 13:07:07 | INFO | Fetched 135 rows from SQL Server
|
|
2026-06-22 13:07:07 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
|
|
2026-06-22 13:07:07 | INFO | Fetched 135 rows
|
|
2026-06-22 13:07:07 | INFO | Truncated table display_master
|
|
2026-06-22 13:07:07 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:07:07 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-22 13:07:07 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-22 13:07:07 | INFO | ================================================================================
|
|
2026-06-22 13:07:07 | INFO | Processing Table: Employee_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
|
|
2026-06-22 13:07:07 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
|
|
2026-06-22 13:07:07 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:07:07 | INFO | Fetching in progress ....
|
|
2026-06-22 13:07:08 | INFO | Fetched 2,288 rows from SQL Server
|
|
2026-06-22 13:07:08 | INFO | Fetched total row -: 2288 from sql server for table-:Employee_Master ...........!!!
|
|
2026-06-22 13:07:08 | INFO | Fetched 2288 rows
|
|
2026-06-22 13:07:08 | INFO | Truncated table Employee_Master
|
|
2026-06-22 13:07:08 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:07:08 | INFO | Employee_Master: inserted 2,288 rows into ClickHouse
|
|
2026-06-22 13:07:08 | INFO | Employee_Master loaded successfully (2288 rows)
|
|
2026-06-22 13:07:08 | INFO | ================================================================================
|
|
2026-06-22 13:07:08 | INFO | Processing Table: coverage_remarks | Table type is -: DIMENSION | Based on -reason_id and operation is used -DELETE+INSERT
|
|
2026-06-22 13:07:08 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
|
|
2026-06-22 13:07:09 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:07:09 | INFO | Fetching in progress ....
|
|
2026-06-22 13:07:09 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 13:07:09 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 13:07:09 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
|
|
2026-06-22 13:07:09 | WARNING | coverage_remarks returned no rows
|
|
2026-06-22 13:07:09 | INFO | ================================================================================
|
|
2026-06-22 13:07:09 | INFO | Processing Table: mapping_storevisibility | Table type is -: BRIDGE | Based on -run_date and operation is used -ONLY_INSERT
|
|
2026-06-22 13:07:09 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
|
|
2026-06-22 13:07:10 | WARNING | Table 'mapping_storevisibility' does not exist. During collecting store_ids
|
|
2026-06-22 13:07:10 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:07:10 | INFO | Fetching in progress ....
|
|
2026-06-22 13:07:10 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 13:07:10 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 13:07:10 | INFO | Fetched total row -: 0 from sql server for table-:mapping_storevisibility ...........!!!
|
|
2026-06-22 13:07:10 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-22 13:07:10 | INFO | ================================================================================
|
|
2026-06-22 13:07:10 | INFO | Processing Table: Master_VisibilityReason | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
|
|
2026-06-22 13:07:10 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason ..............
|
|
2026-06-22 13:07:10 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:07:10 | INFO | Fetching in progress ....
|
|
2026-06-22 13:07:10 | INFO | Fetched 17 rows from SQL Server
|
|
2026-06-22 13:07:10 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!!
|
|
2026-06-22 13:07:10 | INFO | Fetched 17 rows
|
|
2026-06-22 13:07:11 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-22 13:07:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:07:11 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-22 13:07:11 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-22 13:07:11 | INFO | ================================================================================
|
|
2026-06-22 13:07:11 | INFO | Processing Table: Master_VisibilityDefinition | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
|
|
2026-06-22 13:07:11 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition ..............
|
|
2026-06-22 13:07:11 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:07:11 | INFO | Fetching in progress ....
|
|
2026-06-22 13:07:11 | INFO | Fetched 861 rows from SQL Server
|
|
2026-06-22 13:07:11 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!!
|
|
2026-06-22 13:07:11 | INFO | Fetched 861 rows
|
|
2026-06-22 13:07:11 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-22 13:07:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:07:11 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-22 13:07:11 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-22 13:07:11 | INFO | ================================================================================
|
|
2026-06-22 13:07:11 | INFO | Processing Table: Master_Salesterritorylayer | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
|
|
2026-06-22 13:07:11 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer ..............
|
|
2026-06-22 13:07:11 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:07:11 | INFO | Fetching in progress ....
|
|
2026-06-22 13:07:11 | INFO | Fetched 33 rows from SQL Server
|
|
2026-06-22 13:07:11 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!!
|
|
2026-06-22 13:07:11 | INFO | Fetched 33 rows
|
|
2026-06-22 13:07:11 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-22 13:07:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:07:12 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-22 13:07:12 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-22 13:07:12 | INFO | ================================================================================
|
|
2026-06-22 13:07:12 | INFO | Pipeline Completed Successfully
|
|
2026-06-22 13:07:12 | INFO | ================================================================================
|
|
2026-06-22 13:15:45 | INFO | ================================================================================
|
|
2026-06-22 13:15:45 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 13:15:45 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 13:15:45 | INFO | Connecting to databases...
|
|
2026-06-22 13:15:46 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000015AA76E09F0>
|
|
2026-06-22 13:15:47 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000015AA879C7D0>
|
|
2026-06-22 13:15:48 | INFO | Database connections established
|
|
2026-06-22 13:15:48 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 13:15:49 | INFO | Found 569 MIDs
|
|
2026-06-22 13:15:49 | INFO | ================================================================================
|
|
2026-06-22 13:15:49 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:15:49 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-22 13:15:49 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 13:15:49 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:15:49 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:15:51 | INFO | Fetched 1,907 rows from SQL Server
|
|
2026-06-22 13:15:51 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-22 13:15:51 | INFO | Fetched 1907 rows
|
|
2026-06-22 13:15:51 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-22 13:15:51 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:15:52 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse
|
|
2026-06-22 13:15:52 | INFO | SOS_OneApp loaded successfully (1907 rows)
|
|
2026-06-22 13:15:52 | INFO | ================================================================================
|
|
2026-06-22 13:15:52 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 13:15:52 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-22 13:15:52 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-22 13:16:09 | INFO | Fetched 291,166 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-22 13:16:10 | INFO | Fetched 290,722 matched empids fetched for OQAD
|
|
2026-06-22 13:16:10 | INFO | Exists: True
|
|
2026-06-22 13:16:10 | INFO | Path: D:\z\data_move\src\sql\fact\OQaD.sql
|
|
2026-06-22 13:16:10 | INFO | Fetching data for 290,722 EMPIDs
|
|
2026-06-22 13:16:10 | INFO | Fetching OQaD data for run_date=2026-06-21
|
|
2026-06-22 13:16:46 | INFO | fn name is fetch_OQad ------Fetched 0 rows
|
|
2026-06-22 13:16:46 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 13:16:46 | INFO | Fetched total row -: 0 from sql server for table-:OQaD ...........!!!
|
|
2026-06-22 13:16:46 | WARNING | OQaD returned no rows
|
|
2026-06-22 13:16:46 | INFO | ================================================================================
|
|
2026-06-22 13:16:46 | INFO | Processing Table: Survey | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:16:46 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-22 13:16:46 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:16:46 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:16:47 | INFO | Fetched 165 rows from SQL Server
|
|
2026-06-22 13:16:47 | INFO | Fetched total row -: 165 from sql server for table-:Survey ...........!!!
|
|
2026-06-22 13:16:47 | INFO | Fetched 165 rows
|
|
2026-06-22 13:16:47 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
|
|
2026-06-22 13:16:47 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:16:47 | INFO | Survey: inserted 165 rows into ClickHouse
|
|
2026-06-22 13:16:47 | INFO | Survey loaded successfully (165 rows)
|
|
2026-06-22 13:16:47 | INFO | ================================================================================
|
|
2026-06-22 13:16:47 | INFO | Processing Table: additional_visibility | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:16:47 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-22 13:16:47 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 13:16:47 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:16:47 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:16:47 | INFO | Fetched 1,579 rows from SQL Server
|
|
2026-06-22 13:16:47 | INFO | Fetched total row -: 1579 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-22 13:16:47 | INFO | Fetched 1579 rows
|
|
2026-06-22 13:16:48 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
|
|
2026-06-22 13:16:48 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:16:48 | INFO | additional_visibility: inserted 1,579 rows into ClickHouse
|
|
2026-06-22 13:16:48 | INFO | additional_visibility loaded successfully (1579 rows)
|
|
2026-06-22 13:16:48 | INFO | ================================================================================
|
|
2026-06-22 13:16:48 | INFO | Processing Table: Coverage | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:16:48 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-22 13:16:48 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:16:48 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:17:03 | INFO | Fetched 512 rows from SQL Server
|
|
2026-06-22 13:17:03 | INFO | Fetched total row -: 512 from sql server for table-:Coverage ...........!!!
|
|
2026-06-22 13:17:03 | INFO | Fetched 512 rows
|
|
2026-06-22 13:17:03 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
|
|
2026-06-22 13:17:03 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:03 | INFO | Coverage: inserted 512 rows into ClickHouse
|
|
2026-06-22 13:17:03 | INFO | Coverage loaded successfully (512 rows)
|
|
2026-06-22 13:17:03 | INFO | ================================================================================
|
|
2026-06-22 13:17:03 | INFO | Processing Table: Login | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 13:17:03 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-22 13:17:03 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:17:03 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:17:05 | INFO | Fetched 454 rows from SQL Server
|
|
2026-06-22 13:17:05 | INFO | Fetched total row -: 454 from sql server for table-:Login ...........!!!
|
|
2026-06-22 13:17:05 | INFO | Fetched 454 rows
|
|
2026-06-22 13:17:05 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _
|
|
2026-06-22 13:17:05 | INFO | No delete logic required for Login
|
|
2026-06-22 13:17:05 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:05 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-22 13:17:05 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-22 13:17:05 | INFO | ================================================================================
|
|
2026-06-22 13:17:05 | INFO | Processing Table: Stock_Details | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:17:05 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-22 13:17:05 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:17:05 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:17:10 | INFO | Fetched 27,359 rows from SQL Server
|
|
2026-06-22 13:17:10 | INFO | Fetched total row -: 27359 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-22 13:17:10 | INFO | Fetched 27359 rows
|
|
2026-06-22 13:17:10 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _
|
|
2026-06-22 13:17:10 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:11 | INFO | Stock_Details: inserted 27,359 rows into ClickHouse
|
|
2026-06-22 13:17:11 | INFO | Stock_Details loaded successfully (27359 rows)
|
|
2026-06-22 13:17:11 | INFO | ================================================================================
|
|
2026-06-22 13:17:11 | INFO | Processing Table: Attendance | Table type is -: FACT | Based on -run_date and operation is used -DELETE+INSERT
|
|
2026-06-22 13:17:11 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-22 13:17:11 | INFO | Fetching Attendance data from 2026-06-06 to 2026-06-21
|
|
2026-06-22 13:17:27 | INFO | Fetched 121,639 attendance rows for 544 employees
|
|
2026-06-22 13:17:27 | INFO | Fetched total row -: 121639 from sql server for table-:Attendance ...........!!!
|
|
2026-06-22 13:17:27 | INFO | Fetched 121639 rows
|
|
2026-06-22 13:17:27 | INFO | Truncated table Attendance
|
|
2026-06-22 13:17:27 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:28 | INFO | Attendance: inserted 121,639 rows into ClickHouse
|
|
2026-06-22 13:17:28 | INFO | Attendance loaded successfully (121639 rows)
|
|
2026-06-22 13:17:28 | INFO | ================================================================================
|
|
2026-06-22 13:17:28 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 13:17:28 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-22 13:17:28 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 13:17:28 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:17:28 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:17:29 | INFO | Fetched 21,625 rows from SQL Server
|
|
2026-06-22 13:17:29 | INFO | Fetched total row -: 21625 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-22 13:17:29 | INFO | Fetched 21625 rows
|
|
2026-06-22 13:17:30 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
|
|
2026-06-22 13:17:30 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:30 | INFO | Journey_Plan: inserted 21,625 rows into ClickHouse
|
|
2026-06-22 13:17:30 | INFO | Journey_Plan loaded successfully (21625 rows)
|
|
2026-06-22 13:17:30 | INFO | ================================================================================
|
|
2026-06-22 13:17:30 | INFO | Processing Table: Web_Logins | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 13:17:30 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-22 13:17:30 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 13:17:30 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:17:30 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:17:31 | INFO | Fetched 12 rows from SQL Server
|
|
2026-06-22 13:17:31 | INFO | Fetched total row -: 12 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-22 13:17:31 | INFO | Fetched 12 rows
|
|
2026-06-22 13:17:31 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
|
|
2026-06-22 13:17:31 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:31 | INFO | Web_Logins: inserted 12 rows into ClickHouse
|
|
2026-06-22 13:17:31 | INFO | Web_Logins loaded successfully (12 rows)
|
|
2026-06-22 13:17:31 | INFO | ================================================================================
|
|
2026-06-22 13:17:31 | INFO | Processing Table: Promotion | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:17:31 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-22 13:17:31 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:17:32 | INFO | Fetched 4,223 rows from SQL Server
|
|
2026-06-22 13:17:32 | INFO | Fetched total row -: 4223 from sql server for table-:Promotion ...........!!!
|
|
2026-06-22 13:17:32 | INFO | Fetched 4223 rows
|
|
2026-06-22 13:17:32 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-22 13:17:32 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:33 | INFO | Promotion: inserted 4,223 rows into ClickHouse
|
|
2026-06-22 13:17:33 | INFO | Promotion loaded successfully (4223 rows)
|
|
2026-06-22 13:17:33 | INFO | ================================================================================
|
|
2026-06-22 13:17:33 | INFO | Processing Table: PaidVisibility | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 13:17:33 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
|
|
2026-06-22 13:17:33 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 13:17:33 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 13:17:33 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 13:17:34 | INFO | Fetched 1,284 rows from SQL Server
|
|
2026-06-22 13:17:34 | INFO | Fetched total row -: 1284 from sql server for table-:PaidVisibility ...........!!!
|
|
2026-06-22 13:17:34 | INFO | Fetched 1284 rows
|
|
2026-06-22 13:17:34 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _
|
|
2026-06-22 13:17:34 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:34 | INFO | PaidVisibility: inserted 1,284 rows into ClickHouse
|
|
2026-06-22 13:17:34 | INFO | PaidVisibility loaded successfully (1284 rows)
|
|
2026-06-22 13:17:34 | INFO | ================================================================================
|
|
2026-06-22 13:17:34 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
|
|
2026-06-22 13:17:34 | INFO | Fetching Data from sql server for table-: Store_Master ..............
|
|
2026-06-22 13:17:34 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:17:34 | INFO | Fetching in progress ....
|
|
2026-06-22 13:17:38 | INFO | Fetched 5,998 rows from SQL Server
|
|
2026-06-22 13:17:38 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
|
|
2026-06-22 13:17:38 | INFO | Fetched 5998 rows
|
|
2026-06-22 13:17:38 | INFO | Truncated table Store_Master
|
|
2026-06-22 13:17:38 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:39 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
|
|
2026-06-22 13:17:39 | INFO | Store_Master loaded successfully (5998 rows)
|
|
2026-06-22 13:17:39 | INFO | ================================================================================
|
|
2026-06-22 13:17:39 | INFO | Processing Table: SKU_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
|
|
2026-06-22 13:17:39 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
|
|
2026-06-22 13:17:39 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:17:39 | INFO | Fetching in progress ....
|
|
2026-06-22 13:17:39 | INFO | Fetched 160 rows from SQL Server
|
|
2026-06-22 13:17:39 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
|
|
2026-06-22 13:17:39 | INFO | Fetched 160 rows
|
|
2026-06-22 13:17:39 | INFO | Truncated table SKU_Master
|
|
2026-06-22 13:17:39 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:40 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-22 13:17:40 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-22 13:17:40 | INFO | ================================================================================
|
|
2026-06-22 13:17:40 | INFO | Processing Table: display_master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
|
|
2026-06-22 13:17:40 | INFO | Fetching Data from sql server for table-: display_master ..............
|
|
2026-06-22 13:17:40 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:17:40 | INFO | Fetching in progress ....
|
|
2026-06-22 13:17:40 | INFO | Fetched 135 rows from SQL Server
|
|
2026-06-22 13:17:40 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
|
|
2026-06-22 13:17:40 | INFO | Fetched 135 rows
|
|
2026-06-22 13:17:40 | INFO | Truncated table display_master
|
|
2026-06-22 13:17:40 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:40 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-22 13:17:40 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-22 13:17:40 | INFO | ================================================================================
|
|
2026-06-22 13:17:40 | INFO | Processing Table: Employee_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
|
|
2026-06-22 13:17:40 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
|
|
2026-06-22 13:17:40 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:17:40 | INFO | Fetching in progress ....
|
|
2026-06-22 13:17:41 | INFO | Fetched 2,288 rows from SQL Server
|
|
2026-06-22 13:17:41 | INFO | Fetched total row -: 2288 from sql server for table-:Employee_Master ...........!!!
|
|
2026-06-22 13:17:41 | INFO | Fetched 2288 rows
|
|
2026-06-22 13:17:41 | INFO | Truncated table Employee_Master
|
|
2026-06-22 13:17:41 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:41 | INFO | Employee_Master: inserted 2,288 rows into ClickHouse
|
|
2026-06-22 13:17:41 | INFO | Employee_Master loaded successfully (2288 rows)
|
|
2026-06-22 13:17:41 | INFO | ================================================================================
|
|
2026-06-22 13:17:41 | INFO | Processing Table: coverage_remarks | Table type is -: DIMENSION | Based on -reason_id and operation is used -DELETE+INSERT
|
|
2026-06-22 13:17:41 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
|
|
2026-06-22 13:17:42 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:17:42 | INFO | Fetching in progress ....
|
|
2026-06-22 13:17:42 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 13:17:42 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 13:17:42 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
|
|
2026-06-22 13:17:42 | WARNING | coverage_remarks returned no rows
|
|
2026-06-22 13:17:42 | INFO | ================================================================================
|
|
2026-06-22 13:17:42 | INFO | Processing Table: mapping_storevisibility | Table type is -: BRIDGE | Based on -run_date and operation is used -ONLY_INSERT
|
|
2026-06-22 13:17:42 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
|
|
2026-06-22 13:17:42 | WARNING | Table 'mapping_storevisibility' does not exist. During collecting store_ids
|
|
2026-06-22 13:17:42 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:17:42 | INFO | Fetching in progress ....
|
|
2026-06-22 13:17:42 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 13:17:42 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 13:17:42 | INFO | Fetched total row -: 0 from sql server for table-:mapping_storevisibility ...........!!!
|
|
2026-06-22 13:17:42 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-22 13:17:42 | INFO | ================================================================================
|
|
2026-06-22 13:17:42 | INFO | Processing Table: Master_VisibilityReason | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
|
|
2026-06-22 13:17:42 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason ..............
|
|
2026-06-22 13:17:42 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:17:42 | INFO | Fetching in progress ....
|
|
2026-06-22 13:17:42 | INFO | Fetched 17 rows from SQL Server
|
|
2026-06-22 13:17:42 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!!
|
|
2026-06-22 13:17:42 | INFO | Fetched 17 rows
|
|
2026-06-22 13:17:43 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-22 13:17:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:43 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-22 13:17:43 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-22 13:17:43 | INFO | ================================================================================
|
|
2026-06-22 13:17:43 | INFO | Processing Table: Master_VisibilityDefinition | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
|
|
2026-06-22 13:17:43 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition ..............
|
|
2026-06-22 13:17:43 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:17:43 | INFO | Fetching in progress ....
|
|
2026-06-22 13:17:43 | INFO | Fetched 861 rows from SQL Server
|
|
2026-06-22 13:17:43 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!!
|
|
2026-06-22 13:17:43 | INFO | Fetched 861 rows
|
|
2026-06-22 13:17:43 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-22 13:17:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:43 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-22 13:17:43 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-22 13:17:43 | INFO | ================================================================================
|
|
2026-06-22 13:17:43 | INFO | Processing Table: Master_Salesterritorylayer | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
|
|
2026-06-22 13:17:43 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer ..............
|
|
2026-06-22 13:17:43 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 13:17:43 | INFO | Fetching in progress ....
|
|
2026-06-22 13:17:43 | INFO | Fetched 33 rows from SQL Server
|
|
2026-06-22 13:17:43 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!!
|
|
2026-06-22 13:17:43 | INFO | Fetched 33 rows
|
|
2026-06-22 13:17:43 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-22 13:17:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 13:17:44 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-22 13:17:44 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-22 13:17:44 | INFO | ================================================================================
|
|
2026-06-22 13:17:44 | INFO | Pipeline Completed Successfully
|
|
2026-06-22 13:17:44 | INFO | ================================================================================
|
|
2026-06-22 15:38:19 | INFO | ================================================================================
|
|
2026-06-22 15:38:19 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 15:38:19 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 15:38:19 | INFO | Connecting to databases...
|
|
2026-06-22 15:38:21 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000021419871230>
|
|
2026-06-22 15:38:25 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002141A9A60D0>
|
|
2026-06-22 15:38:26 | INFO | Database connections established
|
|
2026-06-22 15:38:26 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 15:38:26 | INFO | Found 569 MIDs
|
|
2026-06-22 15:38:26 | INFO | ================================================================================
|
|
2026-06-22 15:38:26 | INFO | Processing Table: SOS_OneApp | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 15:38:26 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-22 15:38:26 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:38:26 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:38:26 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:38:36 | INFO | Fetched 1,907 rows from SQL Server
|
|
2026-06-22 15:38:36 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-22 15:38:36 | INFO | Fetched 1907 rows
|
|
2026-06-22 15:38:36 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-22 15:38:36 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:38:37 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse
|
|
2026-06-22 15:38:37 | INFO | SOS_OneApp loaded successfully (1907 rows)
|
|
2026-06-22 15:38:37 | INFO | ================================================================================
|
|
2026-06-22 15:38:37 | INFO | Processing Table: OQaD | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 15:38:37 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-22 15:38:37 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-22 15:39:02 | INFO | Fetched 291,198 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-22 15:39:06 | INFO | Fetched 290,722 matched empids fetched for OQAD
|
|
2026-06-22 15:39:06 | INFO | Exists: True
|
|
2026-06-22 15:39:06 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-22 15:39:06 | INFO | Fetching data for 290,722 EMPIDs
|
|
2026-06-22 15:39:06 | INFO | Fetching OQaD data for run_date=2026-06-21
|
|
2026-06-22 15:40:18 | INFO | fn name is fetch_OQad ------Fetched 0 rows
|
|
2026-06-22 15:40:18 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:40:18 | INFO | Fetched total row -: 0 from sql server for table-:OQaD ...........!!!
|
|
2026-06-22 15:40:18 | WARNING | OQaD returned no rows
|
|
2026-06-22 15:40:18 | INFO | ================================================================================
|
|
2026-06-22 15:40:18 | INFO | Processing Table: Survey | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 15:40:18 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-22 15:40:18 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:40:18 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:40:19 | INFO | Fetched 165 rows from SQL Server
|
|
2026-06-22 15:40:19 | INFO | Fetched total row -: 165 from sql server for table-:Survey ...........!!!
|
|
2026-06-22 15:40:19 | INFO | Fetched 165 rows
|
|
2026-06-22 15:40:19 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
|
|
2026-06-22 15:40:19 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:40:19 | INFO | Survey: inserted 165 rows into ClickHouse
|
|
2026-06-22 15:40:19 | INFO | Survey loaded successfully (165 rows)
|
|
2026-06-22 15:40:19 | INFO | ================================================================================
|
|
2026-06-22 15:40:19 | INFO | Processing Table: additional_visibility | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 15:40:19 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-22 15:40:19 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:40:19 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:40:19 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:40:20 | INFO | Fetched 1,579 rows from SQL Server
|
|
2026-06-22 15:40:20 | INFO | Fetched total row -: 1579 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-22 15:40:20 | INFO | Fetched 1579 rows
|
|
2026-06-22 15:40:20 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
|
|
2026-06-22 15:40:20 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:40:20 | INFO | additional_visibility: inserted 1,579 rows into ClickHouse
|
|
2026-06-22 15:40:20 | INFO | additional_visibility loaded successfully (1579 rows)
|
|
2026-06-22 15:40:20 | INFO | ================================================================================
|
|
2026-06-22 15:40:20 | INFO | Processing Table: Coverage | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 15:40:20 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-22 15:40:20 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:40:20 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:40:42 | INFO | Fetched 512 rows from SQL Server
|
|
2026-06-22 15:40:42 | INFO | Fetched total row -: 512 from sql server for table-:Coverage ...........!!!
|
|
2026-06-22 15:40:42 | INFO | Fetched 512 rows
|
|
2026-06-22 15:40:43 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
|
|
2026-06-22 15:40:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:40:43 | INFO | Coverage: inserted 512 rows into ClickHouse
|
|
2026-06-22 15:40:43 | INFO | Coverage loaded successfully (512 rows)
|
|
2026-06-22 15:40:43 | INFO | ================================================================================
|
|
2026-06-22 15:40:43 | INFO | Processing Table: Login | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 15:40:43 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-22 15:40:43 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:40:43 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:40:46 | INFO | Fetched 454 rows from SQL Server
|
|
2026-06-22 15:40:46 | INFO | Fetched total row -: 454 from sql server for table-:Login ...........!!!
|
|
2026-06-22 15:40:46 | INFO | Fetched 454 rows
|
|
2026-06-22 15:40:46 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _
|
|
2026-06-22 15:40:46 | INFO | No delete logic required for Login
|
|
2026-06-22 15:40:46 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:40:46 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-22 15:40:46 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-22 15:40:46 | INFO | ================================================================================
|
|
2026-06-22 15:40:46 | INFO | Processing Table: Stock_Details | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 15:40:46 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-22 15:40:46 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:40:46 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:40:54 | INFO | Fetched 27,359 rows from SQL Server
|
|
2026-06-22 15:40:54 | INFO | Fetched total row -: 27359 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-22 15:40:54 | INFO | Fetched 27359 rows
|
|
2026-06-22 15:40:54 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _
|
|
2026-06-22 15:40:54 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:40:55 | INFO | Stock_Details: inserted 27,359 rows into ClickHouse
|
|
2026-06-22 15:40:55 | INFO | Stock_Details loaded successfully (27359 rows)
|
|
2026-06-22 15:40:55 | INFO | ================================================================================
|
|
2026-06-22 15:40:55 | INFO | Processing Table: Attendance | Table type is -: FACT | Based on -run_date and operation is used -DELETE+INSERT
|
|
2026-06-22 15:40:55 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-22 15:40:55 | INFO | Fetching Attendance data from 2026-06-06 to 2026-06-21
|
|
2026-06-22 15:41:17 | INFO | Fetched 121,642 attendance rows for 545 employees
|
|
2026-06-22 15:41:17 | INFO | Fetched total row -: 121642 from sql server for table-:Attendance ...........!!!
|
|
2026-06-22 15:41:17 | INFO | Fetched 121642 rows
|
|
2026-06-22 15:41:17 | INFO | Truncated table Attendance
|
|
2026-06-22 15:41:17 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:41:18 | INFO | Attendance: inserted 121,642 rows into ClickHouse
|
|
2026-06-22 15:41:18 | INFO | Attendance loaded successfully (121642 rows)
|
|
2026-06-22 15:41:18 | INFO | ================================================================================
|
|
2026-06-22 15:41:18 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 15:41:18 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-22 15:41:18 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:41:18 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:41:18 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:41:20 | INFO | Fetched 21,643 rows from SQL Server
|
|
2026-06-22 15:41:20 | INFO | Fetched total row -: 21643 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-22 15:41:20 | INFO | Fetched 21643 rows
|
|
2026-06-22 15:41:20 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
|
|
2026-06-22 15:41:20 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:41:21 | INFO | Journey_Plan: inserted 21,643 rows into ClickHouse
|
|
2026-06-22 15:41:21 | INFO | Journey_Plan loaded successfully (21643 rows)
|
|
2026-06-22 15:41:21 | INFO | ================================================================================
|
|
2026-06-22 15:41:21 | INFO | Processing Table: Web_Logins | Table type is -: FACT | Based on -run_date and operation is used -INSERT
|
|
2026-06-22 15:41:21 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-22 15:41:21 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:41:21 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:41:21 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:41:21 | INFO | Fetched 12 rows from SQL Server
|
|
2026-06-22 15:41:21 | INFO | Fetched total row -: 12 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-22 15:41:21 | INFO | Fetched 12 rows
|
|
2026-06-22 15:41:21 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
|
|
2026-06-22 15:41:21 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:41:22 | INFO | Web_Logins: inserted 12 rows into ClickHouse
|
|
2026-06-22 15:41:22 | INFO | Web_Logins loaded successfully (12 rows)
|
|
2026-06-22 15:41:22 | INFO | ================================================================================
|
|
2026-06-22 15:41:22 | INFO | Processing Table: Promotion | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 15:41:22 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-22 15:41:22 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:41:25 | INFO | Fetched 4,223 rows from SQL Server
|
|
2026-06-22 15:41:25 | INFO | Fetched total row -: 4223 from sql server for table-:Promotion ...........!!!
|
|
2026-06-22 15:41:25 | INFO | Fetched 4223 rows
|
|
2026-06-22 15:41:25 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-22 15:41:25 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:41:25 | INFO | Promotion: inserted 4,223 rows into ClickHouse
|
|
2026-06-22 15:41:25 | INFO | Promotion loaded successfully (4223 rows)
|
|
2026-06-22 15:41:25 | INFO | ================================================================================
|
|
2026-06-22 15:41:25 | INFO | Processing Table: PaidVisibility | Table type is -: FACT | Based on -mids and operation is used -INSERT
|
|
2026-06-22 15:41:25 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
|
|
2026-06-22 15:41:25 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:41:25 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:41:25 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:41:27 | INFO | Fetched 1,284 rows from SQL Server
|
|
2026-06-22 15:41:27 | INFO | Fetched total row -: 1284 from sql server for table-:PaidVisibility ...........!!!
|
|
2026-06-22 15:41:27 | INFO | Fetched 1284 rows
|
|
2026-06-22 15:41:27 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _
|
|
2026-06-22 15:41:27 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:41:28 | INFO | PaidVisibility: inserted 1,284 rows into ClickHouse
|
|
2026-06-22 15:41:28 | INFO | PaidVisibility loaded successfully (1284 rows)
|
|
2026-06-22 15:41:28 | INFO | ================================================================================
|
|
2026-06-22 15:41:28 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
|
|
2026-06-22 15:41:28 | INFO | Fetching Data from sql server for table-: Store_Master ..............
|
|
2026-06-22 15:41:28 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:41:28 | INFO | Fetching in progress ....
|
|
2026-06-22 15:41:34 | INFO | Fetched 5,998 rows from SQL Server
|
|
2026-06-22 15:41:34 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
|
|
2026-06-22 15:41:34 | INFO | Fetched 5998 rows
|
|
2026-06-22 15:41:35 | INFO | Truncated table Store_Master
|
|
2026-06-22 15:41:35 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:41:35 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
|
|
2026-06-22 15:41:35 | INFO | Store_Master loaded successfully (5998 rows)
|
|
2026-06-22 15:41:35 | INFO | ================================================================================
|
|
2026-06-22 15:41:35 | INFO | Processing Table: SKU_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
|
|
2026-06-22 15:41:35 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
|
|
2026-06-22 15:41:35 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:41:35 | INFO | Fetching in progress ....
|
|
2026-06-22 15:41:36 | INFO | Fetched 160 rows from SQL Server
|
|
2026-06-22 15:41:36 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
|
|
2026-06-22 15:41:36 | INFO | Fetched 160 rows
|
|
2026-06-22 15:41:36 | INFO | Truncated table SKU_Master
|
|
2026-06-22 15:41:36 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:41:36 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-22 15:41:36 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-22 15:41:36 | INFO | ================================================================================
|
|
2026-06-22 15:41:36 | INFO | Processing Table: display_master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
|
|
2026-06-22 15:41:36 | INFO | Fetching Data from sql server for table-: display_master ..............
|
|
2026-06-22 15:41:36 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:41:36 | INFO | Fetching in progress ....
|
|
2026-06-22 15:41:36 | INFO | Fetched 135 rows from SQL Server
|
|
2026-06-22 15:41:36 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
|
|
2026-06-22 15:41:36 | INFO | Fetched 135 rows
|
|
2026-06-22 15:41:36 | INFO | Truncated table display_master
|
|
2026-06-22 15:41:36 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:41:36 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-22 15:41:36 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-22 15:41:36 | INFO | ================================================================================
|
|
2026-06-22 15:41:36 | INFO | Processing Table: Employee_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
|
|
2026-06-22 15:41:36 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
|
|
2026-06-22 15:41:36 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:41:36 | INFO | Fetching in progress ....
|
|
2026-06-22 15:41:37 | INFO | Fetched 2,289 rows from SQL Server
|
|
2026-06-22 15:41:37 | INFO | Fetched total row -: 2289 from sql server for table-:Employee_Master ...........!!!
|
|
2026-06-22 15:41:37 | INFO | Fetched 2289 rows
|
|
2026-06-22 15:41:37 | INFO | Truncated table Employee_Master
|
|
2026-06-22 15:41:37 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:41:37 | INFO | Employee_Master: inserted 2,289 rows into ClickHouse
|
|
2026-06-22 15:41:37 | INFO | Employee_Master loaded successfully (2289 rows)
|
|
2026-06-22 15:41:37 | INFO | ================================================================================
|
|
2026-06-22 15:41:37 | INFO | Processing Table: coverage_remarks | Table type is -: DIMENSION | Based on -reason_id and operation is used -DELETE+INSERT
|
|
2026-06-22 15:41:37 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
|
|
2026-06-22 15:41:38 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:41:38 | INFO | Fetching in progress ....
|
|
2026-06-22 15:41:38 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:41:38 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:41:38 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
|
|
2026-06-22 15:41:38 | WARNING | coverage_remarks returned no rows
|
|
2026-06-22 15:41:38 | INFO | ================================================================================
|
|
2026-06-22 15:41:38 | INFO | Processing Table: mapping_storevisibility | Table type is -: BRIDGE | Based on -run_date and operation is used -ONLY_INSERT
|
|
2026-06-22 15:41:38 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
|
|
2026-06-22 15:41:38 | WARNING | Table 'mapping_storevisibility' does not exist. During collecting store_ids
|
|
2026-06-22 15:41:38 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:41:38 | INFO | Fetching in progress ....
|
|
2026-06-22 15:41:39 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:41:39 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:41:39 | INFO | Fetched total row -: 0 from sql server for table-:mapping_storevisibility ...........!!!
|
|
2026-06-22 15:41:39 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-22 15:41:39 | INFO | ================================================================================
|
|
2026-06-22 15:41:39 | INFO | Processing Table: Master_VisibilityReason | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
|
|
2026-06-22 15:41:39 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason ..............
|
|
2026-06-22 15:41:39 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:41:39 | INFO | Fetching in progress ....
|
|
2026-06-22 15:41:39 | INFO | Fetched 17 rows from SQL Server
|
|
2026-06-22 15:41:39 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!!
|
|
2026-06-22 15:41:39 | INFO | Fetched 17 rows
|
|
2026-06-22 15:41:39 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-22 15:41:39 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:41:39 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-22 15:41:39 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-22 15:41:39 | INFO | ================================================================================
|
|
2026-06-22 15:41:39 | INFO | Processing Table: Master_VisibilityDefinition | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
|
|
2026-06-22 15:41:39 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition ..............
|
|
2026-06-22 15:41:39 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:41:39 | INFO | Fetching in progress ....
|
|
2026-06-22 15:41:39 | INFO | Fetched 861 rows from SQL Server
|
|
2026-06-22 15:41:39 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!!
|
|
2026-06-22 15:41:39 | INFO | Fetched 861 rows
|
|
2026-06-22 15:41:39 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-22 15:41:39 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:41:39 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-22 15:41:39 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-22 15:41:39 | INFO | ================================================================================
|
|
2026-06-22 15:41:39 | INFO | Processing Table: Master_Salesterritorylayer | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
|
|
2026-06-22 15:41:39 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer ..............
|
|
2026-06-22 15:41:39 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:41:39 | INFO | Fetching in progress ....
|
|
2026-06-22 15:41:40 | INFO | Fetched 33 rows from SQL Server
|
|
2026-06-22 15:41:40 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!!
|
|
2026-06-22 15:41:40 | INFO | Fetched 33 rows
|
|
2026-06-22 15:41:40 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-22 15:41:40 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:41:40 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-22 15:41:40 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-22 15:41:40 | INFO | ================================================================================
|
|
2026-06-22 15:41:40 | INFO | Pipeline Completed Successfully
|
|
2026-06-22 15:41:40 | INFO | ================================================================================
|
|
2026-06-22 15:47:12 | INFO | ================================================================================
|
|
2026-06-22 15:47:12 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 15:47:12 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 15:47:12 | INFO | Connecting to databases...
|
|
2026-06-22 15:47:13 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001703E4D4890>
|
|
2026-06-22 15:47:14 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001703F6087D0>
|
|
2026-06-22 15:47:15 | INFO | Database connections established
|
|
2026-06-22 15:47:15 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 15:47:15 | INFO | Found 569 MIDs
|
|
2026-06-22 15:47:16 | INFO | ================================================================================
|
|
2026-06-22 15:47:16 | INFO | Processing Table-:SOS_OneApp | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:47:16 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-22 15:47:16 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:47:16 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:47:16 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:47:19 | INFO | Fetched 1,907 rows from SQL Server
|
|
2026-06-22 15:47:19 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-22 15:47:19 | INFO | Fetched 1907 rows
|
|
2026-06-22 15:47:19 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-22 15:47:19 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:47:20 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse
|
|
2026-06-22 15:47:20 | INFO | SOS_OneApp loaded successfully (1907 rows)
|
|
2026-06-22 15:47:20 | INFO | ================================================================================
|
|
2026-06-22 15:47:20 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-22 15:47:20 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-22 15:47:20 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-22 15:47:35 | INFO | Fetched 291,198 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-22 15:47:37 | INFO | Fetched 290,722 matched empids fetched for OQAD
|
|
2026-06-22 15:47:37 | INFO | Exists: True
|
|
2026-06-22 15:47:37 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-22 15:47:37 | INFO | Fetching data for 290,722 EMPIDs
|
|
2026-06-22 15:47:37 | INFO | Fetching OQaD data for run_date=2026-06-21
|
|
2026-06-22 15:48:22 | INFO | fn name is fetch_OQad ------Fetched 0 rows
|
|
2026-06-22 15:48:22 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:48:22 | INFO | Fetched total row -: 0 from sql server for table-:OQaD ...........!!!
|
|
2026-06-22 15:48:22 | WARNING | OQaD returned no rows
|
|
2026-06-22 15:48:22 | INFO | ================================================================================
|
|
2026-06-22 15:48:22 | INFO | Processing Table-:Survey | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:48:22 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-22 15:48:22 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:48:22 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:48:22 | INFO | Fetched 165 rows from SQL Server
|
|
2026-06-22 15:48:22 | INFO | Fetched total row -: 165 from sql server for table-:Survey ...........!!!
|
|
2026-06-22 15:48:22 | INFO | Fetched 165 rows
|
|
2026-06-22 15:48:22 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
|
|
2026-06-22 15:48:22 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:48:23 | INFO | Survey: inserted 165 rows into ClickHouse
|
|
2026-06-22 15:48:23 | INFO | Survey loaded successfully (165 rows)
|
|
2026-06-22 15:48:23 | INFO | ================================================================================
|
|
2026-06-22 15:48:23 | INFO | Processing Table-:additional_visibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:48:23 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-22 15:48:23 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:48:23 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:48:23 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:48:23 | INFO | Fetched 1,579 rows from SQL Server
|
|
2026-06-22 15:48:23 | INFO | Fetched total row -: 1579 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-22 15:48:23 | INFO | Fetched 1579 rows
|
|
2026-06-22 15:48:23 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
|
|
2026-06-22 15:48:23 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:48:24 | INFO | additional_visibility: inserted 1,579 rows into ClickHouse
|
|
2026-06-22 15:48:24 | INFO | additional_visibility loaded successfully (1579 rows)
|
|
2026-06-22 15:48:24 | INFO | ================================================================================
|
|
2026-06-22 15:48:24 | INFO | Processing Table-:Coverage | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:48:24 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-22 15:48:24 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:48:24 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:48:38 | INFO | Fetched 512 rows from SQL Server
|
|
2026-06-22 15:48:38 | INFO | Fetched total row -: 512 from sql server for table-:Coverage ...........!!!
|
|
2026-06-22 15:48:38 | INFO | Fetched 512 rows
|
|
2026-06-22 15:48:38 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
|
|
2026-06-22 15:48:38 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:48:38 | INFO | Coverage: inserted 512 rows into ClickHouse
|
|
2026-06-22 15:48:38 | INFO | Coverage loaded successfully (512 rows)
|
|
2026-06-22 15:48:38 | INFO | ================================================================================
|
|
2026-06-22 15:48:38 | INFO | Processing Table-:Login | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-22 15:48:38 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-22 15:48:38 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:48:38 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:48:40 | INFO | Fetched 454 rows from SQL Server
|
|
2026-06-22 15:48:40 | INFO | Fetched total row -: 454 from sql server for table-:Login ...........!!!
|
|
2026-06-22 15:48:40 | INFO | Fetched 454 rows
|
|
2026-06-22 15:48:40 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _
|
|
2026-06-22 15:48:40 | INFO | No delete logic required for Login
|
|
2026-06-22 15:48:40 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:48:40 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-22 15:48:40 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-22 15:48:40 | INFO | ================================================================================
|
|
2026-06-22 15:48:40 | INFO | Processing Table-:Stock_Details | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:48:40 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-22 15:48:40 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:48:40 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:48:45 | INFO | Fetched 27,359 rows from SQL Server
|
|
2026-06-22 15:48:45 | INFO | Fetched total row -: 27359 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-22 15:48:45 | INFO | Fetched 27359 rows
|
|
2026-06-22 15:48:45 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _
|
|
2026-06-22 15:48:45 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:48:45 | INFO | Stock_Details: inserted 27,359 rows into ClickHouse
|
|
2026-06-22 15:48:45 | INFO | Stock_Details loaded successfully (27359 rows)
|
|
2026-06-22 15:48:45 | INFO | ================================================================================
|
|
2026-06-22 15:48:45 | INFO | Processing Table-:Attendance | Table type -:FACT | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-22 15:48:45 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-22 15:48:45 | INFO | Fetching Attendance data from 2026-06-06 to 2026-06-21
|
|
2026-06-22 15:49:08 | INFO | Fetched 121,642 attendance rows for 545 employees
|
|
2026-06-22 15:49:08 | INFO | Fetched total row -: 121642 from sql server for table-:Attendance ...........!!!
|
|
2026-06-22 15:49:08 | INFO | Fetched 121642 rows
|
|
2026-06-22 15:49:08 | INFO | Truncated table Attendance
|
|
2026-06-22 15:49:08 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:49:09 | INFO | Attendance: inserted 121,642 rows into ClickHouse
|
|
2026-06-22 15:49:09 | INFO | Attendance loaded successfully (121642 rows)
|
|
2026-06-22 15:49:09 | INFO | ================================================================================
|
|
2026-06-22 15:49:09 | INFO | Processing Table-:Journey_Plan | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-22 15:49:09 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-22 15:49:09 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:49:09 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:49:09 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:49:10 | INFO | Fetched 21,643 rows from SQL Server
|
|
2026-06-22 15:49:10 | INFO | Fetched total row -: 21643 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-22 15:49:10 | INFO | Fetched 21643 rows
|
|
2026-06-22 15:49:10 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
|
|
2026-06-22 15:49:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:49:11 | INFO | Journey_Plan: inserted 21,643 rows into ClickHouse
|
|
2026-06-22 15:49:11 | INFO | Journey_Plan loaded successfully (21643 rows)
|
|
2026-06-22 15:49:11 | INFO | ================================================================================
|
|
2026-06-22 15:49:11 | INFO | Processing Table-:Web_Logins | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-22 15:49:11 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-22 15:49:11 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:49:11 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:49:11 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:49:11 | INFO | Fetched 12 rows from SQL Server
|
|
2026-06-22 15:49:11 | INFO | Fetched total row -: 12 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-22 15:49:11 | INFO | Fetched 12 rows
|
|
2026-06-22 15:49:11 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
|
|
2026-06-22 15:49:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:49:11 | INFO | Web_Logins: inserted 12 rows into ClickHouse
|
|
2026-06-22 15:49:11 | INFO | Web_Logins loaded successfully (12 rows)
|
|
2026-06-22 15:49:11 | INFO | ================================================================================
|
|
2026-06-22 15:49:11 | INFO | Processing Table-:Promotion | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:49:11 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-22 15:49:11 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:49:14 | INFO | Fetched 4,223 rows from SQL Server
|
|
2026-06-22 15:49:14 | INFO | Fetched total row -: 4223 from sql server for table-:Promotion ...........!!!
|
|
2026-06-22 15:49:14 | INFO | Fetched 4223 rows
|
|
2026-06-22 15:49:14 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-22 15:49:14 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:49:14 | INFO | Promotion: inserted 4,223 rows into ClickHouse
|
|
2026-06-22 15:49:14 | INFO | Promotion loaded successfully (4223 rows)
|
|
2026-06-22 15:49:14 | INFO | ================================================================================
|
|
2026-06-22 15:49:14 | INFO | Processing Table-:PaidVisibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:49:14 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
|
|
2026-06-22 15:49:14 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:49:14 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:49:14 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:49:16 | INFO | Fetched 1,284 rows from SQL Server
|
|
2026-06-22 15:49:16 | INFO | Fetched total row -: 1284 from sql server for table-:PaidVisibility ...........!!!
|
|
2026-06-22 15:49:16 | INFO | Fetched 1284 rows
|
|
2026-06-22 15:49:16 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _
|
|
2026-06-22 15:49:16 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:49:16 | INFO | PaidVisibility: inserted 1,284 rows into ClickHouse
|
|
2026-06-22 15:49:16 | INFO | PaidVisibility loaded successfully (1284 rows)
|
|
2026-06-22 15:49:16 | INFO | ================================================================================
|
|
2026-06-22 15:49:16 | INFO | Processing Table-:Store_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-22 15:49:16 | INFO | Fetching Data from sql server for table-: Store_Master ..............
|
|
2026-06-22 15:49:16 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:49:16 | INFO | Fetching in progress ....
|
|
2026-06-22 15:49:20 | INFO | Fetched 5,998 rows from SQL Server
|
|
2026-06-22 15:49:20 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
|
|
2026-06-22 15:49:20 | INFO | Fetched 5998 rows
|
|
2026-06-22 15:49:20 | INFO | Truncated table Store_Master
|
|
2026-06-22 15:49:20 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:49:21 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
|
|
2026-06-22 15:49:21 | INFO | Store_Master loaded successfully (5998 rows)
|
|
2026-06-22 15:49:21 | INFO | ================================================================================
|
|
2026-06-22 15:49:21 | INFO | Processing Table-:SKU_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-22 15:49:21 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
|
|
2026-06-22 15:49:21 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:49:21 | INFO | Fetching in progress ....
|
|
2026-06-22 15:49:21 | INFO | Fetched 160 rows from SQL Server
|
|
2026-06-22 15:49:21 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
|
|
2026-06-22 15:49:21 | INFO | Fetched 160 rows
|
|
2026-06-22 15:49:21 | INFO | Truncated table SKU_Master
|
|
2026-06-22 15:49:21 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:49:21 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-22 15:49:21 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-22 15:49:21 | INFO | ================================================================================
|
|
2026-06-22 15:49:21 | INFO | Processing Table-:display_master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-22 15:49:21 | INFO | Fetching Data from sql server for table-: display_master ..............
|
|
2026-06-22 15:49:21 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:49:21 | INFO | Fetching in progress ....
|
|
2026-06-22 15:49:21 | INFO | Fetched 135 rows from SQL Server
|
|
2026-06-22 15:49:21 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
|
|
2026-06-22 15:49:21 | INFO | Fetched 135 rows
|
|
2026-06-22 15:49:22 | INFO | Truncated table display_master
|
|
2026-06-22 15:49:22 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:49:22 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-22 15:49:22 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-22 15:49:22 | INFO | ================================================================================
|
|
2026-06-22 15:49:22 | INFO | Processing Table-:Employee_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-22 15:49:22 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
|
|
2026-06-22 15:49:22 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:49:22 | INFO | Fetching in progress ....
|
|
2026-06-22 15:49:22 | INFO | Fetched 2,289 rows from SQL Server
|
|
2026-06-22 15:49:22 | INFO | Fetched total row -: 2289 from sql server for table-:Employee_Master ...........!!!
|
|
2026-06-22 15:49:22 | INFO | Fetched 2289 rows
|
|
2026-06-22 15:49:22 | INFO | Truncated table Employee_Master
|
|
2026-06-22 15:49:22 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:49:23 | INFO | Employee_Master: inserted 2,289 rows into ClickHouse
|
|
2026-06-22 15:49:23 | INFO | Employee_Master loaded successfully (2289 rows)
|
|
2026-06-22 15:49:23 | INFO | ================================================================================
|
|
2026-06-22 15:49:23 | INFO | Processing Table-:coverage_remarks | Table type -:DIMENSION | fetcht by-:reason_id | operation-:DELETE+INSERT
|
|
2026-06-22 15:49:23 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
|
|
2026-06-22 15:49:23 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:49:23 | INFO | Fetching in progress ....
|
|
2026-06-22 15:49:23 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:49:23 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:49:23 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
|
|
2026-06-22 15:49:23 | WARNING | coverage_remarks returned no rows
|
|
2026-06-22 15:49:23 | INFO | ================================================================================
|
|
2026-06-22 15:49:23 | INFO | Processing Table-:mapping_storevisibility | Table type -:BRIDGE | fetcht by-:run_date | operation-:ONLY_INSERT
|
|
2026-06-22 15:49:23 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
|
|
2026-06-22 15:49:24 | WARNING | Table 'mapping_storevisibility' does not exist. During collecting store_ids
|
|
2026-06-22 15:49:24 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:49:24 | INFO | Fetching in progress ....
|
|
2026-06-22 15:49:24 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:49:24 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:49:24 | INFO | Fetched total row -: 0 from sql server for table-:mapping_storevisibility ...........!!!
|
|
2026-06-22 15:49:24 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-22 15:49:24 | INFO | ================================================================================
|
|
2026-06-22 15:49:24 | INFO | Processing Table-:Master_VisibilityReason | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-22 15:49:24 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason ..............
|
|
2026-06-22 15:49:24 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:49:24 | INFO | Fetching in progress ....
|
|
2026-06-22 15:49:24 | INFO | Fetched 17 rows from SQL Server
|
|
2026-06-22 15:49:24 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!!
|
|
2026-06-22 15:49:24 | INFO | Fetched 17 rows
|
|
2026-06-22 15:49:24 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-22 15:49:24 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:49:24 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-22 15:49:24 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-22 15:49:24 | INFO | ================================================================================
|
|
2026-06-22 15:49:24 | INFO | Processing Table-:Master_VisibilityDefinition | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-22 15:49:24 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition ..............
|
|
2026-06-22 15:49:24 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:49:24 | INFO | Fetching in progress ....
|
|
2026-06-22 15:49:25 | INFO | Fetched 861 rows from SQL Server
|
|
2026-06-22 15:49:25 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!!
|
|
2026-06-22 15:49:25 | INFO | Fetched 861 rows
|
|
2026-06-22 15:49:25 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-22 15:49:25 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:49:25 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-22 15:49:25 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-22 15:49:25 | INFO | ================================================================================
|
|
2026-06-22 15:49:25 | INFO | Processing Table-:Master_Salesterritorylayer | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-22 15:49:25 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer ..............
|
|
2026-06-22 15:49:25 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:49:25 | INFO | Fetching in progress ....
|
|
2026-06-22 15:49:25 | INFO | Fetched 33 rows from SQL Server
|
|
2026-06-22 15:49:25 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!!
|
|
2026-06-22 15:49:25 | INFO | Fetched 33 rows
|
|
2026-06-22 15:49:25 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-22 15:49:25 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:49:25 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-22 15:49:25 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-22 15:49:25 | INFO | ================================================================================
|
|
2026-06-22 15:49:25 | INFO | Pipeline Completed Successfully
|
|
2026-06-22 15:49:25 | INFO | ================================================================================
|
|
2026-06-22 15:49:34 | INFO | ================================================================================
|
|
2026-06-22 15:49:34 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-22 15:49:34 | INFO | Pipeline Run Date: 2026-06-21
|
|
2026-06-22 15:49:34 | INFO | Connecting to databases...
|
|
2026-06-22 15:49:35 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001E8B4840890>
|
|
2026-06-22 15:49:37 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001E8B59047D0>
|
|
2026-06-22 15:49:37 | INFO | Database connections established
|
|
2026-06-22 15:49:37 | INFO | Collecting MIDs for: 2026-06-21
|
|
2026-06-22 15:49:38 | INFO | Found 569 MIDs
|
|
2026-06-22 15:49:38 | INFO | ================================================================================
|
|
2026-06-22 15:49:38 | INFO | Processing Table-:SOS_OneApp | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:49:38 | INFO | Fetching Data from sql server for table-: SOS_OneApp ..............
|
|
2026-06-22 15:49:38 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:49:38 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:49:38 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:49:39 | INFO | Fetched 1,907 rows from SQL Server
|
|
2026-06-22 15:49:39 | INFO | Fetched total row -: 1907 from sql server for table-:SOS_OneApp ...........!!!
|
|
2026-06-22 15:49:39 | INFO | Fetched 1907 rows
|
|
2026-06-22 15:49:39 | INFO | _ _ _ _ Deleting Data from ClickHouse for SOS_OneApp _ _ _ _
|
|
2026-06-22 15:49:40 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:49:40 | INFO | SOS_OneApp: inserted 1,907 rows into ClickHouse
|
|
2026-06-22 15:49:40 | INFO | SOS_OneApp loaded successfully (1907 rows)
|
|
2026-06-22 15:49:40 | INFO | ================================================================================
|
|
2026-06-22 15:49:40 | INFO | Processing Table-:OQaD | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-22 15:49:40 | INFO | Fetching Data from sql server for table-: OQaD ..............
|
|
2026-06-22 15:49:41 | INFO | Fetching quiz_empids data for EMPID and Visitid
|
|
2026-06-22 15:49:54 | INFO | Fetched 291,198 total empid and visitdate fetched for OQAD from SQL Server
|
|
2026-06-22 15:49:58 | INFO | Fetched 290,722 matched empids fetched for OQAD
|
|
2026-06-22 15:49:58 | INFO | Exists: True
|
|
2026-06-22 15:49:58 | INFO | Path: D:\z\airflow-project\src\sql\fact\OQaD.sql
|
|
2026-06-22 15:49:58 | INFO | Fetching data for 290,722 EMPIDs
|
|
2026-06-22 15:49:58 | INFO | Fetching OQaD data for run_date=2026-06-21
|
|
2026-06-22 15:49:59 | INFO | fn name is fetch_OQad ------Fetched 0 rows
|
|
2026-06-22 15:49:59 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:49:59 | INFO | Fetched total row -: 0 from sql server for table-:OQaD ...........!!!
|
|
2026-06-22 15:49:59 | WARNING | OQaD returned no rows
|
|
2026-06-22 15:49:59 | INFO | ================================================================================
|
|
2026-06-22 15:49:59 | INFO | Processing Table-:Survey | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:49:59 | INFO | Fetching Data from sql server for table-: Survey ..............
|
|
2026-06-22 15:49:59 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:49:59 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:50:00 | INFO | Fetched 165 rows from SQL Server
|
|
2026-06-22 15:50:00 | INFO | Fetched total row -: 165 from sql server for table-:Survey ...........!!!
|
|
2026-06-22 15:50:00 | INFO | Fetched 165 rows
|
|
2026-06-22 15:50:00 | INFO | _ _ _ _ Deleting Data from ClickHouse for Survey _ _ _ _
|
|
2026-06-22 15:50:00 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:50:00 | INFO | Survey: inserted 165 rows into ClickHouse
|
|
2026-06-22 15:50:00 | INFO | Survey loaded successfully (165 rows)
|
|
2026-06-22 15:50:00 | INFO | ================================================================================
|
|
2026-06-22 15:50:00 | INFO | Processing Table-:additional_visibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:50:00 | INFO | Fetching Data from sql server for table-: additional_visibility ..............
|
|
2026-06-22 15:50:00 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:50:00 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:50:00 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:50:01 | INFO | Fetched 1,579 rows from SQL Server
|
|
2026-06-22 15:50:01 | INFO | Fetched total row -: 1579 from sql server for table-:additional_visibility ...........!!!
|
|
2026-06-22 15:50:01 | INFO | Fetched 1579 rows
|
|
2026-06-22 15:50:01 | INFO | _ _ _ _ Deleting Data from ClickHouse for additional_visibility _ _ _ _
|
|
2026-06-22 15:50:01 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:50:01 | INFO | additional_visibility: inserted 1,579 rows into ClickHouse
|
|
2026-06-22 15:50:01 | INFO | additional_visibility loaded successfully (1579 rows)
|
|
2026-06-22 15:50:01 | INFO | ================================================================================
|
|
2026-06-22 15:50:01 | INFO | Processing Table-:Coverage | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:50:01 | INFO | Fetching Data from sql server for table-: Coverage ..............
|
|
2026-06-22 15:50:01 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:50:01 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:50:31 | INFO | Fetched 512 rows from SQL Server
|
|
2026-06-22 15:50:31 | INFO | Fetched total row -: 512 from sql server for table-:Coverage ...........!!!
|
|
2026-06-22 15:50:31 | INFO | Fetched 512 rows
|
|
2026-06-22 15:50:33 | INFO | _ _ _ _ Deleting Data from ClickHouse for Coverage _ _ _ _
|
|
2026-06-22 15:50:33 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:50:34 | INFO | Coverage: inserted 512 rows into ClickHouse
|
|
2026-06-22 15:50:34 | INFO | Coverage loaded successfully (512 rows)
|
|
2026-06-22 15:50:34 | INFO | ================================================================================
|
|
2026-06-22 15:50:34 | INFO | Processing Table-:Login | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-22 15:50:34 | INFO | Fetching Data from sql server for table-: Login ..............
|
|
2026-06-22 15:50:34 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:50:34 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:50:37 | INFO | Fetched 454 rows from SQL Server
|
|
2026-06-22 15:50:37 | INFO | Fetched total row -: 454 from sql server for table-:Login ...........!!!
|
|
2026-06-22 15:50:37 | INFO | Fetched 454 rows
|
|
2026-06-22 15:50:38 | INFO | _ _ _ _ Deleting Data from ClickHouse for Login _ _ _ _
|
|
2026-06-22 15:50:38 | INFO | No delete logic required for Login
|
|
2026-06-22 15:50:38 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:50:38 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-22 15:50:38 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-22 15:50:38 | INFO | ================================================================================
|
|
2026-06-22 15:50:38 | INFO | Processing Table-:Stock_Details | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:50:38 | INFO | Fetching Data from sql server for table-: Stock_Details ..............
|
|
2026-06-22 15:50:38 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:50:38 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:50:46 | INFO | Fetched 27,359 rows from SQL Server
|
|
2026-06-22 15:50:46 | INFO | Fetched total row -: 27359 from sql server for table-:Stock_Details ...........!!!
|
|
2026-06-22 15:50:46 | INFO | Fetched 27359 rows
|
|
2026-06-22 15:50:46 | INFO | _ _ _ _ Deleting Data from ClickHouse for Stock_Details _ _ _ _
|
|
2026-06-22 15:50:46 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:50:47 | INFO | Stock_Details: inserted 27,359 rows into ClickHouse
|
|
2026-06-22 15:50:47 | INFO | Stock_Details loaded successfully (27359 rows)
|
|
2026-06-22 15:50:47 | INFO | ================================================================================
|
|
2026-06-22 15:50:47 | INFO | Processing Table-:Attendance | Table type -:FACT | fetcht by-:run_date | operation-:DELETE+INSERT
|
|
2026-06-22 15:50:47 | INFO | Fetching Data from sql server for table-: Attendance ..............
|
|
2026-06-22 15:50:47 | INFO | Fetching Attendance data from 2026-06-06 to 2026-06-21
|
|
2026-06-22 15:51:08 | INFO | Fetched 121,642 attendance rows for 545 employees
|
|
2026-06-22 15:51:08 | INFO | Fetched total row -: 121642 from sql server for table-:Attendance ...........!!!
|
|
2026-06-22 15:51:08 | INFO | Fetched 121642 rows
|
|
2026-06-22 15:51:08 | INFO | Truncated table Attendance
|
|
2026-06-22 15:51:08 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:51:09 | INFO | Attendance: inserted 121,642 rows into ClickHouse
|
|
2026-06-22 15:51:09 | INFO | Attendance loaded successfully (121642 rows)
|
|
2026-06-22 15:51:09 | INFO | ================================================================================
|
|
2026-06-22 15:51:09 | INFO | Processing Table-:Journey_Plan | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-22 15:51:09 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
|
|
2026-06-22 15:51:09 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:51:09 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:51:09 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:51:10 | INFO | Fetched 21,643 rows from SQL Server
|
|
2026-06-22 15:51:10 | INFO | Fetched total row -: 21643 from sql server for table-:Journey_Plan ...........!!!
|
|
2026-06-22 15:51:10 | INFO | Fetched 21643 rows
|
|
2026-06-22 15:51:11 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
|
|
2026-06-22 15:51:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:51:11 | INFO | Journey_Plan: inserted 21,643 rows into ClickHouse
|
|
2026-06-22 15:51:11 | INFO | Journey_Plan loaded successfully (21643 rows)
|
|
2026-06-22 15:51:11 | INFO | ================================================================================
|
|
2026-06-22 15:51:11 | INFO | Processing Table-:Web_Logins | Table type -:FACT | fetcht by-:run_date | operation-:INSERT
|
|
2026-06-22 15:51:11 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
|
|
2026-06-22 15:51:11 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:51:11 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:51:11 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:51:11 | INFO | Fetched 12 rows from SQL Server
|
|
2026-06-22 15:51:11 | INFO | Fetched total row -: 12 from sql server for table-:Web_Logins ...........!!!
|
|
2026-06-22 15:51:11 | INFO | Fetched 12 rows
|
|
2026-06-22 15:51:11 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
|
|
2026-06-22 15:51:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:51:11 | INFO | Web_Logins: inserted 12 rows into ClickHouse
|
|
2026-06-22 15:51:11 | INFO | Web_Logins loaded successfully (12 rows)
|
|
2026-06-22 15:51:11 | INFO | ================================================================================
|
|
2026-06-22 15:51:11 | INFO | Processing Table-:Promotion | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:51:11 | INFO | Fetching Data from sql server for table-: Promotion ..............
|
|
2026-06-22 15:51:11 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:51:14 | INFO | Fetched 4,223 rows from SQL Server
|
|
2026-06-22 15:51:14 | INFO | Fetched total row -: 4223 from sql server for table-:Promotion ...........!!!
|
|
2026-06-22 15:51:14 | INFO | Fetched 4223 rows
|
|
2026-06-22 15:51:14 | INFO | _ _ _ _ Deleting Data from ClickHouse for Promotion _ _ _ _
|
|
2026-06-22 15:51:14 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:51:15 | INFO | Promotion: inserted 4,223 rows into ClickHouse
|
|
2026-06-22 15:51:15 | INFO | Promotion loaded successfully (4223 rows)
|
|
2026-06-22 15:51:15 | INFO | ================================================================================
|
|
2026-06-22 15:51:15 | INFO | Processing Table-:PaidVisibility | Table type -:FACT | fetcht by-:mids | operation-:INSERT
|
|
2026-06-22 15:51:15 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
|
|
2026-06-22 15:51:15 | INFO | Start Fetching data for these 569 MIDs
|
|
2026-06-22 15:51:15 | INFO | Start Fetching data for these 569 MIDs or based on this date 2026-06-21
|
|
2026-06-22 15:51:15 | INFO | Fetching data for 569 MIDs
|
|
2026-06-22 15:51:16 | INFO | Fetched 1,284 rows from SQL Server
|
|
2026-06-22 15:51:16 | INFO | Fetched total row -: 1284 from sql server for table-:PaidVisibility ...........!!!
|
|
2026-06-22 15:51:16 | INFO | Fetched 1284 rows
|
|
2026-06-22 15:51:16 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _
|
|
2026-06-22 15:51:16 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:51:16 | INFO | PaidVisibility: inserted 1,284 rows into ClickHouse
|
|
2026-06-22 15:51:16 | INFO | PaidVisibility loaded successfully (1284 rows)
|
|
2026-06-22 15:51:16 | INFO | ================================================================================
|
|
2026-06-22 15:51:16 | INFO | Processing Table-:Store_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-22 15:51:16 | INFO | Fetching Data from sql server for table-: Store_Master ..............
|
|
2026-06-22 15:51:16 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:51:16 | INFO | Fetching in progress ....
|
|
2026-06-22 15:51:18 | INFO | Fetched 5,998 rows from SQL Server
|
|
2026-06-22 15:51:18 | INFO | Fetched total row -: 5998 from sql server for table-:Store_Master ...........!!!
|
|
2026-06-22 15:51:18 | INFO | Fetched 5998 rows
|
|
2026-06-22 15:51:18 | INFO | Truncated table Store_Master
|
|
2026-06-22 15:51:18 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:51:18 | INFO | Store_Master: inserted 5,998 rows into ClickHouse
|
|
2026-06-22 15:51:18 | INFO | Store_Master loaded successfully (5998 rows)
|
|
2026-06-22 15:51:18 | INFO | ================================================================================
|
|
2026-06-22 15:51:18 | INFO | Processing Table-:SKU_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-22 15:51:18 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
|
|
2026-06-22 15:51:18 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:51:18 | INFO | Fetching in progress ....
|
|
2026-06-22 15:51:19 | INFO | Fetched 160 rows from SQL Server
|
|
2026-06-22 15:51:19 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
|
|
2026-06-22 15:51:19 | INFO | Fetched 160 rows
|
|
2026-06-22 15:51:19 | INFO | Truncated table SKU_Master
|
|
2026-06-22 15:51:19 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:51:19 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-22 15:51:19 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-22 15:51:19 | INFO | ================================================================================
|
|
2026-06-22 15:51:19 | INFO | Processing Table-:display_master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-22 15:51:19 | INFO | Fetching Data from sql server for table-: display_master ..............
|
|
2026-06-22 15:51:19 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:51:19 | INFO | Fetching in progress ....
|
|
2026-06-22 15:51:19 | INFO | Fetched 135 rows from SQL Server
|
|
2026-06-22 15:51:19 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
|
|
2026-06-22 15:51:19 | INFO | Fetched 135 rows
|
|
2026-06-22 15:51:19 | INFO | Truncated table display_master
|
|
2026-06-22 15:51:19 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:51:19 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-22 15:51:19 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-22 15:51:19 | INFO | ================================================================================
|
|
2026-06-22 15:51:19 | INFO | Processing Table-:Employee_Master | Table type -:DIMENSION | fetcht by-:master | operation-:DELETE+INSERT
|
|
2026-06-22 15:51:19 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
|
|
2026-06-22 15:51:19 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:51:19 | INFO | Fetching in progress ....
|
|
2026-06-22 15:51:20 | INFO | Fetched 2,289 rows from SQL Server
|
|
2026-06-22 15:51:20 | INFO | Fetched total row -: 2289 from sql server for table-:Employee_Master ...........!!!
|
|
2026-06-22 15:51:20 | INFO | Fetched 2289 rows
|
|
2026-06-22 15:51:20 | INFO | Truncated table Employee_Master
|
|
2026-06-22 15:51:20 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:51:21 | INFO | Employee_Master: inserted 2,289 rows into ClickHouse
|
|
2026-06-22 15:51:21 | INFO | Employee_Master loaded successfully (2289 rows)
|
|
2026-06-22 15:51:21 | INFO | ================================================================================
|
|
2026-06-22 15:51:21 | INFO | Processing Table-:coverage_remarks | Table type -:DIMENSION | fetcht by-:reason_id | operation-:DELETE+INSERT
|
|
2026-06-22 15:51:21 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
|
|
2026-06-22 15:51:21 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:51:21 | INFO | Fetching in progress ....
|
|
2026-06-22 15:51:21 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:51:21 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:51:21 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
|
|
2026-06-22 15:51:21 | WARNING | coverage_remarks returned no rows
|
|
2026-06-22 15:51:21 | INFO | ================================================================================
|
|
2026-06-22 15:51:21 | INFO | Processing Table-:mapping_storevisibility | Table type -:BRIDGE | fetcht by-:run_date | operation-:ONLY_INSERT
|
|
2026-06-22 15:51:21 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
|
|
2026-06-22 15:51:22 | WARNING | Table 'mapping_storevisibility' does not exist. During collecting store_ids
|
|
2026-06-22 15:51:22 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:51:22 | INFO | Fetching in progress ....
|
|
2026-06-22 15:51:22 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:51:22 | INFO | Fetched 0 rows from SQL Server
|
|
2026-06-22 15:51:22 | INFO | Fetched total row -: 0 from sql server for table-:mapping_storevisibility ...........!!!
|
|
2026-06-22 15:51:22 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-22 15:51:22 | INFO | ================================================================================
|
|
2026-06-22 15:51:22 | INFO | Processing Table-:Master_VisibilityReason | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-22 15:51:22 | INFO | Fetching Data from sql server for table-: Master_VisibilityReason ..............
|
|
2026-06-22 15:51:22 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:51:22 | INFO | Fetching in progress ....
|
|
2026-06-22 15:51:22 | INFO | Fetched 17 rows from SQL Server
|
|
2026-06-22 15:51:22 | INFO | Fetched total row -: 17 from sql server for table-:Master_VisibilityReason ...........!!!
|
|
2026-06-22 15:51:22 | INFO | Fetched 17 rows
|
|
2026-06-22 15:51:22 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-22 15:51:22 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:51:22 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-22 15:51:22 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-22 15:51:22 | INFO | ================================================================================
|
|
2026-06-22 15:51:22 | INFO | Processing Table-:Master_VisibilityDefinition | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-22 15:51:22 | INFO | Fetching Data from sql server for table-: Master_VisibilityDefinition ..............
|
|
2026-06-22 15:51:22 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:51:22 | INFO | Fetching in progress ....
|
|
2026-06-22 15:51:23 | INFO | Fetched 861 rows from SQL Server
|
|
2026-06-22 15:51:23 | INFO | Fetched total row -: 861 from sql server for table-:Master_VisibilityDefinition ...........!!!
|
|
2026-06-22 15:51:23 | INFO | Fetched 861 rows
|
|
2026-06-22 15:51:23 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-22 15:51:23 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:51:23 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-22 15:51:23 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-22 15:51:23 | INFO | ================================================================================
|
|
2026-06-22 15:51:23 | INFO | Processing Table-:Master_Salesterritorylayer | Table type -:DIMENSION | fetcht by-:none | operation-:DELETE+INSERT
|
|
2026-06-22 15:51:23 | INFO | Fetching Data from sql server for table-: Master_Salesterritorylayer ..............
|
|
2026-06-22 15:51:23 | INFO | Fetching data from sql server for Master table......
|
|
2026-06-22 15:51:23 | INFO | Fetching in progress ....
|
|
2026-06-22 15:51:23 | INFO | Fetched 33 rows from SQL Server
|
|
2026-06-22 15:51:23 | INFO | Fetched total row -: 33 from sql server for table-:Master_Salesterritorylayer ...........!!!
|
|
2026-06-22 15:51:23 | INFO | Fetched 33 rows
|
|
2026-06-22 15:51:23 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-22 15:51:23 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
|
|
2026-06-22 15:51:23 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-22 15:51:23 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-22 15:51:23 | INFO | ================================================================================
|
|
2026-06-22 15:51:23 | INFO | Pipeline Completed Successfully
|
|
2026-06-22 15:51:23 | INFO | ================================================================================
|
|
2026-06-22 15:51:23 | INFO | Pipeline completed successfully. last_successful_run_date=2026-06-22
|