ankitm/data_pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
97b22ea0f8edb31a5b54a07de1d03a85a2548eac
data_pipeline/logs/etl_20260619.log
T
Ankit Malik 97b22ea0f8 19-06-2026 1st commit
2026-06-19 16:51:03 +05:30

1296 lines
97 KiB
Plaintext
Raw Blame History

2026-06-19 10:14:16 | INFO | ================================================================================
2026-06-19 10:14:16 | INFO | Hello from data-move Python data pipeline!
2026-06-19 10:14:16 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 10:14:16 | INFO | Connecting to databases...
2026-06-19 10:14:18 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001622ED6C1B0>
2026-06-19 10:14:22 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001622FD94A50>
2026-06-19 10:14:23 | INFO | Database connections established
2026-06-19 10:14:23 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 10:14:24 | INFO | Found 859 MIDs
2026-06-19 10:14:25 | INFO | ================================================================================
2026-06-19 10:14:25 | INFO | Processing Table: Attendance | Table type is -: FACT | Based on -run_date and operation is used -DELETE+INSERT
2026-06-19 10:14:25 | INFO | Fetching Data from sql server for table-: Attendance ..............
2026-06-19 10:14:25 | ERROR | Failed processing table Attendance
Traceback (most recent call last):
File "D:\data_move\main2.py", line 181, in main
df = get_dataframe(
sql_engine,
...<5 lines>...
run_date=run_date,
)
File "D:\data_move\main2.py", line 80, in get_dataframe
return fn(sql_engine, table_name , table_type, mids, run_date)
File "D:\data_move\src\fact.py", line 550, in fetch_Attendance
sql = sql_template.format(
star_date=start_date.strftime("%Y-%m-%d"),
run_date=run_date.strftime("%Y-%m-%d")
)
KeyError: 'start_date'
2026-06-19 10:14:55 | INFO | ================================================================================
2026-06-19 10:14:55 | INFO | Hello from data-move Python data pipeline!
2026-06-19 10:14:55 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 10:14:55 | INFO | Connecting to databases...
2026-06-19 10:14:56 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001FD69918310>
2026-06-19 10:14:57 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001FD6A9907D0>
2026-06-19 10:14:58 | INFO | Database connections established
2026-06-19 10:14:58 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 10:14:58 | INFO | Found 859 MIDs
2026-06-19 10:14:58 | INFO | ================================================================================
2026-06-19 10:14:58 | INFO | Processing Table: Attendance | Table type is -: FACT | Based on -run_date and operation is used -DELETE+INSERT
2026-06-19 10:14:58 | INFO | Fetching Data from sql server for table-: Attendance ..............
2026-06-19 10:14:58 | ERROR | Failed processing table Attendance
Traceback (most recent call last):
File "D:\data_move\main2.py", line 181, in main
df = get_dataframe(
sql_engine,
...<5 lines>...
run_date=run_date,
)
File "D:\data_move\main2.py", line 80, in get_dataframe
return fn(sql_engine, table_name , table_type, mids, run_date)
File "D:\data_move\src\fact.py", line 550, in fetch_Attendance
sql = sql_template.format(
star_date=start_date.strftime("%Y-%m-%d"),
run_date=run_date.strftime("%Y-%m-%d")
)
KeyError: 'start_date'
2026-06-19 10:15:38 | INFO | ================================================================================
2026-06-19 10:15:38 | INFO | Hello from data-move Python data pipeline!
2026-06-19 10:15:38 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 10:15:38 | INFO | Connecting to databases...
2026-06-19 10:15:39 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002D95666C1B0>
2026-06-19 10:15:41 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002D957740A50>
2026-06-19 10:15:41 | INFO | Database connections established
2026-06-19 10:15:41 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 10:15:42 | INFO | Found 859 MIDs
2026-06-19 10:15:42 | INFO | ================================================================================
2026-06-19 10:15:42 | INFO | Processing Table: Attendance | Table type is -: FACT | Based on -run_date and operation is used -DELETE+INSERT
2026-06-19 10:15:42 | INFO | Fetching Data from sql server for table-: Attendance ..............
2026-06-19 10:15:42 | INFO | Fetching Attendance data from 2026-06-03 to 2026-06-18
2026-06-19 10:15:51 | INFO | Fetched 121,124 attendance rows for 540 employees
2026-06-19 10:15:51 | INFO | Fetched total row -: 121124 from sql server for table-:Attendance ...........!!!
2026-06-19 10:15:51 | INFO | Fetched 121124 rows
2026-06-19 10:15:51 | INFO | Creating table Attendance
2026-06-19 10:15:51 | INFO | Table ready: Attendance
2026-06-19 10:15:51 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 10:15:53 | INFO | Attendance: inserted 121,124 rows into ClickHouse
2026-06-19 10:15:53 | INFO | Attendance loaded successfully (121124 rows)
2026-06-19 10:15:53 | INFO | ================================================================================
2026-06-19 10:15:53 | INFO | Pipeline Completed Successfully
2026-06-19 10:15:53 | INFO | ================================================================================
2026-06-19 10:15:59 | INFO | ================================================================================
2026-06-19 10:15:59 | INFO | Hello from data-move Python data pipeline!
2026-06-19 10:15:59 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 10:15:59 | INFO | Connecting to databases...
2026-06-19 10:16:00 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000018F1AB58310>
2026-06-19 10:16:01 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000018F1BBB07D0>
2026-06-19 10:16:02 | INFO | Database connections established
2026-06-19 10:16:02 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 10:16:02 | INFO | Found 859 MIDs
2026-06-19 10:16:02 | INFO | ================================================================================
2026-06-19 10:16:02 | INFO | Processing Table: Attendance | Table type is -: FACT | Based on -run_date and operation is used -DELETE+INSERT
2026-06-19 10:16:02 | INFO | Fetching Data from sql server for table-: Attendance ..............
2026-06-19 10:16:02 | INFO | Fetching Attendance data from 2026-06-03 to 2026-06-18
2026-06-19 10:16:17 | INFO | Fetched 121,124 attendance rows for 540 employees
2026-06-19 10:16:17 | INFO | Fetched total row -: 121124 from sql server for table-:Attendance ...........!!!
2026-06-19 10:16:17 | INFO | Fetched 121124 rows
2026-06-19 10:16:18 | INFO | Truncated table Attendance
2026-06-19 10:16:18 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 10:16:19 | INFO | Attendance: inserted 121,124 rows into ClickHouse
2026-06-19 10:16:19 | INFO | Attendance loaded successfully (121124 rows)
2026-06-19 10:16:19 | INFO | ================================================================================
2026-06-19 10:16:19 | INFO | Pipeline Completed Successfully
2026-06-19 10:16:19 | INFO | ================================================================================
2026-06-19 11:17:09 | INFO | ================================================================================
2026-06-19 11:17:09 | INFO | Hello from data-move Python data pipeline!
2026-06-19 11:17:09 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 11:17:09 | INFO | Connecting to databases...
2026-06-19 11:17:11 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002925951C310>
2026-06-19 11:17:12 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002925A5CCA50>
2026-06-19 11:17:13 | INFO | Database connections established
2026-06-19 11:17:13 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 11:17:14 | INFO | Found 859 MIDs
2026-06-19 11:17:14 | INFO | ================================================================================
2026-06-19 11:17:14 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 11:17:14 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
2026-06-19 11:17:14 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 11:17:14 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 11:17:14 | INFO | Fetching data for 859 MIDs
2026-06-19 11:17:16 | INFO | Fetched 21,439 rows from SQL Server
2026-06-19 11:17:16 | INFO | Fetched total row -: 21439 from sql server for table-:Journey_Plan ...........!!!
2026-06-19 11:17:16 | INFO | Fetched 21439 rows
2026-06-19 11:17:16 | INFO | Creating table Journey_Plan
2026-06-19 11:17:16 | INFO | Table ready: Journey_Plan
2026-06-19 11:17:16 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 11:17:17 | INFO | Journey_Plan: inserted 21,439 rows into ClickHouse
2026-06-19 11:17:17 | INFO | Journey_Plan loaded successfully (21439 rows)
2026-06-19 11:17:17 | INFO | ================================================================================
2026-06-19 11:17:17 | INFO | Pipeline Completed Successfully
2026-06-19 11:17:17 | INFO | ================================================================================
2026-06-19 11:17:27 | INFO | ================================================================================
2026-06-19 11:17:27 | INFO | Hello from data-move Python data pipeline!
2026-06-19 11:17:27 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 11:17:27 | INFO | Connecting to databases...
2026-06-19 11:17:28 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000021375AAC470>
2026-06-19 11:17:30 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000021376B207D0>
2026-06-19 11:17:30 | INFO | Database connections established
2026-06-19 11:17:30 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 11:17:31 | INFO | Found 859 MIDs
2026-06-19 11:17:31 | INFO | ================================================================================
2026-06-19 11:17:31 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 11:17:31 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
2026-06-19 11:17:31 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 11:17:31 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 11:17:31 | INFO | Fetching data for 859 MIDs
2026-06-19 11:17:33 | INFO | Fetched 21,439 rows from SQL Server
2026-06-19 11:17:33 | INFO | Fetched total row -: 21439 from sql server for table-:Journey_Plan ...........!!!
2026-06-19 11:17:33 | INFO | Fetched 21439 rows
2026-06-19 11:17:33 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
2026-06-19 11:17:33 | ERROR | Failed processing table Journey_Plan
Traceback (most recent call last):
File "D:\data_move\main2.py", line 243, in main
delete_existing_data(
~~~~~~~~~~~~~~~~~~~~^
client=client,
^^^^^^^^^^^^^^
...<3 lines>...
emp_visit_df=emp_visit_df,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "D:\data_move\clickhouse_task\delete_task.py", line 91, in delete_existing_data
delete_rows(
~~~~~~~~~~~^
client,
^^^^^^^
...<4 lines>...
""",
^^^^
)
^
File "D:\data_move\clickhouse_task\delete_task.py", line 44, in delete_rows
client.command(query)
~~~~~~~~~~~~~~^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 486, in command
response = self._raw_request(payload, params, headers, method, fields=fields, server_wait=False)
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
self._error_handler(response)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
raise err_type(err_str, code=code, name=name) from None
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 43, server response: Code: 43. DB::Exception: Illegal type String of argument of function toMonth. Should be Date, Date32, DateTime or DateTime64: While processing isZeroOrNull((toMonth(visit_date) = 6) AND (toYear(visit_date) = 2026)). (ILLEGAL_TYPE_OF_ARGUMENT) (for url http://172.188.12.194:8123)
2026-06-19 11:21:40 | INFO | ================================================================================
2026-06-19 11:21:40 | INFO | Hello from data-move Python data pipeline!
2026-06-19 11:21:40 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 11:21:40 | INFO | Connecting to databases...
2026-06-19 11:21:41 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000028235FCC470>
2026-06-19 11:21:43 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000282370007D0>
2026-06-19 11:21:43 | INFO | Database connections established
2026-06-19 11:21:43 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 11:21:44 | INFO | Found 859 MIDs
2026-06-19 11:21:44 | INFO | ================================================================================
2026-06-19 11:21:44 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 11:21:44 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
2026-06-19 11:21:44 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 11:21:44 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 11:21:44 | INFO | Fetching data for 859 MIDs
2026-06-19 11:21:46 | INFO | Fetched 21,440 rows from SQL Server
2026-06-19 11:21:46 | INFO | Fetched total row -: 21440 from sql server for table-:Journey_Plan ...........!!!
2026-06-19 11:21:46 | INFO | Fetched 21440 rows
2026-06-19 11:21:46 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
2026-06-19 11:21:46 | ERROR | Failed processing table Journey_Plan
Traceback (most recent call last):
File "D:\data_move\main2.py", line 243, in main
delete_existing_data(
~~~~~~~~~~~~~~~~~~~~^
client=client,
^^^^^^^^^^^^^^
...<3 lines>...
emp_visit_df=emp_visit_df,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "D:\data_move\clickhouse_task\delete_task.py", line 91, in delete_existing_data
delete_rows(
~~~~~~~~~~~^
client,
^^^^^^^
...<5 lines>...
""",
^^^^
)
^
File "D:\data_move\clickhouse_task\delete_task.py", line 44, in delete_rows
client.command(query)
~~~~~~~~~~~~~~^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 486, in command
response = self._raw_request(payload, params, headers, method, fields=fields, server_wait=False)
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
self._error_handler(response)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
raise err_type(err_str, code=code, name=name) from None
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 47, server response: Code: 47. DB::Exception: Missing columns: 'Visit_Date' while processing: 'isZeroOrNull((project_id = 40148) AND (toMonth(Visit_Date) = toMonth(toDate('2026-06-18'))) AND (toYear(Visit_Date) = toYear(toDate('2026-06-18')))), _block_offset, _part_granule_offset, _disk_name, _sample_factor, _partition_id, _part_index, _part, UpdateDate, _block_number, CreateDate, CreateBy, _part_offset, _part_starting_offset, employee_id, _part_data_version, process_id, _database, _distance, UpdateBy, _table, _part_uuid, visit_date, store_id, project_id', required columns: 'project_id' 'Visit_Date' '_block_offset' 'UpdateBy' '_table' '_part_uuid' '_part_granule_offset' '_part_index' '_part' '_sample_factor' '_partition_id' '_disk_name' 'UpdateDate' '_block_number' 'CreateDate' 'CreateBy' '_part_data_version' '_part_starting_offset' 'employee_id' '_database' '_distance' '_part_offset' 'visit_date' 'process_id' 'store_id', maybe you meant: 'project_id', 'visit_date', 'UpdateBy', 'UpdateDate', 'CreateDate', 'CreateBy', 'employee_id', 'process_id' or 'store_id'. (UNKNOWN_IDENTIFIER) (for url http://172.188.12.194:8123)
2026-06-19 11:22:52 | INFO | ================================================================================
2026-06-19 11:22:52 | INFO | Hello from data-move Python data pipeline!
2026-06-19 11:22:52 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 11:22:52 | INFO | Connecting to databases...
2026-06-19 11:22:53 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001AD9BC0C470>
2026-06-19 11:22:55 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001AD9CC407D0>
2026-06-19 11:22:56 | INFO | Database connections established
2026-06-19 11:22:56 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 11:22:56 | INFO | Found 859 MIDs
2026-06-19 11:22:57 | INFO | ================================================================================
2026-06-19 11:22:57 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 11:22:57 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
2026-06-19 11:22:57 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 11:22:57 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 11:22:57 | INFO | Fetching data for 859 MIDs
2026-06-19 11:22:58 | INFO | Fetched 21,441 rows from SQL Server
2026-06-19 11:22:58 | INFO | Fetched total row -: 21441 from sql server for table-:Journey_Plan ...........!!!
2026-06-19 11:22:58 | INFO | Fetched 21441 rows
2026-06-19 11:22:58 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
2026-06-19 11:22:58 | ERROR | Failed processing table Journey_Plan
Traceback (most recent call last):
File "D:\data_move\main2.py", line 243, in main
delete_existing_data(
~~~~~~~~~~~~~~~~~~~~^
client=client,
^^^^^^^^^^^^^^
...<3 lines>...
emp_visit_df=emp_visit_df,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "D:\data_move\clickhouse_task\delete_task.py", line 91, in delete_existing_data
delete_rows(
~~~~~~~~~~~^
client,
^^^^^^^
...<5 lines>...
""",
^^^^
)
^
File "D:\data_move\clickhouse_task\delete_task.py", line 44, in delete_rows
client.command(query)
~~~~~~~~~~~~~~^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 486, in command
response = self._raw_request(payload, params, headers, method, fields=fields, server_wait=False)
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
self._error_handler(response)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
raise err_type(err_str, code=code, name=name) from None
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 43, server response: Code: 43. DB::Exception: Illegal type String of argument of function toMonth. Should be Date, Date32, DateTime or DateTime64: While processing isZeroOrNull((project_id = 40148) AND (toMonth(visit_date) = toMonth(toDate('2026-06-18'))) AND (toYear(visit_date) = toYear(toDate('2026-06-18')))). (ILLEGAL_TYPE_OF_ARGUMENT) (for url http://172.188.12.194:8123)
2026-06-19 11:25:44 | INFO | ================================================================================
2026-06-19 11:25:44 | INFO | Hello from data-move Python data pipeline!
2026-06-19 11:25:44 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 11:25:44 | INFO | Connecting to databases...
2026-06-19 11:25:46 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001BE6108C470>
2026-06-19 11:25:47 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001BE620C07D0>
2026-06-19 11:25:48 | INFO | Database connections established
2026-06-19 11:25:48 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 11:25:48 | INFO | Found 859 MIDs
2026-06-19 11:25:49 | INFO | ================================================================================
2026-06-19 11:25:49 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 11:25:49 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
2026-06-19 11:25:49 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 11:25:49 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 11:25:49 | INFO | Fetching data for 859 MIDs
2026-06-19 11:25:50 | INFO | Fetched 21,441 rows from SQL Server
2026-06-19 11:25:50 | INFO | Fetched total row -: 21441 from sql server for table-:Journey_Plan ...........!!!
2026-06-19 11:25:50 | INFO | Fetched 21441 rows
2026-06-19 11:25:50 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
2026-06-19 11:25:51 | ERROR | Failed processing table Journey_Plan
Traceback (most recent call last):
File "D:\data_move\main2.py", line 243, in main
delete_existing_data(
~~~~~~~~~~~~~~~~~~~~^
client=client,
^^^^^^^^^^^^^^
...<3 lines>...
emp_visit_df=emp_visit_df,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "D:\data_move\clickhouse_task\delete_task.py", line 91, in delete_existing_data
delete_rows(
~~~~~~~~~~~^
client,
^^^^^^^
...<5 lines>...
""",
^^^^
)
^
File "D:\data_move\clickhouse_task\delete_task.py", line 44, in delete_rows
client.command(query)
~~~~~~~~~~~~~~^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 486, in command
response = self._raw_request(payload, params, headers, method, fields=fields, server_wait=False)
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
self._error_handler(response)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
raise err_type(err_str, code=code, name=name) from None
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 43, server response: Code: 43. DB::Exception: Illegal type String of argument of function toMonth. Should be Date, Date32, DateTime or DateTime64: While processing isZeroOrNull((project_id = 40148) AND (toMonth(visit_date) = toMonth(toDate('2026-06-18'))) AND (toYear(visit_date) = toYear(toDate('2026-06-18')))). (ILLEGAL_TYPE_OF_ARGUMENT) (for url http://172.188.12.194:8123)
2026-06-19 11:27:56 | INFO | ================================================================================
2026-06-19 11:27:56 | INFO | Hello from data-move Python data pipeline!
2026-06-19 11:27:56 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 11:27:56 | INFO | Connecting to databases...
2026-06-19 11:27:57 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B5C8F7C470>
2026-06-19 11:27:59 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B5C9FB07D0>
2026-06-19 11:28:00 | INFO | Database connections established
2026-06-19 11:28:00 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 11:28:00 | INFO | Found 859 MIDs
2026-06-19 11:28:00 | INFO | ================================================================================
2026-06-19 11:28:00 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 11:28:00 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
2026-06-19 11:28:00 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 11:28:00 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 11:28:00 | INFO | Fetching data for 859 MIDs
2026-06-19 11:28:02 | INFO | Fetched 21,441 rows from SQL Server
2026-06-19 11:28:02 | INFO | Fetched total row -: 21441 from sql server for table-:Journey_Plan ...........!!!
2026-06-19 11:28:02 | INFO | Fetched 21441 rows
2026-06-19 11:28:02 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
2026-06-19 11:28:02 | ERROR | Failed processing table Journey_Plan
Traceback (most recent call last):
File "D:\data_move\main2.py", line 243, in main
delete_existing_data(
~~~~~~~~~~~~~~~~~~~~^
client=client,
^^^^^^^^^^^^^^
...<3 lines>...
emp_visit_df=emp_visit_df,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "D:\data_move\clickhouse_task\delete_task.py", line 91, in delete_existing_data
delete_rows(
~~~~~~~~~~~^
client,
^^^^^^^
...<5 lines>...
""",
^^^^
)
^
File "D:\data_move\clickhouse_task\delete_task.py", line 44, in delete_rows
client.command(query)
~~~~~~~~~~~~~~^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 486, in command
response = self._raw_request(payload, params, headers, method, fields=fields, server_wait=False)
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
self._error_handler(response)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
raise err_type(err_str, code=code, name=name) from None
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 43, server response: Code: 43. DB::Exception: Illegal type String of argument of function toMonth. Should be Date, Date32, DateTime or DateTime64: While processing isZeroOrNull((project_id = 40148) AND (toMonth(visit_date) = toMonth(toDate('2026-06-18'))) AND (toYear(visit_date) = toYear(toDate('2026-06-18')))). (ILLEGAL_TYPE_OF_ARGUMENT) (for url http://172.188.12.194:8123)
2026-06-19 11:28:09 | INFO | ================================================================================
2026-06-19 11:28:09 | INFO | Hello from data-move Python data pipeline!
2026-06-19 11:28:09 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 11:28:09 | INFO | Connecting to databases...
2026-06-19 11:28:10 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000028AA136C470>
2026-06-19 11:28:11 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000028AA23B07D0>
2026-06-19 11:28:13 | INFO | Database connections established
2026-06-19 11:28:13 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 11:28:13 | INFO | Found 859 MIDs
2026-06-19 11:28:13 | INFO | ================================================================================
2026-06-19 11:28:13 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 11:28:13 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
2026-06-19 11:28:13 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 11:28:13 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 11:28:13 | INFO | Fetching data for 859 MIDs
2026-06-19 11:28:14 | INFO | Fetched 21,441 rows from SQL Server
2026-06-19 11:28:14 | INFO | Fetched total row -: 21441 from sql server for table-:Journey_Plan ...........!!!
2026-06-19 11:28:14 | INFO | Fetched 21441 rows
2026-06-19 11:28:15 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
2026-06-19 11:28:15 | ERROR | Failed processing table Journey_Plan
Traceback (most recent call last):
File "D:\data_move\main2.py", line 243, in main
delete_existing_data(
~~~~~~~~~~~~~~~~~~~~^
client=client,
^^^^^^^^^^^^^^
...<3 lines>...
emp_visit_df=emp_visit_df,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "D:\data_move\clickhouse_task\delete_task.py", line 91, in delete_existing_data
delete_rows(
~~~~~~~~~~~^
client,
^^^^^^^
...<5 lines>...
""",
^^^^
)
^
File "D:\data_move\clickhouse_task\delete_task.py", line 44, in delete_rows
client.command(query)
~~~~~~~~~~~~~~^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 486, in command
response = self._raw_request(payload, params, headers, method, fields=fields, server_wait=False)
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
self._error_handler(response)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
raise err_type(err_str, code=code, name=name) from None
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 43, server response: Code: 43. DB::Exception: Illegal type String of argument of function toMonth. Should be Date, Date32, DateTime or DateTime64: While processing isZeroOrNull((project_id = 40148) AND (toMonth(visit_date) = toMonth(toDate('2026-06-18'))) AND (toYear(visit_date) = toYear(toDate('2026-06-18')))). (ILLEGAL_TYPE_OF_ARGUMENT) (for url http://172.188.12.194:8123)
2026-06-19 11:33:41 | INFO | ================================================================================
2026-06-19 11:33:41 | INFO | Hello from data-move Python data pipeline!
2026-06-19 11:33:41 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 11:33:41 | INFO | Connecting to databases...
2026-06-19 11:33:43 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000017FE98CC470>
2026-06-19 11:33:44 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000017FEA9007D0>
2026-06-19 11:33:44 | INFO | Database connections established
2026-06-19 11:33:44 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 11:33:45 | INFO | Found 859 MIDs
2026-06-19 11:33:46 | INFO | ================================================================================
2026-06-19 11:33:46 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 11:33:46 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
2026-06-19 11:33:46 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 11:33:46 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 11:33:46 | INFO | Fetching data for 859 MIDs
2026-06-19 11:33:48 | INFO | Fetched 21,442 rows from SQL Server
2026-06-19 11:33:48 | INFO | Fetched total row -: 21442 from sql server for table-:Journey_Plan ...........!!!
2026-06-19 11:33:48 | INFO | Fetched 21442 rows
2026-06-19 11:33:48 | INFO | Creating table Journey_Plan
2026-06-19 11:33:48 | INFO | Table ready: Journey_Plan
2026-06-19 11:33:48 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 11:33:49 | INFO | Journey_Plan: inserted 21,442 rows into ClickHouse
2026-06-19 11:33:49 | INFO | Journey_Plan loaded successfully (21442 rows)
2026-06-19 11:33:49 | INFO | ================================================================================
2026-06-19 11:33:49 | INFO | Pipeline Completed Successfully
2026-06-19 11:33:49 | INFO | ================================================================================
2026-06-19 11:33:53 | INFO | ================================================================================
2026-06-19 11:33:53 | INFO | Hello from data-move Python data pipeline!
2026-06-19 11:33:53 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 11:33:53 | INFO | Connecting to databases...
2026-06-19 11:33:55 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B0CAACC470>
2026-06-19 11:33:56 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B0CBB007D0>
2026-06-19 11:33:56 | INFO | Database connections established
2026-06-19 11:33:56 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 11:33:57 | INFO | Found 859 MIDs
2026-06-19 11:33:58 | INFO | ================================================================================
2026-06-19 11:33:58 | INFO | Processing Table: Journey_Plan | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 11:33:58 | INFO | Fetching Data from sql server for table-: Journey_Plan ..............
2026-06-19 11:33:58 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 11:33:58 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 11:33:58 | INFO | Fetching data for 859 MIDs
2026-06-19 11:33:59 | INFO | Fetched 21,442 rows from SQL Server
2026-06-19 11:33:59 | INFO | Fetched total row -: 21442 from sql server for table-:Journey_Plan ...........!!!
2026-06-19 11:33:59 | INFO | Fetched 21442 rows
2026-06-19 11:33:59 | INFO | _ _ _ _ Deleting Data from ClickHouse for Journey_Plan _ _ _ _
2026-06-19 11:33:59 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 11:34:00 | INFO | Journey_Plan: inserted 21,442 rows into ClickHouse
2026-06-19 11:34:00 | INFO | Journey_Plan loaded successfully (21442 rows)
2026-06-19 11:34:00 | INFO | ================================================================================
2026-06-19 11:34:00 | INFO | Pipeline Completed Successfully
2026-06-19 11:34:00 | INFO | ================================================================================
2026-06-19 12:02:14 | INFO | ================================================================================
2026-06-19 12:02:14 | INFO | Hello from data-move Python data pipeline!
2026-06-19 12:02:14 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 12:02:14 | INFO | Connecting to databases...
2026-06-19 12:02:15 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000220C84B4470>
2026-06-19 12:02:17 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000220C94D0A50>
2026-06-19 12:02:17 | INFO | Database connections established
2026-06-19 12:02:17 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 12:02:18 | INFO | Found 859 MIDs
2026-06-19 12:02:19 | INFO | ================================================================================
2026-06-19 12:02:19 | INFO | Processing Table: PaidVisibility | Table type is -: FACT | Based on -mids and operation is used -INSERT
2026-06-19 12:02:19 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
2026-06-19 12:02:19 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 12:02:19 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 12:02:19 | INFO | Fetching data for 859 MIDs
2026-06-19 12:02:23 | INFO | Fetched 937 rows from SQL Server
2026-06-19 12:02:23 | INFO | Fetched total row -: 937 from sql server for table-:PaidVisibility ...........!!!
2026-06-19 12:02:23 | INFO | Fetched 937 rows
2026-06-19 12:02:23 | INFO | Creating table PaidVisibility
2026-06-19 12:02:23 | INFO | Table ready: PaidVisibility
2026-06-19 12:02:23 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 12:02:24 | INFO | PaidVisibility: inserted 937 rows into ClickHouse
2026-06-19 12:02:24 | INFO | PaidVisibility loaded successfully (937 rows)
2026-06-19 12:02:24 | INFO | ================================================================================
2026-06-19 12:02:24 | INFO | Pipeline Completed Successfully
2026-06-19 12:02:24 | INFO | ================================================================================
2026-06-19 12:03:50 | INFO | ================================================================================
2026-06-19 12:03:50 | INFO | Hello from data-move Python data pipeline!
2026-06-19 12:03:50 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 12:03:50 | INFO | Connecting to databases...
2026-06-19 12:03:58 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000217A5FDC5D0>
2026-06-19 12:04:00 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000217A70A07D0>
2026-06-19 12:04:01 | INFO | Database connections established
2026-06-19 12:04:01 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 12:04:01 | INFO | Found 859 MIDs
2026-06-19 12:04:02 | INFO | ================================================================================
2026-06-19 12:04:02 | INFO | Processing Table: PaidVisibility | Table type is -: FACT | Based on -mids and operation is used -INSERT
2026-06-19 12:04:02 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
2026-06-19 12:04:02 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 12:04:02 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 12:04:02 | INFO | Fetching data for 859 MIDs
2026-06-19 12:04:05 | INFO | Fetched 937 rows from SQL Server
2026-06-19 12:04:05 | INFO | Fetched total row -: 937 from sql server for table-:PaidVisibility ...........!!!
2026-06-19 12:04:05 | INFO | Fetched 937 rows
2026-06-19 12:04:05 | INFO | Creating table PaidVisibility
2026-06-19 12:04:05 | INFO | Table ready: PaidVisibility
2026-06-19 12:04:05 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 12:04:05 | INFO | PaidVisibility: inserted 937 rows into ClickHouse
2026-06-19 12:04:05 | INFO | PaidVisibility loaded successfully (937 rows)
2026-06-19 12:04:05 | INFO | ================================================================================
2026-06-19 12:04:05 | INFO | Pipeline Completed Successfully
2026-06-19 12:04:05 | INFO | ================================================================================
2026-06-19 12:04:52 | INFO | ================================================================================
2026-06-19 12:04:52 | INFO | Hello from data-move Python data pipeline!
2026-06-19 12:04:52 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 12:04:52 | INFO | Connecting to databases...
2026-06-19 12:04:53 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C25EDF85D0>
2026-06-19 12:04:54 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002C25FEB07D0>
2026-06-19 12:04:55 | INFO | Database connections established
2026-06-19 12:04:55 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 12:04:56 | INFO | Found 859 MIDs
2026-06-19 12:04:57 | INFO | ================================================================================
2026-06-19 12:04:57 | INFO | Processing Table: PaidVisibility | Table type is -: FACT | Based on -mids and operation is used -INSERT
2026-06-19 12:04:57 | INFO | Fetching Data from sql server for table-: PaidVisibility ..............
2026-06-19 12:04:57 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 12:04:57 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 12:04:57 | INFO | Fetching data for 859 MIDs
2026-06-19 12:04:59 | INFO | Fetched 937 rows from SQL Server
2026-06-19 12:04:59 | INFO | Fetched total row -: 937 from sql server for table-:PaidVisibility ...........!!!
2026-06-19 12:04:59 | INFO | Fetched 937 rows
2026-06-19 12:04:59 | INFO | _ _ _ _ Deleting Data from ClickHouse for PaidVisibility _ _ _ _
2026-06-19 12:04:59 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 12:04:59 | INFO | PaidVisibility: inserted 937 rows into ClickHouse
2026-06-19 12:04:59 | INFO | PaidVisibility loaded successfully (937 rows)
2026-06-19 12:04:59 | INFO | ================================================================================
2026-06-19 12:04:59 | INFO | Pipeline Completed Successfully
2026-06-19 12:04:59 | INFO | ================================================================================
2026-06-19 12:21:12 | INFO | ================================================================================
2026-06-19 12:21:12 | INFO | Hello from data-move Python data pipeline!
2026-06-19 12:21:12 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 12:21:12 | INFO | Connecting to databases...
2026-06-19 12:21:14 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001A87473C5D0>
2026-06-19 12:21:15 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001A8758007D0>
2026-06-19 12:21:16 | INFO | Database connections established
2026-06-19 12:21:16 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 12:21:16 | INFO | Found 859 MIDs
2026-06-19 12:21:17 | INFO | ================================================================================
2026-06-19 12:21:17 | INFO | Processing Table: Web_Logins | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 12:21:17 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
2026-06-19 12:21:17 | ERROR | Failed processing table Web_Logins
Traceback (most recent call last):
File "D:\data_move\main2.py", line 181, in main
df = get_dataframe(
sql_engine,
...<5 lines>...
run_date=run_date,
)
File "D:\data_move\main2.py", line 77, in get_dataframe
fn = globals()[fn_name]
~~~~~~~~~^^^^^^^^^
KeyError: 'fetch_Web_Logins'
2026-06-19 12:22:20 | INFO | ================================================================================
2026-06-19 12:22:20 | INFO | Hello from data-move Python data pipeline!
2026-06-19 12:22:20 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 12:22:20 | INFO | Connecting to databases...
2026-06-19 12:22:22 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000026A3D5A45D0>
2026-06-19 12:22:24 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000026A3E648A50>
2026-06-19 12:22:25 | INFO | Database connections established
2026-06-19 12:22:25 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 12:22:25 | INFO | Found 859 MIDs
2026-06-19 12:22:26 | INFO | ================================================================================
2026-06-19 12:22:26 | INFO | Processing Table: Web_Logins | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 12:22:26 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
2026-06-19 12:22:26 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 12:22:26 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 12:22:26 | INFO | Fetching data for 859 MIDs
2026-06-19 12:22:27 | INFO | Fetched 136 rows from SQL Server
2026-06-19 12:22:27 | INFO | Fetched total row -: 136 from sql server for table-:Web_Logins ...........!!!
2026-06-19 12:22:27 | INFO | Fetched 136 rows
2026-06-19 12:22:27 | INFO | Creating table Web_Logins
2026-06-19 12:22:27 | INFO | Table ready: Web_Logins
2026-06-19 12:22:27 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 12:22:27 | INFO | Web_Logins: inserted 136 rows into ClickHouse
2026-06-19 12:22:27 | INFO | Web_Logins loaded successfully (136 rows)
2026-06-19 12:22:27 | INFO | ================================================================================
2026-06-19 12:22:27 | INFO | Pipeline Completed Successfully
2026-06-19 12:22:27 | INFO | ================================================================================
2026-06-19 12:26:29 | INFO | ================================================================================
2026-06-19 12:26:29 | INFO | Hello from data-move Python data pipeline!
2026-06-19 12:26:29 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 12:26:29 | INFO | Connecting to databases...
2026-06-19 12:26:31 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000025ADC8FC730>
2026-06-19 12:26:33 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000025ADD9C07D0>
2026-06-19 12:26:34 | INFO | Database connections established
2026-06-19 12:26:34 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 12:26:34 | INFO | Found 859 MIDs
2026-06-19 12:26:35 | INFO | ================================================================================
2026-06-19 12:26:35 | INFO | Processing Table: Web_Logins | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 12:26:35 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
2026-06-19 12:26:35 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 12:26:35 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 12:26:35 | INFO | Fetching data for 859 MIDs
2026-06-19 12:26:36 | INFO | Fetched 136 rows from SQL Server
2026-06-19 12:26:36 | INFO | Fetched total row -: 136 from sql server for table-:Web_Logins ...........!!!
2026-06-19 12:26:36 | INFO | Fetched 136 rows
2026-06-19 12:26:36 | INFO | Creating table Web_Logins
2026-06-19 12:26:36 | INFO | Table ready: Web_Logins
2026-06-19 12:26:36 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 12:26:36 | INFO | Web_Logins: inserted 136 rows into ClickHouse
2026-06-19 12:26:36 | INFO | Web_Logins loaded successfully (136 rows)
2026-06-19 12:26:36 | INFO | ================================================================================
2026-06-19 12:26:36 | INFO | Pipeline Completed Successfully
2026-06-19 12:26:36 | INFO | ================================================================================
2026-06-19 12:26:54 | INFO | ================================================================================
2026-06-19 12:26:54 | INFO | Hello from data-move Python data pipeline!
2026-06-19 12:26:54 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 12:26:54 | INFO | Connecting to databases...
2026-06-19 12:26:55 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000021E040D0730>
2026-06-19 12:26:56 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000021E051907D0>
2026-06-19 12:26:57 | INFO | Database connections established
2026-06-19 12:26:57 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 12:26:57 | INFO | Found 859 MIDs
2026-06-19 12:26:57 | INFO | ================================================================================
2026-06-19 12:26:57 | INFO | Processing Table: Web_Logins | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 12:26:57 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
2026-06-19 12:26:57 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 12:26:57 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 12:26:57 | INFO | Fetching data for 859 MIDs
2026-06-19 12:26:58 | INFO | Fetched 136 rows from SQL Server
2026-06-19 12:26:58 | INFO | Fetched total row -: 136 from sql server for table-:Web_Logins ...........!!!
2026-06-19 12:26:58 | INFO | Fetched 136 rows
2026-06-19 12:26:58 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
2026-06-19 12:26:58 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 12:26:58 | INFO | Web_Logins: inserted 136 rows into ClickHouse
2026-06-19 12:26:58 | INFO | Web_Logins loaded successfully (136 rows)
2026-06-19 12:26:58 | INFO | ================================================================================
2026-06-19 12:26:58 | INFO | Pipeline Completed Successfully
2026-06-19 12:26:58 | INFO | ================================================================================
2026-06-19 12:27:05 | INFO | ================================================================================
2026-06-19 12:27:05 | INFO | Hello from data-move Python data pipeline!
2026-06-19 12:27:05 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 12:27:05 | INFO | Connecting to databases...
2026-06-19 12:27:07 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000291F60CC730>
2026-06-19 12:27:09 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000291F71807D0>
2026-06-19 12:27:10 | INFO | Database connections established
2026-06-19 12:27:10 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 12:27:10 | INFO | Found 859 MIDs
2026-06-19 12:27:10 | INFO | ================================================================================
2026-06-19 12:27:10 | INFO | Processing Table: Web_Logins | Table type is -: FACT | Based on -run_date and operation is used -INSERT
2026-06-19 12:27:10 | INFO | Fetching Data from sql server for table-: Web_Logins ..............
2026-06-19 12:27:10 | INFO | Start Fetching data for these 859 MIDs
2026-06-19 12:27:10 | INFO | Start Fetching data for these 859 MIDs or based on this date 2026-06-18
2026-06-19 12:27:10 | INFO | Fetching data for 859 MIDs
2026-06-19 12:27:11 | INFO | Fetched 136 rows from SQL Server
2026-06-19 12:27:11 | INFO | Fetched total row -: 136 from sql server for table-:Web_Logins ...........!!!
2026-06-19 12:27:11 | INFO | Fetched 136 rows
2026-06-19 12:27:11 | INFO | _ _ _ _ Deleting Data from ClickHouse for Web_Logins _ _ _ _
2026-06-19 12:27:11 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 12:27:11 | INFO | Web_Logins: inserted 136 rows into ClickHouse
2026-06-19 12:27:11 | INFO | Web_Logins loaded successfully (136 rows)
2026-06-19 12:27:11 | INFO | ================================================================================
2026-06-19 12:27:11 | INFO | Pipeline Completed Successfully
2026-06-19 12:27:11 | INFO | ================================================================================
2026-06-19 12:53:21 | INFO | ================================================================================
2026-06-19 12:53:21 | INFO | Hello from data-move Python data pipeline!
2026-06-19 12:53:21 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 12:53:21 | INFO | Connecting to databases...
2026-06-19 12:53:22 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000022CD0178730>
2026-06-19 12:53:24 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000022CD12407D0>
2026-06-19 12:53:24 | INFO | Database connections established
2026-06-19 12:53:24 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 12:53:25 | INFO | Found 859 MIDs
2026-06-19 12:53:25 | INFO | ================================================================================
2026-06-19 12:53:25 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
2026-06-19 12:53:25 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 12:53:25 | ERROR | Failed processing table Store_Master
Traceback (most recent call last):
File "D:\data_move\main2.py", line 181, in main
df = get_dataframe(
sql_engine,
...<5 lines>...
run_date=run_date,
)
File "D:\data_move\main2.py", line 80, in get_dataframe
return fn(sql_engine, table_name , table_type, mids, run_date)
TypeError: fetch_Store_Master() takes 1 positional argument but 5 were given
2026-06-19 12:57:22 | INFO | ================================================================================
2026-06-19 12:57:22 | INFO | Hello from data-move Python data pipeline!
2026-06-19 12:57:22 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 12:57:22 | INFO | Connecting to databases...
2026-06-19 12:57:24 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000024BEF650310>
2026-06-19 12:57:27 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000024BF066CA50>
2026-06-19 12:57:27 | INFO | Database connections established
2026-06-19 12:57:27 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 12:57:28 | INFO | Found 859 MIDs
2026-06-19 12:57:29 | INFO | ================================================================================
2026-06-19 12:57:29 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
2026-06-19 12:57:29 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 12:57:29 | ERROR | Failed processing table Store_Master
Traceback (most recent call last):
File "D:\data_move\main2.py", line 181, in main
df = get_dataframe(
sql_engine,
...<5 lines>...
run_date=run_date,
)
File "D:\data_move\main2.py", line 77, in get_dataframe
fn = globals()[fn_name]
~~~~~~~~~^^^^^^^^^
KeyError: 'fetch_Store_Master'
2026-06-19 13:00:35 | INFO | ================================================================================
2026-06-19 13:00:35 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:00:35 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:00:35 | INFO | Connecting to databases...
2026-06-19 13:00:36 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001EF23298470>
2026-06-19 13:00:38 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001EF242C87D0>
2026-06-19 13:00:39 | INFO | Database connections established
2026-06-19 13:00:39 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:00:39 | INFO | Found 859 MIDs
2026-06-19 13:00:41 | INFO | ================================================================================
2026-06-19 13:00:41 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
2026-06-19 13:00:41 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:00:41 | ERROR | Failed processing table Store_Master
Traceback (most recent call last):
File "D:\data_move\main2.py", line 181, in main
df = get_dataframe(
sql_engine,
...<5 lines>...
run_date=run_date,
)
File "D:\data_move\main2.py", line 77, in get_dataframe
fn = globals()[fn_name]
~~~~~~~~~^^^^^^^^^
KeyError: 'fetch_Store_Master'
2026-06-19 13:01:45 | INFO | ================================================================================
2026-06-19 13:01:45 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:01:45 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:01:45 | INFO | Connecting to databases...
2026-06-19 13:01:46 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000020FE73B8470>
2026-06-19 13:01:48 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000020FE84A87D0>
2026-06-19 13:01:49 | INFO | Database connections established
2026-06-19 13:01:49 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:01:50 | INFO | Found 859 MIDs
2026-06-19 13:01:51 | INFO | ================================================================================
2026-06-19 13:01:51 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -none and operation is used -DELETE+INSERT
2026-06-19 13:01:51 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:01:51 | ERROR | Failed processing table Store_Master
Traceback (most recent call last):
File "D:\data_move\main2.py", line 181, in main
df = get_dataframe(
sql_engine,
...<5 lines>...
run_date=run_date,
)
File "D:\data_move\main2.py", line 77, in get_dataframe
fn = globals()[fn_name]
~~~~~~~~~^^^^^^^^^
KeyError: 'fetch_Store_Master'
2026-06-19 13:03:00 | INFO | ================================================================================
2026-06-19 13:03:00 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:03:00 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:03:00 | INFO | Connecting to databases...
2026-06-19 13:03:01 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000025AC24C8470>
2026-06-19 13:03:03 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000025AC34F87D0>
2026-06-19 13:03:03 | INFO | Database connections established
2026-06-19 13:03:03 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:03:04 | INFO | Found 859 MIDs
2026-06-19 13:03:05 | INFO | ================================================================================
2026-06-19 13:03:05 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:03:05 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:03:05 | ERROR | Failed processing table Store_Master
Traceback (most recent call last):
File "D:\data_move\main2.py", line 182, in main
df = get_dataframe(
sql_engine,
...<5 lines>...
run_date=run_date,
)
File "D:\data_move\main2.py", line 80, in get_dataframe
fn = globals()[fn_name]
~~~~~~~~~^^^^^^^^^
KeyError: 'fetch_Store_Master'
2026-06-19 13:03:18 | INFO | ================================================================================
2026-06-19 13:03:18 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:03:18 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:03:18 | INFO | Connecting to databases...
2026-06-19 13:03:19 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000020D83C38470>
2026-06-19 13:03:21 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000020D84C787D0>
2026-06-19 13:03:22 | INFO | Database connections established
2026-06-19 13:03:22 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:03:22 | INFO | Found 859 MIDs
2026-06-19 13:03:23 | INFO | ================================================================================
2026-06-19 13:03:23 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:03:23 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:03:23 | ERROR | Failed processing table Store_Master
Traceback (most recent call last):
File "D:\data_move\main2.py", line 182, in main
df = get_dataframe(
sql_engine,
...<5 lines>...
run_date=run_date,
)
File "D:\data_move\main2.py", line 80, in get_dataframe
fn = globals()[fn_name]
~~~~~~~~~^^^^^^^^^
KeyError: 'fetch_Store_Master'
2026-06-19 13:07:20 | INFO | ================================================================================
2026-06-19 13:07:20 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:07:20 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:07:20 | INFO | Connecting to databases...
2026-06-19 13:07:22 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000116DFD445D0>
2026-06-19 13:07:23 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000116E0DF87D0>
2026-06-19 13:07:24 | INFO | Database connections established
2026-06-19 13:07:24 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:07:25 | INFO | Found 859 MIDs
2026-06-19 13:07:26 | INFO | ================================================================================
2026-06-19 13:07:26 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:07:26 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:07:26 | ERROR | Failed processing table Store_Master
Traceback (most recent call last):
File "D:\data_move\main2.py", line 182, in main
df = get_dataframe(
sql_engine,
...<4 lines>...
run_date=run_date,
)
File "D:\data_move\main2.py", line 80, in get_dataframe
fn = globals()[fn_name]
~~~~~~~~~^^^^^^^^^
KeyError: 'fetch_Store_Master'
2026-06-19 13:13:46 | INFO | ================================================================================
2026-06-19 13:13:46 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:13:46 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:13:46 | INFO | Connecting to databases...
2026-06-19 13:13:47 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001F3330A8470>
2026-06-19 13:13:48 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001F3341487D0>
2026-06-19 13:13:49 | INFO | Database connections established
2026-06-19 13:13:49 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:13:49 | INFO | Found 859 MIDs
2026-06-19 13:13:50 | INFO | ================================================================================
2026-06-19 13:13:50 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:13:50 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:13:50 | ERROR | Failed processing table Store_Master
Traceback (most recent call last):
File "D:\data_move\main2.py", line 184, in main
fn = globals()[fn_name]
~~~~~~~~~^^^^^^^^^
KeyError: 'fetch_Store_Master'
2026-06-19 13:17:37 | INFO | ================================================================================
2026-06-19 13:17:37 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:17:37 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:17:37 | INFO | Connecting to databases...
2026-06-19 13:17:38 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000012F92B38470>
2026-06-19 13:17:39 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000012F93B747D0>
2026-06-19 13:17:40 | INFO | Database connections established
2026-06-19 13:17:40 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:17:40 | INFO | Found 859 MIDs
2026-06-19 13:17:41 | INFO | ================================================================================
2026-06-19 13:17:41 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:17:41 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:17:41 | ERROR | Failed processing table Store_Master
Traceback (most recent call last):
File "D:\data_move\main2.py", line 184, in main
fn = globals()[fn_name]
~~~~~~~~~^^^^^^^^^
KeyError: 'fetch_Store_Master'
2026-06-19 13:18:26 | INFO | ================================================================================
2026-06-19 13:18:26 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:18:26 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:18:26 | INFO | Connecting to databases...
2026-06-19 13:18:27 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000016668918470>
2026-06-19 13:18:29 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000166699347D0>
2026-06-19 13:18:29 | INFO | Database connections established
2026-06-19 13:18:29 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:18:30 | INFO | Found 859 MIDs
2026-06-19 13:18:30 | INFO | ================================================================================
2026-06-19 13:18:30 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:18:30 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:18:30 | INFO | ================================================================================
2026-06-19 13:18:30 | INFO | Pipeline Completed Successfully
2026-06-19 13:18:30 | INFO | ================================================================================
2026-06-19 13:19:45 | INFO | ================================================================================
2026-06-19 13:19:45 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:19:45 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:19:45 | INFO | Connecting to databases...
2026-06-19 13:19:46 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B0DAF48470>
2026-06-19 13:19:48 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B0DBF847D0>
2026-06-19 13:19:48 | INFO | Database connections established
2026-06-19 13:19:48 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:19:49 | INFO | Found 859 MIDs
2026-06-19 13:19:49 | INFO | ================================================================================
2026-06-19 13:19:49 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:19:49 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:19:49 | INFO | ================================================================================
2026-06-19 13:19:49 | INFO | Pipeline Completed Successfully
2026-06-19 13:19:49 | INFO | ================================================================================
2026-06-19 13:21:52 | INFO | ================================================================================
2026-06-19 13:21:52 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:21:52 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:21:52 | INFO | Connecting to databases...
2026-06-19 13:21:54 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000011AB15A8470>
2026-06-19 13:21:56 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000011AB25D47D0>
2026-06-19 13:21:56 | INFO | Database connections established
2026-06-19 13:21:56 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:21:57 | INFO | Found 859 MIDs
2026-06-19 13:21:57 | INFO | ================================================================================
2026-06-19 13:21:57 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:21:57 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:21:57 | INFO | ================================================================================
2026-06-19 13:21:57 | INFO | Pipeline Completed Successfully
2026-06-19 13:21:57 | INFO | ================================================================================
2026-06-19 13:24:49 | INFO | ================================================================================
2026-06-19 13:24:49 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:24:49 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:24:49 | INFO | Connecting to databases...
2026-06-19 13:24:50 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D9C5688470>
2026-06-19 13:24:51 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D9C66E47D0>
2026-06-19 13:24:52 | INFO | Database connections established
2026-06-19 13:24:52 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:24:52 | INFO | Found 859 MIDs
2026-06-19 13:24:53 | INFO | ================================================================================
2026-06-19 13:24:53 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:24:53 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:24:53 | INFO | ================================================================================
2026-06-19 13:24:53 | INFO | Pipeline Completed Successfully
2026-06-19 13:24:53 | INFO | ================================================================================
2026-06-19 13:26:17 | INFO | ================================================================================
2026-06-19 13:26:17 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:26:17 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:26:17 | INFO | Connecting to databases...
2026-06-19 13:26:18 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B57F448470>
2026-06-19 13:26:20 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001B5004847D0>
2026-06-19 13:26:21 | INFO | Database connections established
2026-06-19 13:26:21 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:26:21 | INFO | Found 859 MIDs
2026-06-19 13:26:21 | INFO | ================================================================================
2026-06-19 13:26:21 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:26:21 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:26:21 | INFO | ================================================================================
2026-06-19 13:26:21 | INFO | Pipeline Completed Successfully
2026-06-19 13:26:21 | INFO | ================================================================================
2026-06-19 13:27:00 | INFO | ================================================================================
2026-06-19 13:27:00 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:27:00 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:27:00 | INFO | Connecting to databases...
2026-06-19 13:27:01 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000029869D78470>
2026-06-19 13:27:03 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002986AE447D0>
2026-06-19 13:27:03 | INFO | Database connections established
2026-06-19 13:27:03 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:27:04 | INFO | Found 859 MIDs
2026-06-19 13:27:04 | INFO | ================================================================================
2026-06-19 13:27:04 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:27:04 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:27:04 | INFO | ================================================================================
2026-06-19 13:27:04 | INFO | Pipeline Completed Successfully
2026-06-19 13:27:04 | INFO | ================================================================================
2026-06-19 13:28:58 | INFO | ================================================================================
2026-06-19 13:28:58 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:28:58 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:28:58 | INFO | Connecting to databases...
2026-06-19 13:28:59 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000116347E8470>
2026-06-19 13:29:01 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000116358287D0>
2026-06-19 13:29:01 | INFO | Database connections established
2026-06-19 13:29:01 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:29:02 | INFO | Found 859 MIDs
2026-06-19 13:29:02 | INFO | ================================================================================
2026-06-19 13:29:02 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:29:02 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:29:02 | INFO | Fetching data from sql server for Master table......
2026-06-19 13:29:02 | ERROR | Failed processing table Store_Master
Traceback (most recent call last):
File "D:\data_move\main2.py", line 188, in main
df = fetch_data(sql_engine ,table_name,table_type)
File "D:\data_move\src\dim.py", line 38, in fetch_data
with open(sql_file, "r", encoding="utf-8") as f:
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'src\\sql\\dimension\\Store_Master.sql'
2026-06-19 13:29:53 | INFO | ================================================================================
2026-06-19 13:29:53 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:29:53 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:29:53 | INFO | Connecting to databases...
2026-06-19 13:29:55 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000193BF1B8470>
2026-06-19 13:29:56 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000193C01F87D0>
2026-06-19 13:29:57 | INFO | Database connections established
2026-06-19 13:29:57 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:29:57 | INFO | Found 859 MIDs
2026-06-19 13:29:57 | INFO | ================================================================================
2026-06-19 13:29:57 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:29:57 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:29:57 | INFO | Fetching data from sql server for Master table......
2026-06-19 13:29:57 | INFO | Fetching in progress ....
2026-06-19 13:30:00 | INFO | Fetched 5,991 rows from SQL Server
2026-06-19 13:30:00 | INFO | Fetched total row -: 5991 from sql server for table-:Store_Master ...........!!!
2026-06-19 13:30:00 | INFO | Fetched 5991 rows
2026-06-19 13:30:00 | INFO | Creating table Store_Master
2026-06-19 13:30:01 | INFO | Table ready: Store_Master
2026-06-19 13:30:01 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 13:30:01 | INFO | Store_Master: inserted 5,991 rows into ClickHouse
2026-06-19 13:30:01 | INFO | Store_Master loaded successfully (5991 rows)
2026-06-19 13:30:01 | INFO | ================================================================================
2026-06-19 13:30:01 | INFO | Pipeline Completed Successfully
2026-06-19 13:30:01 | INFO | ================================================================================
2026-06-19 13:30:48 | INFO | ================================================================================
2026-06-19 13:30:48 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:30:48 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:30:48 | INFO | Connecting to databases...
2026-06-19 13:30:49 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000021E0757C470>
2026-06-19 13:30:50 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000021E085A87D0>
2026-06-19 13:30:51 | INFO | Database connections established
2026-06-19 13:30:51 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:30:51 | INFO | Found 859 MIDs
2026-06-19 13:30:52 | INFO | ================================================================================
2026-06-19 13:30:52 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:30:52 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:30:52 | ERROR | Failed processing table Store_Master
Traceback (most recent call last):
File "D:\data_move\main2.py", line 181, in main
fn = globals()[fn_name]
~~~~~~~~~^^^^^^^^^
KeyError: 'fetch_Store_Master'
2026-06-19 13:32:15 | INFO | ================================================================================
2026-06-19 13:32:15 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:32:15 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:32:15 | INFO | Connecting to databases...
2026-06-19 13:32:16 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001761DCA8470>
2026-06-19 13:32:18 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001761ECE87D0>
2026-06-19 13:32:19 | INFO | Database connections established
2026-06-19 13:32:19 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:32:19 | INFO | Found 859 MIDs
2026-06-19 13:32:20 | INFO | ================================================================================
2026-06-19 13:32:20 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:32:20 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:32:20 | INFO | Fetching data from sql server for Master table......
2026-06-19 13:32:20 | INFO | Fetching in progress ....
2026-06-19 13:32:26 | INFO | Fetched 5,991 rows from SQL Server
2026-06-19 13:32:26 | INFO | Fetched total row -: 5991 from sql server for table-:Store_Master ...........!!!
2026-06-19 13:32:26 | INFO | Fetched 5991 rows
2026-06-19 13:32:26 | INFO | Truncated table Store_Master
2026-06-19 13:32:26 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 13:32:29 | INFO | Store_Master: inserted 5,991 rows into ClickHouse
2026-06-19 13:32:29 | INFO | Store_Master loaded successfully (5991 rows)
2026-06-19 13:32:29 | INFO | ================================================================================
2026-06-19 13:32:29 | INFO | Pipeline Completed Successfully
2026-06-19 13:32:29 | INFO | ================================================================================
2026-06-19 13:32:34 | INFO | ================================================================================
2026-06-19 13:32:34 | INFO | Hello from data-move Python data pipeline!
2026-06-19 13:32:34 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 13:32:34 | INFO | Connecting to databases...
2026-06-19 13:32:35 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000018D238C8470>
2026-06-19 13:32:37 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000018D249787D0>
2026-06-19 13:32:38 | INFO | Database connections established
2026-06-19 13:32:38 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 13:32:38 | INFO | Found 859 MIDs
2026-06-19 13:32:38 | INFO | ================================================================================
2026-06-19 13:32:38 | INFO | Processing Table: Store_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 13:32:38 | INFO | Fetching Data from sql server for table-: Store_Master ..............
2026-06-19 13:32:38 | INFO | Fetching data from sql server for Master table......
2026-06-19 13:32:38 | INFO | Fetching in progress ....
2026-06-19 13:32:43 | INFO | Fetched 5,991 rows from SQL Server
2026-06-19 13:32:43 | INFO | Fetched total row -: 5991 from sql server for table-:Store_Master ...........!!!
2026-06-19 13:32:43 | INFO | Fetched 5991 rows
2026-06-19 13:32:43 | INFO | Truncated table Store_Master
2026-06-19 13:32:43 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 13:32:44 | INFO | Store_Master: inserted 5,991 rows into ClickHouse
2026-06-19 13:32:44 | INFO | Store_Master loaded successfully (5991 rows)
2026-06-19 13:32:44 | INFO | ================================================================================
2026-06-19 13:32:44 | INFO | Pipeline Completed Successfully
2026-06-19 13:32:44 | INFO | ================================================================================
2026-06-19 15:28:51 | INFO | ================================================================================
2026-06-19 15:28:51 | INFO | Hello from data-move Python data pipeline!
2026-06-19 15:28:51 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 15:28:51 | INFO | Connecting to databases...
2026-06-19 15:28:53 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000023B818A45D0>
2026-06-19 15:28:54 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000023B829387D0>
2026-06-19 15:28:55 | INFO | Database connections established
2026-06-19 15:28:55 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 15:28:55 | INFO | Found 859 MIDs
2026-06-19 15:28:56 | INFO | ================================================================================
2026-06-19 15:28:56 | INFO | Processing Table: coverage_remarks | Table type is -: DIMENSION | Based on -reason_id and operation is used -DELETE+INSERT
2026-06-19 15:28:56 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
2026-06-19 15:28:57 | WARNING | Table 'coverage_remarks' does not exist.
2026-06-19 15:28:57 | INFO | Fetching data from sql server for Master table......
2026-06-19 15:28:57 | INFO | Fetching in progress ....
2026-06-19 15:28:57 | INFO | Fetched 29 rows from SQL Server
2026-06-19 15:28:57 | INFO | Fetched 29 rows from SQL Server
2026-06-19 15:28:57 | INFO | Fetched total row -: 29 from sql server for table-:coverage_remarks ...........!!!
2026-06-19 15:28:57 | INFO | Fetched 29 rows
2026-06-19 15:28:57 | INFO | Creating table coverage_remarks
2026-06-19 15:28:57 | INFO | Table ready: coverage_remarks
2026-06-19 15:28:57 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 15:28:57 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
2026-06-19 15:28:57 | INFO | coverage_remarks loaded successfully (29 rows)
2026-06-19 15:28:57 | INFO | ================================================================================
2026-06-19 15:28:57 | INFO | Pipeline Completed Successfully
2026-06-19 15:28:57 | INFO | ================================================================================
2026-06-19 15:29:04 | INFO | ================================================================================
2026-06-19 15:29:04 | INFO | Hello from data-move Python data pipeline!
2026-06-19 15:29:04 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 15:29:04 | INFO | Connecting to databases...
2026-06-19 15:29:05 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001F48D3F05D0>
2026-06-19 15:29:07 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001F48E4D87D0>
2026-06-19 15:29:07 | INFO | Database connections established
2026-06-19 15:29:07 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 15:29:08 | INFO | Found 859 MIDs
2026-06-19 15:29:08 | INFO | ================================================================================
2026-06-19 15:29:08 | INFO | Processing Table: coverage_remarks | Table type is -: DIMENSION | Based on -reason_id and operation is used -DELETE+INSERT
2026-06-19 15:29:08 | INFO | Fetching Data from sql server for table-: coverage_remarks ..............
2026-06-19 15:29:09 | INFO | Fetching data from sql server for Master table......
2026-06-19 15:29:09 | INFO | Fetching in progress ....
2026-06-19 15:29:09 | INFO | Fetched 0 rows from SQL Server
2026-06-19 15:29:09 | INFO | Fetched 0 rows from SQL Server
2026-06-19 15:29:09 | INFO | Fetched total row -: 0 from sql server for table-:coverage_remarks ...........!!!
2026-06-19 15:29:09 | WARNING | coverage_remarks returned no rows
2026-06-19 15:29:09 | INFO | ================================================================================
2026-06-19 15:29:09 | INFO | Pipeline Completed Successfully
2026-06-19 15:29:09 | INFO | ================================================================================
2026-06-19 15:51:47 | INFO | ================================================================================
2026-06-19 15:51:47 | INFO | Hello from data-move Python data pipeline!
2026-06-19 15:51:47 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 15:51:47 | INFO | Connecting to databases...
2026-06-19 15:51:49 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000148195005D0>
2026-06-19 15:51:51 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001481A6787D0>
2026-06-19 15:51:52 | INFO | Database connections established
2026-06-19 15:51:52 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 15:51:53 | INFO | Found 859 MIDs
2026-06-19 15:51:54 | INFO | ================================================================================
2026-06-19 15:51:54 | INFO | Processing Table: SKU_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 15:51:54 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
2026-06-19 15:51:54 | INFO | Fetching data from sql server for Master table......
2026-06-19 15:51:54 | INFO | Fetching in progress ....
2026-06-19 15:51:55 | INFO | Fetched 160 rows from SQL Server
2026-06-19 15:51:55 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
2026-06-19 15:51:55 | INFO | Fetched 160 rows
2026-06-19 15:51:55 | INFO | Creating table SKU_Master
2026-06-19 15:51:55 | INFO | Table ready: SKU_Master
2026-06-19 15:51:55 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 15:51:55 | INFO | SKU_Master: inserted 160 rows into ClickHouse
2026-06-19 15:51:55 | INFO | SKU_Master loaded successfully (160 rows)
2026-06-19 15:51:55 | INFO | ================================================================================
2026-06-19 15:51:55 | INFO | Pipeline Completed Successfully
2026-06-19 15:51:55 | INFO | ================================================================================
2026-06-19 15:52:22 | INFO | ================================================================================
2026-06-19 15:52:22 | INFO | Hello from data-move Python data pipeline!
2026-06-19 15:52:22 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 15:52:22 | INFO | Connecting to databases...
2026-06-19 15:52:23 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000231ECDA05D0>
2026-06-19 15:52:25 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000231EDE487D0>
2026-06-19 15:52:28 | INFO | Database connections established
2026-06-19 15:52:28 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 15:52:28 | INFO | Found 859 MIDs
2026-06-19 15:52:29 | INFO | ================================================================================
2026-06-19 15:52:29 | INFO | Processing Table: SKU_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 15:52:29 | INFO | Fetching Data from sql server for table-: SKU_Master ..............
2026-06-19 15:52:29 | INFO | Fetching data from sql server for Master table......
2026-06-19 15:52:29 | INFO | Fetching in progress ....
2026-06-19 15:52:29 | INFO | Fetched 160 rows from SQL Server
2026-06-19 15:52:29 | INFO | Fetched total row -: 160 from sql server for table-:SKU_Master ...........!!!
2026-06-19 15:52:29 | INFO | Fetched 160 rows
2026-06-19 15:52:30 | INFO | Truncated table SKU_Master
2026-06-19 15:52:30 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 15:52:30 | INFO | SKU_Master: inserted 160 rows into ClickHouse
2026-06-19 15:52:30 | INFO | SKU_Master loaded successfully (160 rows)
2026-06-19 15:52:30 | INFO | ================================================================================
2026-06-19 15:52:30 | INFO | Pipeline Completed Successfully
2026-06-19 15:52:30 | INFO | ================================================================================
2026-06-19 15:56:07 | INFO | ================================================================================
2026-06-19 15:56:07 | INFO | Hello from data-move Python data pipeline!
2026-06-19 15:56:07 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 15:56:07 | INFO | Connecting to databases...
2026-06-19 15:56:08 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002D354D3C5D0>
2026-06-19 15:56:09 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002D355E387D0>
2026-06-19 15:56:10 | INFO | Database connections established
2026-06-19 15:56:10 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 15:56:11 | INFO | Found 859 MIDs
2026-06-19 15:56:12 | INFO | ================================================================================
2026-06-19 15:56:12 | INFO | Processing Table: display_master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 15:56:12 | INFO | Fetching Data from sql server for table-: display_master ..............
2026-06-19 15:56:12 | INFO | Fetching data from sql server for Master table......
2026-06-19 15:56:12 | INFO | Fetching in progress ....
2026-06-19 15:56:12 | INFO | Fetched 135 rows from SQL Server
2026-06-19 15:56:12 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
2026-06-19 15:56:12 | INFO | Fetched 135 rows
2026-06-19 15:56:12 | INFO | Creating table display_master
2026-06-19 15:56:12 | INFO | Table ready: display_master
2026-06-19 15:56:12 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 15:56:12 | INFO | display_master: inserted 135 rows into ClickHouse
2026-06-19 15:56:12 | INFO | display_master loaded successfully (135 rows)
2026-06-19 15:56:12 | INFO | ================================================================================
2026-06-19 15:56:12 | INFO | Pipeline Completed Successfully
2026-06-19 15:56:12 | INFO | ================================================================================
2026-06-19 15:56:19 | INFO | ================================================================================
2026-06-19 15:56:19 | INFO | Hello from data-move Python data pipeline!
2026-06-19 15:56:19 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 15:56:19 | INFO | Connecting to databases...
2026-06-19 15:56:20 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001991D7005D0>
2026-06-19 15:56:22 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001991E7887D0>
2026-06-19 15:56:23 | INFO | Database connections established
2026-06-19 15:56:23 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 15:56:24 | INFO | Found 859 MIDs
2026-06-19 15:56:24 | INFO | ================================================================================
2026-06-19 15:56:24 | INFO | Processing Table: display_master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 15:56:24 | INFO | Fetching Data from sql server for table-: display_master ..............
2026-06-19 15:56:24 | INFO | Fetching data from sql server for Master table......
2026-06-19 15:56:24 | INFO | Fetching in progress ....
2026-06-19 15:56:24 | INFO | Fetched 135 rows from SQL Server
2026-06-19 15:56:24 | INFO | Fetched total row -: 135 from sql server for table-:display_master ...........!!!
2026-06-19 15:56:24 | INFO | Fetched 135 rows
2026-06-19 15:56:25 | INFO | Truncated table display_master
2026-06-19 15:56:25 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 15:56:25 | INFO | display_master: inserted 135 rows into ClickHouse
2026-06-19 15:56:25 | INFO | display_master loaded successfully (135 rows)
2026-06-19 15:56:25 | INFO | ================================================================================
2026-06-19 15:56:25 | INFO | Pipeline Completed Successfully
2026-06-19 15:56:25 | INFO | ================================================================================
2026-06-19 16:00:13 | INFO | ================================================================================
2026-06-19 16:00:13 | INFO | Hello from data-move Python data pipeline!
2026-06-19 16:00:13 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 16:00:13 | INFO | Connecting to databases...
2026-06-19 16:00:15 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000019EDD4C05D0>
2026-06-19 16:00:17 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000019EDE5687D0>
2026-06-19 16:00:18 | INFO | Database connections established
2026-06-19 16:00:18 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 16:00:19 | INFO | Found 859 MIDs
2026-06-19 16:00:20 | INFO | ================================================================================
2026-06-19 16:00:20 | INFO | Processing Table: Employee_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 16:00:20 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
2026-06-19 16:00:20 | INFO | Fetching data from sql server for Master table......
2026-06-19 16:00:20 | INFO | Fetching in progress ....
2026-06-19 16:00:23 | INFO | Fetched 2,287 rows from SQL Server
2026-06-19 16:00:23 | INFO | Fetched total row -: 2287 from sql server for table-:Employee_Master ...........!!!
2026-06-19 16:00:23 | INFO | Fetched 2287 rows
2026-06-19 16:00:23 | INFO | Creating table Employee_Master
2026-06-19 16:00:23 | INFO | Table ready: Employee_Master
2026-06-19 16:00:23 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 16:00:24 | INFO | Employee_Master: inserted 2,287 rows into ClickHouse
2026-06-19 16:00:24 | INFO | Employee_Master loaded successfully (2287 rows)
2026-06-19 16:00:24 | INFO | ================================================================================
2026-06-19 16:00:24 | INFO | Pipeline Completed Successfully
2026-06-19 16:00:24 | INFO | ================================================================================
2026-06-19 16:00:33 | INFO | ================================================================================
2026-06-19 16:00:33 | INFO | Hello from data-move Python data pipeline!
2026-06-19 16:00:33 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 16:00:33 | INFO | Connecting to databases...
2026-06-19 16:00:34 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000238D35305D0>
2026-06-19 16:00:36 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000238D46587D0>
2026-06-19 16:00:36 | INFO | Database connections established
2026-06-19 16:00:36 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 16:00:37 | INFO | Found 859 MIDs
2026-06-19 16:00:37 | INFO | ================================================================================
2026-06-19 16:00:37 | INFO | Processing Table: Employee_Master | Table type is -: DIMENSION | Based on -master and operation is used -DELETE+INSERT
2026-06-19 16:00:37 | INFO | Fetching Data from sql server for table-: Employee_Master ..............
2026-06-19 16:00:37 | INFO | Fetching data from sql server for Master table......
2026-06-19 16:00:37 | INFO | Fetching in progress ....
2026-06-19 16:00:38 | INFO | Fetched 2,287 rows from SQL Server
2026-06-19 16:00:38 | INFO | Fetched total row -: 2287 from sql server for table-:Employee_Master ...........!!!
2026-06-19 16:00:38 | INFO | Fetched 2287 rows
2026-06-19 16:00:38 | INFO | Truncated table Employee_Master
2026-06-19 16:00:38 | INFO | _ _ _ _Inserting data into clickhouse db from sql server_ _ _ _
2026-06-19 16:00:39 | INFO | Employee_Master: inserted 2,287 rows into ClickHouse
2026-06-19 16:00:39 | INFO | Employee_Master loaded successfully (2287 rows)
2026-06-19 16:00:39 | INFO | ================================================================================
2026-06-19 16:00:39 | INFO | Pipeline Completed Successfully
2026-06-19 16:00:39 | INFO | ================================================================================
Reference in New Issue View Git Blame Copy Permalink