ankitm/data_pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

22-06-2026 1st commit

Browse Source
This commit is contained in:
Ankit Malik
2026-06-22 12:05:25 +05:30
parent 1ba1545ea8
commit 2ead99553f
8 changed files with 519 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
logs/etl_20260619.log
+156
View File
@@ -1293,3 +1293,159 @@ KeyError: 'fetch_Store_Master'
2026-06-19 16:00:39 | INFO | ================================================================================
2026-06-19 16:00:39 | INFO | Pipeline Completed Successfully
2026-06-19 16:00:39 | INFO | ================================================================================
2026-06-19 18:19:29 | INFO | ================================================================================
2026-06-19 18:19:29 | INFO | Hello from data-move Python data pipeline!
2026-06-19 18:19:29 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 18:19:29 | INFO | Connecting to databases...
2026-06-19 18:19:30 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000016658E305D0>
2026-06-19 18:19:31 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000016659E587D0>
2026-06-19 18:19:31 | INFO | Database connections established
2026-06-19 18:19:31 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 18:19:31 | INFO | Found 859 MIDs
2026-06-19 18:19:32 | INFO | ================================================================================
2026-06-19 18:19:32 | INFO | Processing Table: mapping_storevisibility | Table type is -: BRIDGE | Based on -run_date and operation is used -ONLY_INSERT
2026-06-19 18:19:32 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
2026-06-19 18:19:32 | ERROR | Failed processing table mapping_storevisibility
Traceback (most recent call last):
File "D:\data_move\main2.py", line 159, in main
df=fn(sql_engine, table_name , table_type, mids, run_date)
TypeError: fetch_mapping_storevisibility() takes 2 positional arguments but 5 were given
2026-06-19 18:20:42 | INFO | ================================================================================
2026-06-19 18:20:42 | INFO | Hello from data-move Python data pipeline!
2026-06-19 18:20:42 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 18:20:42 | INFO | Connecting to databases...
2026-06-19 18:20:42 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000028FE2FD05D0>
2026-06-19 18:20:43 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000028FE40787D0>
2026-06-19 18:20:44 | INFO | Database connections established
2026-06-19 18:20:44 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 18:20:44 | INFO | Found 859 MIDs
2026-06-19 18:20:44 | INFO | ================================================================================
2026-06-19 18:20:44 | INFO | Processing Table: mapping_storevisibility | Table type is -: BRIDGE | Based on -run_date and operation is used -ONLY_INSERT
2026-06-19 18:20:44 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
2026-06-19 18:20:44 | ERROR | Failed processing table mapping_storevisibility
Traceback (most recent call last):
File "D:\data_move\main2.py", line 159, in main
df=fn(sql_engine, table_name , table_type, mids, run_date)
TypeError: fetch_mapping_storevisibility() takes 2 positional arguments but 5 were given
2026-06-19 18:25:10 | INFO | ================================================================================
2026-06-19 18:25:10 | INFO | Hello from data-move Python data pipeline!
2026-06-19 18:25:10 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 18:25:10 | INFO | Connecting to databases...
2026-06-19 18:25:11 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000270F1B7C890>
2026-06-19 18:25:12 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000270F2C3C7D0>
2026-06-19 18:25:13 | INFO | Database connections established
2026-06-19 18:25:13 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 18:25:13 | INFO | Found 859 MIDs
2026-06-19 18:25:13 | INFO | ================================================================================
2026-06-19 18:25:13 | INFO | Processing Table: mapping_storevisibility | Table type is -: BRIDGE | Based on -run_date and operation is used -ONLY_INSERT
2026-06-19 18:25:13 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
2026-06-19 18:25:13 | ERROR | Failed processing table mapping_storevisibility
Traceback (most recent call last):
File "D:\data_move\main2.py", line 159, in main
df=fn(sql_engine, table_name , table_type, mids, run_date)
File "D:\data_move\src\bridge.py", line 100, in fetch_mapping_storevisibility
store_id=get_reason_ids_mapping_storevisibility(client, run_date, "coverage_remarks")
File "D:\data_move\src\bridge.py", line 61, in get_reason_ids_mapping_storevisibility
arrow_table = client.query_arrow(query)
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\client.py", line 736, in query_arrow
self.raw_query(
~~~~~~~~~~~~~~^
query,
^^^^^^
...<4 lines>...
transport_settings=transport_settings,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 661, in raw_query
return self._raw_request(body, params, fields=fields, headers=transport_settings, retries=self.query_retries).data
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
self._error_handler(response)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
raise err_type(err_str, code=code, name=name) from None
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 60, server response: Code: 60. DB::Exception: Unknown table expression identifier 'mapping_storevisibility' in scope SELECT DISTINCT StoreId FROM mapping_storevisibility WHERE (toDate(Fromdate) <= toDate('2026-06-20')) AND (toDate(Todate) >= toDate('2026-06-20')) AND (project_Id = '40148'). (UNKNOWN_TABLE) (for url http://172.188.12.194:8123)
2026-06-19 18:25:58 | INFO | ================================================================================
2026-06-19 18:25:58 | INFO | Hello from data-move Python data pipeline!
2026-06-19 18:25:58 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 18:25:58 | INFO | Connecting to databases...
2026-06-19 18:25:59 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000029019974890>
2026-06-19 18:26:00 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002901AA7C7D0>
2026-06-19 18:26:01 | INFO | Database connections established
2026-06-19 18:26:01 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 18:26:01 | INFO | Found 859 MIDs
2026-06-19 18:26:01 | INFO | ================================================================================
2026-06-19 18:26:01 | INFO | Processing Table: mapping_storevisibility | Table type is -: BRIDGE | Based on -run_date and operation is used -ONLY_INSERT
2026-06-19 18:26:01 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
2026-06-19 18:26:01 | ERROR | Failed processing table mapping_storevisibility
Traceback (most recent call last):
File "D:\data_move\main2.py", line 159, in main
df=fn(sql_engine, table_name , table_type, mids, run_date)
File "D:\data_move\src\bridge.py", line 100, in fetch_mapping_storevisibility
store_id=get_reason_ids_mapping_storevisibility(client, run_date, "coverage_remarks")
File "D:\data_move\src\bridge.py", line 61, in get_reason_ids_mapping_storevisibility
arrow_table = client.query_arrow(query)
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\client.py", line 736, in query_arrow
self.raw_query(
~~~~~~~~~~~~~~^
query,
^^^^^^
...<4 lines>...
transport_settings=transport_settings,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 661, in raw_query
return self._raw_request(body, params, fields=fields, headers=transport_settings, retries=self.query_retries).data
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
self._error_handler(response)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\main2-f1515603462f26e4\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
raise err_type(err_str, code=code, name=name) from None
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 60, server response: Code: 60. DB::Exception: Unknown table expression identifier 'mapping_storevisibility' in scope SELECT DISTINCT StoreId FROM mapping_storevisibility WHERE (toDate(Fromdate) <= toDate('2026-06-20')) AND (toDate(Todate) >= toDate('2026-06-20')) AND (project_Id = '40148'). (UNKNOWN_TABLE) (for url http://172.188.12.194:8123)
2026-06-19 18:27:24 | INFO | ================================================================================
2026-06-19 18:27:24 | INFO | Hello from data-move Python data pipeline!
2026-06-19 18:27:24 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 18:27:24 | INFO | Connecting to databases...
2026-06-19 18:27:25 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000216D0A0C890>
2026-06-19 18:27:26 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000216D1ABC7D0>
2026-06-19 18:27:26 | INFO | Database connections established
2026-06-19 18:27:26 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 18:27:27 | INFO | Found 859 MIDs
2026-06-19 18:27:27 | INFO | ================================================================================
2026-06-19 18:27:27 | INFO | Processing Table: mapping_storevisibility | Table type is -: BRIDGE | Based on -run_date and operation is used -ONLY_INSERT
2026-06-19 18:27:27 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
2026-06-19 18:27:27 | WARNING | Table 'mapping_storevisibility' does not exist. During collecting store_ids
2026-06-19 18:27:27 | INFO | Fetching data from sql server for Master table......
2026-06-19 18:27:27 | INFO | Fetching in progress ....
2026-06-19 18:27:27 | INFO | Fetched 0 rows from SQL Server
2026-06-19 18:27:27 | INFO | Fetched 0 rows from SQL Server
2026-06-19 18:27:27 | INFO | Fetched total row -: 0 from sql server for table-:mapping_storevisibility ...........!!!
2026-06-19 18:27:27 | WARNING | mapping_storevisibility returned no rows
2026-06-19 18:27:27 | INFO | ================================================================================
2026-06-19 18:27:27 | INFO | Pipeline Completed Successfully
2026-06-19 18:27:27 | INFO | ================================================================================
2026-06-19 18:27:32 | INFO | ================================================================================
2026-06-19 18:27:32 | INFO | Hello from data-move Python data pipeline!
2026-06-19 18:27:32 | INFO | Pipeline Run Date: 2026-06-18
2026-06-19 18:27:32 | INFO | Connecting to databases...
2026-06-19 18:27:33 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000261CC384730>
2026-06-19 18:27:34 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000261CD42C7D0>
2026-06-19 18:27:35 | INFO | Database connections established
2026-06-19 18:27:35 | INFO | Collecting MIDs for: 2026-06-18
2026-06-19 18:27:35 | INFO | Found 859 MIDs
2026-06-19 18:27:35 | INFO | ================================================================================
2026-06-19 18:27:35 | INFO | Processing Table: mapping_storevisibility | Table type is -: BRIDGE | Based on -run_date and operation is used -ONLY_INSERT
2026-06-19 18:27:35 | INFO | Fetching Data from sql server for table-: mapping_storevisibility ..............
2026-06-19 18:27:35 | WARNING | Table 'mapping_storevisibility' does not exist. During collecting store_ids
2026-06-19 18:27:35 | INFO | Fetching data from sql server for Master table......
2026-06-19 18:27:35 | INFO | Fetching in progress ....
2026-06-19 18:27:35 | INFO | Fetched 0 rows from SQL Server
2026-06-19 18:27:35 | INFO | Fetched 0 rows from SQL Server
2026-06-19 18:27:35 | INFO | Fetched total row -: 0 from sql server for table-:mapping_storevisibility ...........!!!
2026-06-19 18:27:35 | WARNING | mapping_storevisibility returned no rows
2026-06-19 18:27:35 | INFO | ================================================================================
2026-06-19 18:27:35 | INFO | Pipeline Completed Successfully
2026-06-19 18:27:35 | INFO | ================================================================================