Ankit Malik
3584 lines
310 KiB
Plaintext
3584 lines
310 KiB
Plaintext
2026-06-15 11:21:29 | INFO | ================================================================================
|
|
2026-06-15 11:21:29 | INFO | Hello from data-move Python data pipeline !
|
|
2026-06-15 11:21:29 | INFO | Pipeline Run Date : 2026-06-14
|
|
2026-06-15 11:21:29 | INFO | Connecting to SQL Server and ClickHouse
|
|
2026-06-15 11:21:31 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001F17D94BCD0>
|
|
2026-06-15 11:21:34 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001F17EB55D10>
|
|
2026-06-15 11:21:34 | INFO | Database connections established
|
|
2026-06-15 11:21:34 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 11:21:35 | INFO | Found 571 MIDs
|
|
2026-06-15 11:21:35 | INFO | Deleting data from additional_visibility
|
|
2026-06-15 11:27:46 | INFO | ================================================================================
|
|
2026-06-15 11:27:46 | INFO | Hello from data-move Python data pipeline !
|
|
2026-06-15 11:27:46 | INFO | Pipeline Run Date : 2026-06-14
|
|
2026-06-15 11:27:46 | INFO | Connecting to SQL Server and ClickHouse
|
|
2026-06-15 11:27:46 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000015FBD95BCD0>
|
|
2026-06-15 11:27:48 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000015FBEAF5D10>
|
|
2026-06-15 11:27:48 | INFO | Database connections established
|
|
2026-06-15 11:27:48 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 11:27:49 | INFO | Found 571 MIDs
|
|
2026-06-15 11:27:49 | INFO | Deleting data from additional_visibility
|
|
2026-06-15 11:27:49 | INFO | Deleting data from Coverage
|
|
2026-06-15 11:27:49 | INFO | Deleting data from Survey
|
|
2026-06-15 11:27:49 | INFO | Deleting data from Promotion
|
|
2026-06-15 11:41:04 | INFO | ================================================================================
|
|
2026-06-15 11:41:04 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 11:41:04 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 11:41:04 | INFO | Connecting to databases...
|
|
2026-06-15 11:41:05 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D06619FCD0>
|
|
2026-06-15 11:41:06 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001D06734DE50>
|
|
2026-06-15 11:41:07 | INFO | Database connections established
|
|
2026-06-15 11:41:07 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 11:41:07 | INFO | Found 571 MIDs
|
|
2026-06-15 11:41:07 | INFO | ================================================================================
|
|
2026-06-15 11:41:07 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 11:41:07 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 11:41:10 | INFO | Fetched 1,995 rows from SQL Server
|
|
2026-06-15 11:41:10 | INFO | Fetched 1995 rows
|
|
2026-06-15 11:41:10 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 11:41:10 | ERROR | Failed processing table SOS_OneApp
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\test2.py", line 232, in main
|
|
load_to_clickhouse(
|
|
~~~~~~~~~~~~~~~~~~^
|
|
client=client,
|
|
^^^^^^^^^^^^^^
|
|
table_name=table_name,
|
|
^^^^^^^^^^^^^^^^^^^^^^
|
|
df=df,
|
|
^^^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\load_table.py", line 22, in load_to_clickhouse
|
|
arrow_table = df.to_arrow()
|
|
File "d:\data_move\.venv\Lib\site-packages\polars\_utils\deprecation.py", line 128, in wrapper
|
|
return function(*args, **kwargs)
|
|
File "d:\data_move\.venv\Lib\site-packages\polars\dataframe\frame.py", line 1797, in to_arrow
|
|
record_batches = self._df.to_arrow(compat_level_py)
|
|
ModuleNotFoundError: No module named 'pyarrow'
|
|
2026-06-15 11:42:30 | INFO | ================================================================================
|
|
2026-06-15 11:42:30 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 11:42:30 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 11:42:30 | INFO | Connecting to databases...
|
|
2026-06-15 11:42:31 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000244DF4AFCD0>
|
|
2026-06-15 11:42:33 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000244E065DE50>
|
|
2026-06-15 11:42:33 | INFO | Database connections established
|
|
2026-06-15 11:42:33 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 11:42:34 | INFO | Found 571 MIDs
|
|
2026-06-15 11:42:34 | INFO | ================================================================================
|
|
2026-06-15 11:42:34 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 11:42:34 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 11:42:34 | INFO | Fetched 1,995 rows from SQL Server
|
|
2026-06-15 11:42:34 | INFO | Fetched 1995 rows
|
|
2026-06-15 11:42:35 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 11:42:35 | ERROR | Failed processing table SOS_OneApp
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\test2.py", line 233, in main
|
|
load_to_clickhouse(
|
|
~~~~~~~~~~~~~~~~~~^
|
|
client=client,
|
|
^^^^^^^^^^^^^^
|
|
table_name=table_name,
|
|
^^^^^^^^^^^^^^^^^^^^^^
|
|
df=df,
|
|
^^^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\load_table.py", line 22, in load_to_clickhouse
|
|
arrow_table = df.to_arrow()
|
|
File "d:\data_move\.venv\Lib\site-packages\polars\_utils\deprecation.py", line 128, in wrapper
|
|
return function(*args, **kwargs)
|
|
File "d:\data_move\.venv\Lib\site-packages\polars\dataframe\frame.py", line 1797, in to_arrow
|
|
record_batches = self._df.to_arrow(compat_level_py)
|
|
ModuleNotFoundError: No module named 'pyarrow'
|
|
2026-06-15 11:45:18 | INFO | ================================================================================
|
|
2026-06-15 11:45:18 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 11:45:18 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 11:45:18 | INFO | Connecting to databases...
|
|
2026-06-15 11:45:20 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001ABBE834050>
|
|
2026-06-15 11:45:24 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001ABBF88AD50>
|
|
2026-06-15 11:45:24 | INFO | Database connections established
|
|
2026-06-15 11:45:24 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 11:45:25 | INFO | Found 571 MIDs
|
|
2026-06-15 11:45:25 | INFO | ================================================================================
|
|
2026-06-15 11:45:25 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 11:45:25 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 11:45:26 | INFO | Fetched 1,995 rows from SQL Server
|
|
2026-06-15 11:45:26 | INFO | Fetched 1995 rows
|
|
2026-06-15 11:45:26 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 11:45:26 | INFO | SOS_OneApp: inserted 1,995 rows into ClickHouse
|
|
2026-06-15 11:45:26 | INFO | SOS_OneApp loaded successfully (1995 rows)
|
|
2026-06-15 11:45:26 | INFO | ================================================================================
|
|
2026-06-15 11:45:26 | INFO | Processing Table: OQaD
|
|
2026-06-15 11:45:26 | INFO | Fetching OQaD data for run_date=[1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650]
|
|
2026-06-15 11:45:27 | ERROR | Failed processing table OQaD
|
|
Traceback (most recent call last):
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1969, in _exec_single_context
|
|
self.dialect.do_execute(
|
|
~~~~~~~~~~~~~~~~~~~~~~~^
|
|
cursor, str_statement, effective_parameters, context
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\default.py", line 952, in do_execute
|
|
cursor.execute(statement, parameters)
|
|
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
|
|
pyodbc.DataError: ('22007', '[22007] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Conversion failed when converting date and/or time from character string. (241) (SQLExecDirectW)')
|
|
|
|
The above exception was the direct cause of the following exception:
|
|
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\test2.py", line 177, in main
|
|
df = get_dataframe(
|
|
fn_name=fn_name,
|
|
...<3 lines>...
|
|
run_date=run_date,
|
|
)
|
|
File "D:\data_move\test2.py", line 78, in get_dataframe
|
|
return fn(sql_engine, mids)
|
|
File "D:\data_move\kpi\facts.py", line 236, in fetch_OQaD
|
|
df = pl.read_database(
|
|
query=sql,
|
|
connection=engine,
|
|
)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\polars\io\database\functions.py", line 284, in read_database
|
|
return cx.execute(
|
|
~~~~~~~~~~^
|
|
query=query,
|
|
^^^^^^^^^^^^
|
|
options=execute_options,
|
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
|
).to_polars(
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\polars\io\database\_executor.py", line 546, in execute
|
|
result = cursor_execute(query, **options)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1421, in execute
|
|
return meth(
|
|
self,
|
|
distilled_parameters,
|
|
execution_options or NO_OPTIONS,
|
|
)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\sql\elements.py", line 526, in _execute_on_connection
|
|
return connection._execute_clauseelement(
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
|
|
self, distilled_params, execution_options
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1643, in _execute_clauseelement
|
|
ret = self._execute_context(
|
|
dialect,
|
|
...<8 lines>...
|
|
cache_hit=cache_hit,
|
|
)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1848, in _execute_context
|
|
return self._exec_single_context(
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~^
|
|
dialect, context, statement, parameters
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1988, in _exec_single_context
|
|
self._handle_dbapi_exception(
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
|
|
e, str_statement, effective_parameters, cursor, context
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 2365, in _handle_dbapi_exception
|
|
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1969, in _exec_single_context
|
|
self.dialect.do_execute(
|
|
~~~~~~~~~~~~~~~~~~~~~~~^
|
|
cursor, str_statement, effective_parameters, context
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\default.py", line 952, in do_execute
|
|
cursor.execute(statement, parameters)
|
|
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
|
|
sqlalchemy.exc.DataError: (pyodbc.DataError) ('22007', '[22007] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Conversion failed when converting date and/or time from character string. (241) (SQLExecDirectW)')
|
|
[SQL:
|
|
WITH MID_TABLE_COV1 AS
|
|
(
|
|
SELECT
|
|
EmpId,
|
|
CAST(VisitDate AS DATE) AS VisitDate
|
|
FROM OneApp_KelloggsMT.dbo.T_OQAD
|
|
WHERE CAST(CreateDate AS DATE) = '[1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650]'
|
|
|
|
UNION
|
|
|
|
SELECT
|
|
EmpId,
|
|
CAST(VisitDate AS DATE) AS VisitDate
|
|
FROM OneApp_KelloggsMT.dbo.T_OQAD
|
|
WHERE CAST(UpdateDate AS DATE) = '[1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650]'
|
|
),
|
|
|
|
QUIZ AS
|
|
(
|
|
SELECT DISTINCT
|
|
E.EmpId,
|
|
CAST(DQ.VisitDate AS DATE) AS VisitDate,
|
|
DQ.QuestionId,
|
|
DQ.AnswerId,
|
|
QC.QuestionCategoryId,
|
|
QC.QuestionCategory
|
|
|
|
FROM OneApp_KelloggsMT.dbo.T_OQAD DQ
|
|
|
|
INNER JOIN OneApp_KelloggsMT.dbo.vw_Employee_Detail E
|
|
ON DQ.EmpId = E.EmpId
|
|
|
|
INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Question QU
|
|
ON DQ.QuestionId = QU.QuestionId
|
|
|
|
INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Category QC
|
|
ON QU.QuestionCategoryId = QC.QuestionCategoryId
|
|
|
|
WHERE E.RightId = 6
|
|
AND E.EmpName NOT LIKE 'test%'
|
|
AND E.EmpName NOT LIKE '%TEST%'
|
|
AND (
|
|
E.ResignDate IS NULL
|
|
OR E.ResignDate >= '[1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650]'
|
|
)
|
|
AND EXISTS
|
|
(
|
|
SELECT 1
|
|
FROM MID_TABLE_COV1 A
|
|
WHERE A.EmpId = DQ.EmpId
|
|
AND A.VisitDate = CAST(DQ.VisitDate AS DATE)
|
|
)
|
|
)
|
|
|
|
SELECT
|
|
40148 AS project_id,
|
|
Q.EmpId AS employee_id,
|
|
0 AS process_id,
|
|
Q.VisitDate AS visit_date,
|
|
Q.QuestionCategoryId AS question_category_id,
|
|
Q.QuestionCategory AS question_category,
|
|
QM.QuestionId AS question_id,
|
|
QM.Question AS question,
|
|
ISNULL(QA.AnswerId, 0) AS answer_id,
|
|
ISNULL(QA.Answer, '') AS answer,
|
|
|
|
CASE
|
|
WHEN QA.AnswerId IS NULL THEN 'Not Answer'
|
|
WHEN QA.RightAnswer = 1 THEN 'Y'
|
|
WHEN QA.RightAnswer IS NULL THEN 'Not Answer'
|
|
ELSE 'N'
|
|
END AS correct_answer,
|
|
|
|
GETDATE() AS update_date,
|
|
'SP-Pius' AS update_by
|
|
|
|
FROM QUIZ Q
|
|
|
|
INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Question QM
|
|
ON Q.QuestionId = QM.QuestionId
|
|
|
|
LEFT JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Answer QA
|
|
ON Q.AnswerId = QA.AnswerId
|
|
]
|
|
(Background on this error at: https://sqlalche.me/e/20/9h9h)
|
|
2026-06-15 11:47:45 | INFO | ================================================================================
|
|
2026-06-15 11:47:45 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 11:47:45 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 11:47:45 | INFO | Connecting to databases...
|
|
2026-06-15 11:47:46 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000213C80675F0>
|
|
2026-06-15 11:47:48 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000213C9239450>
|
|
2026-06-15 11:47:48 | INFO | Database connections established
|
|
2026-06-15 11:47:48 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 11:47:49 | INFO | Found 571 MIDs
|
|
2026-06-15 11:47:49 | INFO | ================================================================================
|
|
2026-06-15 11:47:49 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 11:47:49 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 11:47:49 | INFO | Fetched 1,995 rows from SQL Server
|
|
2026-06-15 11:47:49 | INFO | Fetched 1995 rows
|
|
2026-06-15 11:47:50 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 11:47:50 | INFO | SOS_OneApp: inserted 1,995 rows into ClickHouse
|
|
2026-06-15 11:47:50 | INFO | SOS_OneApp loaded successfully (1995 rows)
|
|
2026-06-15 11:47:50 | INFO | ================================================================================
|
|
2026-06-15 11:47:50 | INFO | Processing Table: OQaD
|
|
2026-06-15 11:47:50 | INFO | Fetching OQaD data for run_date=[1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650]
|
|
2026-06-15 11:47:50 | ERROR | Failed processing table OQaD
|
|
Traceback (most recent call last):
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1969, in _exec_single_context
|
|
self.dialect.do_execute(
|
|
~~~~~~~~~~~~~~~~~~~~~~~^
|
|
cursor, str_statement, effective_parameters, context
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\default.py", line 952, in do_execute
|
|
cursor.execute(statement, parameters)
|
|
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
|
|
pyodbc.DataError: ('22007', '[22007] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Conversion failed when converting date and/or time from character string. (241) (SQLExecDirectW)')
|
|
|
|
The above exception was the direct cause of the following exception:
|
|
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\test2.py", line 177, in main
|
|
df = get_dataframe(
|
|
fn_name=fn_name,
|
|
...<3 lines>...
|
|
run_date=run_date,
|
|
)
|
|
File "D:\data_move\test2.py", line 78, in get_dataframe
|
|
return fn(sql_engine, mids)
|
|
File "D:\data_move\kpi\facts.py", line 236, in fetch_OQaD
|
|
df = pl.read_database(
|
|
query=sql,
|
|
connection=engine,
|
|
)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\polars\io\database\functions.py", line 284, in read_database
|
|
return cx.execute(
|
|
~~~~~~~~~~^
|
|
query=query,
|
|
^^^^^^^^^^^^
|
|
options=execute_options,
|
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
|
).to_polars(
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\polars\io\database\_executor.py", line 546, in execute
|
|
result = cursor_execute(query, **options)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1421, in execute
|
|
return meth(
|
|
self,
|
|
distilled_parameters,
|
|
execution_options or NO_OPTIONS,
|
|
)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\sql\elements.py", line 526, in _execute_on_connection
|
|
return connection._execute_clauseelement(
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
|
|
self, distilled_params, execution_options
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1643, in _execute_clauseelement
|
|
ret = self._execute_context(
|
|
dialect,
|
|
...<8 lines>...
|
|
cache_hit=cache_hit,
|
|
)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1848, in _execute_context
|
|
return self._exec_single_context(
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~^
|
|
dialect, context, statement, parameters
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1988, in _exec_single_context
|
|
self._handle_dbapi_exception(
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
|
|
e, str_statement, effective_parameters, cursor, context
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 2365, in _handle_dbapi_exception
|
|
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1969, in _exec_single_context
|
|
self.dialect.do_execute(
|
|
~~~~~~~~~~~~~~~~~~~~~~~^
|
|
cursor, str_statement, effective_parameters, context
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\default.py", line 952, in do_execute
|
|
cursor.execute(statement, parameters)
|
|
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
|
|
sqlalchemy.exc.DataError: (pyodbc.DataError) ('22007', '[22007] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Conversion failed when converting date and/or time from character string. (241) (SQLExecDirectW)')
|
|
[SQL:
|
|
WITH MID_TABLE_COV1 AS
|
|
(
|
|
SELECT
|
|
EmpId,
|
|
CAST(VisitDate AS DATE) AS VisitDate
|
|
FROM OneApp_KelloggsMT.dbo.T_OQAD
|
|
WHERE CAST(CreateDate AS DATE) = '[1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650]'
|
|
|
|
UNION
|
|
|
|
SELECT
|
|
EmpId,
|
|
CAST(VisitDate AS DATE) AS VisitDate
|
|
FROM OneApp_KelloggsMT.dbo.T_OQAD
|
|
WHERE CAST(UpdateDate AS DATE) = '[1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650]'
|
|
),
|
|
|
|
QUIZ AS
|
|
(
|
|
SELECT DISTINCT
|
|
E.EmpId,
|
|
CAST(DQ.VisitDate AS DATE) AS VisitDate,
|
|
DQ.QuestionId,
|
|
DQ.AnswerId,
|
|
QC.QuestionCategoryId,
|
|
QC.QuestionCategory
|
|
|
|
FROM OneApp_KelloggsMT.dbo.T_OQAD DQ
|
|
|
|
INNER JOIN OneApp_KelloggsMT.dbo.vw_Employee_Detail E
|
|
ON DQ.EmpId = E.EmpId
|
|
|
|
INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Question QU
|
|
ON DQ.QuestionId = QU.QuestionId
|
|
|
|
INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Category QC
|
|
ON QU.QuestionCategoryId = QC.QuestionCategoryId
|
|
|
|
WHERE E.RightId = 6
|
|
AND E.EmpName NOT LIKE 'test%'
|
|
AND E.EmpName NOT LIKE '%TEST%'
|
|
AND (
|
|
E.ResignDate IS NULL
|
|
OR E.ResignDate >= '[1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650]'
|
|
)
|
|
AND EXISTS
|
|
(
|
|
SELECT 1
|
|
FROM MID_TABLE_COV1 A
|
|
WHERE A.EmpId = DQ.EmpId
|
|
AND A.VisitDate = CAST(DQ.VisitDate AS DATE)
|
|
)
|
|
)
|
|
|
|
SELECT
|
|
40148 AS project_id,
|
|
Q.EmpId AS employee_id,
|
|
0 AS process_id,
|
|
Q.VisitDate AS visit_date,
|
|
Q.QuestionCategoryId AS question_category_id,
|
|
Q.QuestionCategory AS question_category,
|
|
QM.QuestionId AS question_id,
|
|
QM.Question AS question,
|
|
ISNULL(QA.AnswerId, 0) AS answer_id,
|
|
ISNULL(QA.Answer, '') AS answer,
|
|
|
|
CASE
|
|
WHEN QA.AnswerId IS NULL THEN 'Not Answer'
|
|
WHEN QA.RightAnswer = 1 THEN 'Y'
|
|
WHEN QA.RightAnswer IS NULL THEN 'Not Answer'
|
|
ELSE 'N'
|
|
END AS correct_answer,
|
|
|
|
GETDATE() AS update_date,
|
|
'SP-Pius' AS update_by
|
|
|
|
FROM QUIZ Q
|
|
|
|
INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Question QM
|
|
ON Q.QuestionId = QM.QuestionId
|
|
|
|
LEFT JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Answer QA
|
|
ON Q.AnswerId = QA.AnswerId
|
|
]
|
|
(Background on this error at: https://sqlalche.me/e/20/9h9h)
|
|
2026-06-15 11:49:49 | INFO | ================================================================================
|
|
2026-06-15 11:49:49 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 11:49:49 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 11:49:49 | INFO | Connecting to databases...
|
|
2026-06-15 11:49:50 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000020C1B1775F0>
|
|
2026-06-15 11:49:51 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000020C1C339450>
|
|
2026-06-15 11:49:52 | INFO | Database connections established
|
|
2026-06-15 11:49:52 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 11:49:52 | INFO | Found 571 MIDs
|
|
2026-06-15 11:49:52 | INFO | ================================================================================
|
|
2026-06-15 11:49:52 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 11:49:52 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 11:49:52 | INFO | Fetched 1,995 rows from SQL Server
|
|
2026-06-15 11:49:52 | INFO | Fetched 1995 rows
|
|
2026-06-15 11:49:52 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 11:49:53 | INFO | SOS_OneApp: inserted 1,995 rows into ClickHouse
|
|
2026-06-15 11:49:53 | INFO | SOS_OneApp loaded successfully (1995 rows)
|
|
2026-06-15 11:49:53 | INFO | ================================================================================
|
|
2026-06-15 11:49:53 | INFO | Processing Table: OQaD
|
|
2026-06-15 11:49:53 | INFO | Fetching OQaD data for run_date=[1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650]
|
|
2026-06-15 11:49:53 | ERROR | Failed processing table OQaD
|
|
Traceback (most recent call last):
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1969, in _exec_single_context
|
|
self.dialect.do_execute(
|
|
~~~~~~~~~~~~~~~~~~~~~~~^
|
|
cursor, str_statement, effective_parameters, context
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\default.py", line 952, in do_execute
|
|
cursor.execute(statement, parameters)
|
|
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
|
|
pyodbc.DataError: ('22007', '[22007] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Conversion failed when converting date and/or time from character string. (241) (SQLExecDirectW)')
|
|
|
|
The above exception was the direct cause of the following exception:
|
|
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\test2.py", line 177, in main
|
|
df = get_dataframe(
|
|
fn_name=fn_name,
|
|
...<3 lines>...
|
|
run_date=run_date,
|
|
)
|
|
File "D:\data_move\test2.py", line 78, in get_dataframe
|
|
return fn(sql_engine, mids)
|
|
File "D:\data_move\kpi\facts.py", line 236, in fetch_OQaD
|
|
df = pl.read_database(
|
|
query=sql,
|
|
connection=engine,
|
|
)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\polars\io\database\functions.py", line 284, in read_database
|
|
return cx.execute(
|
|
~~~~~~~~~~^
|
|
query=query,
|
|
^^^^^^^^^^^^
|
|
options=execute_options,
|
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
|
).to_polars(
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\polars\io\database\_executor.py", line 546, in execute
|
|
result = cursor_execute(query, **options)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1421, in execute
|
|
return meth(
|
|
self,
|
|
distilled_parameters,
|
|
execution_options or NO_OPTIONS,
|
|
)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\sql\elements.py", line 526, in _execute_on_connection
|
|
return connection._execute_clauseelement(
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
|
|
self, distilled_params, execution_options
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1643, in _execute_clauseelement
|
|
ret = self._execute_context(
|
|
dialect,
|
|
...<8 lines>...
|
|
cache_hit=cache_hit,
|
|
)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1848, in _execute_context
|
|
return self._exec_single_context(
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~^
|
|
dialect, context, statement, parameters
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1988, in _exec_single_context
|
|
self._handle_dbapi_exception(
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
|
|
e, str_statement, effective_parameters, cursor, context
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 2365, in _handle_dbapi_exception
|
|
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\base.py", line 1969, in _exec_single_context
|
|
self.dialect.do_execute(
|
|
~~~~~~~~~~~~~~~~~~~~~~~^
|
|
cursor, str_statement, effective_parameters, context
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\sqlalchemy\engine\default.py", line 952, in do_execute
|
|
cursor.execute(statement, parameters)
|
|
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
|
|
sqlalchemy.exc.DataError: (pyodbc.DataError) ('22007', '[22007] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Conversion failed when converting date and/or time from character string. (241) (SQLExecDirectW)')
|
|
[SQL:
|
|
WITH MID_TABLE_COV1 AS
|
|
(
|
|
SELECT
|
|
EmpId,
|
|
CAST(VisitDate AS DATE) AS VisitDate
|
|
FROM OneApp_KelloggsMT.dbo.T_OQAD
|
|
WHERE CAST(CreateDate AS DATE) = '[1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650]'
|
|
|
|
UNION
|
|
|
|
SELECT
|
|
EmpId,
|
|
CAST(VisitDate AS DATE) AS VisitDate
|
|
FROM OneApp_KelloggsMT.dbo.T_OQAD
|
|
WHERE CAST(UpdateDate AS DATE) = '[1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650]'
|
|
),
|
|
|
|
QUIZ AS
|
|
(
|
|
SELECT DISTINCT
|
|
E.EmpId,
|
|
CAST(DQ.VisitDate AS DATE) AS VisitDate,
|
|
DQ.QuestionId,
|
|
DQ.AnswerId,
|
|
QC.QuestionCategoryId,
|
|
QC.QuestionCategory
|
|
|
|
FROM OneApp_KelloggsMT.dbo.T_OQAD DQ
|
|
|
|
INNER JOIN OneApp_KelloggsMT.dbo.vw_Employee_Detail E
|
|
ON DQ.EmpId = E.EmpId
|
|
|
|
INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Question QU
|
|
ON DQ.QuestionId = QU.QuestionId
|
|
|
|
INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Category QC
|
|
ON QU.QuestionCategoryId = QC.QuestionCategoryId
|
|
|
|
WHERE E.RightId = 6
|
|
AND E.EmpName NOT LIKE 'test%'
|
|
AND E.EmpName NOT LIKE '%TEST%'
|
|
AND (
|
|
E.ResignDate IS NULL
|
|
OR E.ResignDate >= '[1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650]'
|
|
)
|
|
AND EXISTS
|
|
(
|
|
SELECT 1
|
|
FROM MID_TABLE_COV1 A
|
|
WHERE A.EmpId = DQ.EmpId
|
|
AND A.VisitDate = CAST(DQ.VisitDate AS DATE)
|
|
)
|
|
)
|
|
|
|
SELECT
|
|
40148 AS project_id,
|
|
Q.EmpId AS employee_id,
|
|
0 AS process_id,
|
|
Q.VisitDate AS visit_date,
|
|
Q.QuestionCategoryId AS question_category_id,
|
|
Q.QuestionCategory AS question_category,
|
|
QM.QuestionId AS question_id,
|
|
QM.Question AS question,
|
|
ISNULL(QA.AnswerId, 0) AS answer_id,
|
|
ISNULL(QA.Answer, '') AS answer,
|
|
|
|
CASE
|
|
WHEN QA.AnswerId IS NULL THEN 'Not Answer'
|
|
WHEN QA.RightAnswer = 1 THEN 'Y'
|
|
WHEN QA.RightAnswer IS NULL THEN 'Not Answer'
|
|
ELSE 'N'
|
|
END AS correct_answer,
|
|
|
|
GETDATE() AS update_date,
|
|
'SP-Pius' AS update_by
|
|
|
|
FROM QUIZ Q
|
|
|
|
INNER JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Question QM
|
|
ON Q.QuestionId = QM.QuestionId
|
|
|
|
LEFT JOIN OneApp_KelloggsMT.dbo.Master_OQAD_Answer QA
|
|
ON Q.AnswerId = QA.AnswerId
|
|
]
|
|
(Background on this error at: https://sqlalche.me/e/20/9h9h)
|
|
2026-06-15 11:51:58 | INFO | ================================================================================
|
|
2026-06-15 11:51:58 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 11:51:58 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 11:51:58 | INFO | Connecting to databases...
|
|
2026-06-15 11:51:59 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001CA338675F0>
|
|
2026-06-15 11:52:01 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001CA34A39450>
|
|
2026-06-15 11:52:01 | INFO | Database connections established
|
|
2026-06-15 11:52:01 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 11:52:01 | INFO | Found 571 MIDs
|
|
2026-06-15 11:52:02 | INFO | ================================================================================
|
|
2026-06-15 11:52:02 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 11:52:02 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 11:52:02 | INFO | Fetched 1,995 rows from SQL Server
|
|
2026-06-15 11:52:02 | INFO | Fetched 1995 rows
|
|
2026-06-15 11:52:02 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 11:52:02 | INFO | SOS_OneApp: inserted 1,995 rows into ClickHouse
|
|
2026-06-15 11:52:02 | INFO | SOS_OneApp loaded successfully (1995 rows)
|
|
2026-06-15 11:52:02 | INFO | ================================================================================
|
|
2026-06-15 11:52:02 | INFO | Processing Table: OQaD
|
|
2026-06-15 11:52:02 | INFO | Fetching OQaD data for run_date=2026-06-14
|
|
2026-06-15 11:52:06 | INFO | Fetched 443 rows
|
|
2026-06-15 11:52:06 | INFO | Fetched 443 rows
|
|
2026-06-15 11:52:07 | INFO | Deleting from OQaD
|
|
2026-06-15 11:52:07 | ERROR | Failed processing table OQaD
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\test2.py", line 248, in main
|
|
load_to_clickhouse(
|
|
~~~~~~~~~~~~~~~~~~^
|
|
client=client,
|
|
^^^^^^^^^^^^^^
|
|
table_name=table_name,
|
|
^^^^^^^^^^^^^^^^^^^^^^
|
|
df=df,
|
|
^^^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\load_table.py", line 25, in load_to_clickhouse
|
|
client.insert_arrow(
|
|
~~~~~~~~~~~~~~~~~~~^
|
|
table=table_name,
|
|
^^^^^^^^^^^^^^^^^
|
|
arrow_table=arrow_table,
|
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\client.py", line 1054, in insert_arrow
|
|
return self.raw_insert(full_table, column_names, insert_block, settings, "Arrow", transport_settings)
|
|
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 432, in raw_insert
|
|
response = self._raw_request(insert_block, params, headers, server_wait=False)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
|
|
self._error_handler(response)
|
|
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
|
|
raise err_type(err_str, code=code, name=name) from None
|
|
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 16, server response: Code: 16. DB::Exception: No such column project_id in table kelloggs.OQaD (baea0f0f-5216-49dd-b156-fa06fe3e62f1). (NO_SUCH_COLUMN_IN_TABLE) (for url http://172.188.12.194:8123)
|
|
2026-06-15 11:55:57 | INFO | ================================================================================
|
|
2026-06-15 11:55:57 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 11:55:57 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 11:55:57 | INFO | Connecting to databases...
|
|
2026-06-15 11:55:58 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000021E578475F0>
|
|
2026-06-15 11:56:00 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000021E589D9450>
|
|
2026-06-15 11:56:00 | INFO | Database connections established
|
|
2026-06-15 11:56:00 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 11:56:00 | INFO | Found 571 MIDs
|
|
2026-06-15 11:56:01 | INFO | ================================================================================
|
|
2026-06-15 11:56:01 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 11:56:01 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 11:56:01 | INFO | Fetched 1,995 rows from SQL Server
|
|
2026-06-15 11:56:01 | INFO | Fetched 1995 rows
|
|
2026-06-15 11:56:01 | INFO | Creating table SOS_OneApp
|
|
2026-06-15 11:56:01 | INFO | Table ready: SOS_OneApp
|
|
2026-06-15 11:56:02 | INFO | SOS_OneApp: inserted 1,995 rows into ClickHouse
|
|
2026-06-15 11:56:02 | INFO | SOS_OneApp loaded successfully (1995 rows)
|
|
2026-06-15 11:56:02 | INFO | ================================================================================
|
|
2026-06-15 11:56:02 | INFO | Processing Table: OQaD
|
|
2026-06-15 11:56:02 | INFO | Fetching OQaD data for run_date=2026-06-14
|
|
2026-06-15 11:56:04 | INFO | Fetched 443 rows
|
|
2026-06-15 11:56:04 | INFO | Fetched 443 rows
|
|
2026-06-15 11:56:04 | INFO | Creating table OQaD
|
|
2026-06-15 11:56:04 | INFO | Table ready: OQaD
|
|
2026-06-15 11:56:04 | INFO | OQaD: inserted 443 rows into ClickHouse
|
|
2026-06-15 11:56:04 | INFO | OQaD loaded successfully (443 rows)
|
|
2026-06-15 11:56:04 | INFO | ================================================================================
|
|
2026-06-15 11:56:04 | INFO | Processing Table: Survey
|
|
2026-06-15 11:56:04 | INFO | Fetching Survey data for 571 MIDs
|
|
2026-06-15 11:56:07 | INFO | Fetched 173 Survey rows
|
|
2026-06-15 11:56:07 | INFO | Fetched 173 rows
|
|
2026-06-15 11:56:07 | INFO | Creating table Survey
|
|
2026-06-15 11:56:07 | INFO | Table ready: Survey
|
|
2026-06-15 11:56:07 | INFO | Survey: inserted 173 rows into ClickHouse
|
|
2026-06-15 11:56:07 | INFO | Survey loaded successfully (173 rows)
|
|
2026-06-15 11:56:07 | INFO | ================================================================================
|
|
2026-06-15 11:56:07 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 11:56:07 | INFO | Fetching Additional Visibility data for 571 MIDs
|
|
2026-06-15 11:56:17 | INFO | Fetched 1,648 Additional Visibility rows
|
|
2026-06-15 11:56:17 | INFO | Fetched 1648 rows
|
|
2026-06-15 11:56:17 | INFO | Creating table additional_visibility
|
|
2026-06-15 11:56:18 | INFO | Table ready: additional_visibility
|
|
2026-06-15 11:56:18 | INFO | additional_visibility: inserted 1,648 rows into ClickHouse
|
|
2026-06-15 11:56:18 | INFO | additional_visibility loaded successfully (1648 rows)
|
|
2026-06-15 11:56:18 | INFO | ================================================================================
|
|
2026-06-15 11:56:18 | INFO | Processing Table: Coverage
|
|
2026-06-15 11:56:18 | INFO | Fetching coverage data for 571 MIDs
|
|
2026-06-15 11:56:20 | INFO | Fetched 514 rows from SQL Server
|
|
2026-06-15 11:56:20 | INFO | Fetched 514 rows
|
|
2026-06-15 11:56:20 | INFO | Creating table Coverage
|
|
2026-06-15 11:56:20 | INFO | Table ready: Coverage
|
|
2026-06-15 11:56:20 | INFO | Coverage: inserted 514 rows into ClickHouse
|
|
2026-06-15 11:56:20 | INFO | Coverage loaded successfully (514 rows)
|
|
2026-06-15 11:56:20 | INFO | ================================================================================
|
|
2026-06-15 11:56:20 | INFO | Processing Table: Login
|
|
2026-06-15 11:56:20 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 11:56:22 | INFO | Fetched 454 Login rows
|
|
2026-06-15 11:56:22 | INFO | Fetched 454 rows
|
|
2026-06-15 11:56:22 | INFO | Creating table Login
|
|
2026-06-15 11:56:22 | INFO | Table ready: Login
|
|
2026-06-15 11:56:23 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 11:56:23 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 11:56:23 | INFO | ================================================================================
|
|
2026-06-15 11:56:23 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 11:56:23 | INFO | Fetching Stock Details data for 571 MIDs
|
|
2026-06-15 11:56:30 | INFO | Fetched 28,493 Stock Details rows
|
|
2026-06-15 11:56:30 | INFO | Fetched 28493 rows
|
|
2026-06-15 11:56:31 | INFO | Creating table Stock_Details
|
|
2026-06-15 11:56:31 | INFO | Table ready: Stock_Details
|
|
2026-06-15 11:56:31 | INFO | Stock_Details: inserted 28,493 rows into ClickHouse
|
|
2026-06-15 11:56:31 | INFO | Stock_Details loaded successfully (28493 rows)
|
|
2026-06-15 11:56:31 | INFO | ================================================================================
|
|
2026-06-15 11:56:31 | INFO | Processing Table: Attendance
|
|
2026-06-15 11:56:31 | INFO | Fetching Attendance data from 2026-05-30 to 2026-06-14
|
|
2026-06-15 11:56:46 | INFO | Fetched 11,543 attendance rows for 594 employees
|
|
2026-06-15 11:56:46 | INFO | Fetched 11543 rows
|
|
2026-06-15 11:56:47 | INFO | Creating table Attendance
|
|
2026-06-15 11:56:47 | INFO | Table ready: Attendance
|
|
2026-06-15 11:56:47 | INFO | Attendance: inserted 11,543 rows into ClickHouse
|
|
2026-06-15 11:56:47 | INFO | Attendance loaded successfully (11543 rows)
|
|
2026-06-15 11:56:47 | INFO | ================================================================================
|
|
2026-06-15 11:56:47 | INFO | Processing Table: Store_Master
|
|
2026-06-15 11:56:47 | INFO | Fetching Store Master data
|
|
2026-06-15 11:57:01 | INFO | Fetched 5,984 stores
|
|
2026-06-15 11:57:01 | INFO | Fetched 5984 rows
|
|
2026-06-15 11:57:01 | INFO | Creating table Store_Master
|
|
2026-06-15 11:57:01 | INFO | Table ready: Store_Master
|
|
2026-06-15 11:57:02 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 11:57:02 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 11:57:02 | INFO | ================================================================================
|
|
2026-06-15 11:57:02 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 11:57:02 | INFO | Fetching SKU Master data
|
|
2026-06-15 11:57:02 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 11:57:02 | INFO | Fetched 160 rows
|
|
2026-06-15 11:57:02 | INFO | Creating table SKU_Master
|
|
2026-06-15 11:57:02 | INFO | Table ready: SKU_Master
|
|
2026-06-15 11:57:02 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 11:57:02 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 11:57:02 | INFO | ================================================================================
|
|
2026-06-15 11:57:02 | INFO | Processing Table: display_master
|
|
2026-06-15 11:57:02 | INFO | Fetching Display Master data
|
|
2026-06-15 11:57:03 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 11:57:03 | INFO | Fetched 135 rows
|
|
2026-06-15 11:57:03 | INFO | Creating table display_master
|
|
2026-06-15 11:57:03 | INFO | Table ready: display_master
|
|
2026-06-15 11:57:03 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 11:57:03 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 11:57:03 | INFO | ================================================================================
|
|
2026-06-15 11:57:03 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 11:57:03 | INFO | Fetching Employee Master data
|
|
2026-06-15 11:57:04 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 11:57:04 | INFO | Fetched 2273 rows
|
|
2026-06-15 11:57:04 | INFO | Creating table Employee_Master
|
|
2026-06-15 11:57:04 | INFO | Table ready: Employee_Master
|
|
2026-06-15 11:57:04 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 11:57:04 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 11:57:04 | INFO | ================================================================================
|
|
2026-06-15 11:57:04 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 11:57:04 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 11:57:10 | INFO | Fetched 21,133 Journey Plan records
|
|
2026-06-15 11:57:10 | INFO | Fetched 21133 rows
|
|
2026-06-15 11:57:10 | INFO | Creating table Journey_Plan
|
|
2026-06-15 11:57:10 | INFO | Table ready: Journey_Plan
|
|
2026-06-15 11:57:10 | INFO | Journey_Plan: inserted 21,133 rows into ClickHouse
|
|
2026-06-15 11:57:10 | INFO | Journey_Plan loaded successfully (21133 rows)
|
|
2026-06-15 11:57:10 | INFO | ================================================================================
|
|
2026-06-15 11:57:10 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 11:57:10 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 11:57:10 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 11:57:10 | INFO | Fetched 29 rows
|
|
2026-06-15 11:57:10 | INFO | Creating table coverage_remarks
|
|
2026-06-15 11:57:10 | INFO | Table ready: coverage_remarks
|
|
2026-06-15 11:57:11 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 11:57:11 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 11:57:11 | INFO | ================================================================================
|
|
2026-06-15 11:57:11 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 11:57:11 | INFO | Fetching Mapping Store Visibility for 2026-06-14
|
|
2026-06-15 11:57:11 | INFO | Fetched 0 Mapping Store Visibility records
|
|
2026-06-15 11:57:11 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-15 11:57:11 | INFO | ================================================================================
|
|
2026-06-15 11:57:11 | INFO | Processing Table: Master_VisibilityReason
|
|
2026-06-15 11:57:11 | INFO | Fetching Master Visibility Reason data
|
|
2026-06-15 11:57:11 | INFO | Fetched 17 Master Visibility Reason records
|
|
2026-06-15 11:57:11 | INFO | Fetched 17 rows
|
|
2026-06-15 11:57:11 | INFO | Creating table Master_VisibilityReason
|
|
2026-06-15 11:57:11 | INFO | Table ready: Master_VisibilityReason
|
|
2026-06-15 11:57:12 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-15 11:57:12 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-15 11:57:12 | INFO | ================================================================================
|
|
2026-06-15 11:57:12 | INFO | Processing Table: Master_VisibilityDefinition
|
|
2026-06-15 11:57:12 | INFO | Fetching Master Visibility Definition data
|
|
2026-06-15 11:57:12 | INFO | Fetched 861 Master Visibility Definition records
|
|
2026-06-15 11:57:12 | INFO | Fetched 861 rows
|
|
2026-06-15 11:57:12 | INFO | Creating table Master_VisibilityDefinition
|
|
2026-06-15 11:57:12 | INFO | Table ready: Master_VisibilityDefinition
|
|
2026-06-15 11:57:12 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-15 11:57:12 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-15 11:57:12 | INFO | ================================================================================
|
|
2026-06-15 11:57:12 | INFO | Processing Table: Web_Logins
|
|
2026-06-15 11:57:12 | INFO | Fetching Web Login data for 2026-06-14
|
|
2026-06-15 11:57:13 | INFO | Fetched 25 Web Login records
|
|
2026-06-15 11:57:13 | INFO | Fetched 25 rows
|
|
2026-06-15 11:57:13 | INFO | Creating table Web_Logins
|
|
2026-06-15 11:57:13 | INFO | Table ready: Web_Logins
|
|
2026-06-15 11:57:13 | INFO | Web_Logins: inserted 25 rows into ClickHouse
|
|
2026-06-15 11:57:13 | INFO | Web_Logins loaded successfully (25 rows)
|
|
2026-06-15 11:57:13 | INFO | ================================================================================
|
|
2026-06-15 11:57:13 | INFO | Processing Table: Promotion
|
|
2026-06-15 11:57:13 | INFO | Fetching Promotion data for 571 MIDs
|
|
2026-06-15 11:57:34 | INFO | Fetched 4,426 Promotion records
|
|
2026-06-15 11:57:34 | INFO | Fetched 4426 rows
|
|
2026-06-15 11:57:34 | INFO | Creating table Promotion
|
|
2026-06-15 11:57:34 | INFO | Table ready: Promotion
|
|
2026-06-15 11:57:34 | INFO | Promotion: inserted 4,426 rows into ClickHouse
|
|
2026-06-15 11:57:34 | INFO | Promotion loaded successfully (4426 rows)
|
|
2026-06-15 11:57:34 | INFO | ================================================================================
|
|
2026-06-15 11:57:34 | INFO | Processing Table: PaidVisibility
|
|
2026-06-15 11:57:34 | INFO | Fetching Paid Visibility data for 571 MIDs
|
|
2026-06-15 11:57:37 | INFO | Fetched 1,296 Paid Visibility records
|
|
2026-06-15 11:57:37 | INFO | Fetched 1296 rows
|
|
2026-06-15 11:57:37 | INFO | Creating table PaidVisibility
|
|
2026-06-15 11:57:37 | INFO | Table ready: PaidVisibility
|
|
2026-06-15 11:57:37 | INFO | PaidVisibility: inserted 1,296 rows into ClickHouse
|
|
2026-06-15 11:57:37 | INFO | PaidVisibility loaded successfully (1296 rows)
|
|
2026-06-15 11:57:37 | INFO | ================================================================================
|
|
2026-06-15 11:57:37 | INFO | Processing Table: Master_Salesterritorylayer
|
|
2026-06-15 11:57:37 | INFO | Fetching Master Sales Territory Layer data
|
|
2026-06-15 11:57:38 | INFO | Fetched 33 Master Sales Territory Layer records
|
|
2026-06-15 11:57:38 | INFO | Fetched 33 rows
|
|
2026-06-15 11:57:38 | INFO | Creating table Master_Salesterritorylayer
|
|
2026-06-15 11:57:38 | INFO | Table ready: Master_Salesterritorylayer
|
|
2026-06-15 11:57:38 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-15 11:57:38 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-15 11:57:38 | INFO | ================================================================================
|
|
2026-06-15 11:57:38 | INFO | Pipeline Completed Successfully
|
|
2026-06-15 11:57:38 | INFO | ================================================================================
|
|
2026-06-15 11:58:05 | INFO | ================================================================================
|
|
2026-06-15 11:58:05 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 11:58:05 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 11:58:05 | INFO | Connecting to databases...
|
|
2026-06-15 11:58:06 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000296AA3A75F0>
|
|
2026-06-15 11:58:07 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000296AB569450>
|
|
2026-06-15 11:58:08 | INFO | Database connections established
|
|
2026-06-15 11:58:08 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 11:58:08 | INFO | Found 571 MIDs
|
|
2026-06-15 11:58:09 | INFO | ================================================================================
|
|
2026-06-15 11:58:09 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 11:58:09 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 11:58:09 | INFO | Fetched 1,995 rows from SQL Server
|
|
2026-06-15 11:58:09 | INFO | Fetched 1995 rows
|
|
2026-06-15 11:58:09 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 11:58:09 | INFO | SOS_OneApp: inserted 1,995 rows into ClickHouse
|
|
2026-06-15 11:58:09 | INFO | SOS_OneApp loaded successfully (1995 rows)
|
|
2026-06-15 11:58:09 | INFO | ================================================================================
|
|
2026-06-15 11:58:09 | INFO | Processing Table: OQaD
|
|
2026-06-15 11:58:09 | INFO | Fetching OQaD data for run_date=2026-06-14
|
|
2026-06-15 11:58:12 | INFO | Fetched 443 rows
|
|
2026-06-15 11:58:12 | INFO | Fetched 443 rows
|
|
2026-06-15 11:58:12 | INFO | Deleting from OQaD
|
|
2026-06-15 11:58:12 | INFO | OQaD: inserted 443 rows into ClickHouse
|
|
2026-06-15 11:58:12 | INFO | OQaD loaded successfully (443 rows)
|
|
2026-06-15 11:58:12 | INFO | ================================================================================
|
|
2026-06-15 11:58:12 | INFO | Processing Table: Survey
|
|
2026-06-15 11:58:12 | INFO | Fetching Survey data for 571 MIDs
|
|
2026-06-15 11:58:12 | INFO | Fetched 173 Survey rows
|
|
2026-06-15 11:58:12 | INFO | Fetched 173 rows
|
|
2026-06-15 11:58:13 | INFO | Deleting from Survey
|
|
2026-06-15 11:58:13 | INFO | Survey: inserted 173 rows into ClickHouse
|
|
2026-06-15 11:58:13 | INFO | Survey loaded successfully (173 rows)
|
|
2026-06-15 11:58:13 | INFO | ================================================================================
|
|
2026-06-15 11:58:13 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 11:58:13 | INFO | Fetching Additional Visibility data for 571 MIDs
|
|
2026-06-15 11:58:13 | INFO | Fetched 1,648 Additional Visibility rows
|
|
2026-06-15 11:58:13 | INFO | Fetched 1648 rows
|
|
2026-06-15 11:58:13 | INFO | Deleting from additional_visibility
|
|
2026-06-15 11:58:14 | INFO | additional_visibility: inserted 1,648 rows into ClickHouse
|
|
2026-06-15 11:58:14 | INFO | additional_visibility loaded successfully (1648 rows)
|
|
2026-06-15 11:58:14 | INFO | ================================================================================
|
|
2026-06-15 11:58:14 | INFO | Processing Table: Coverage
|
|
2026-06-15 11:58:14 | INFO | Fetching coverage data for 571 MIDs
|
|
2026-06-15 11:58:14 | INFO | Fetched 514 rows from SQL Server
|
|
2026-06-15 11:58:14 | INFO | Fetched 514 rows
|
|
2026-06-15 11:58:14 | INFO | Deleting from Coverage
|
|
2026-06-15 11:58:14 | INFO | Coverage: inserted 514 rows into ClickHouse
|
|
2026-06-15 11:58:14 | INFO | Coverage loaded successfully (514 rows)
|
|
2026-06-15 11:58:14 | INFO | ================================================================================
|
|
2026-06-15 11:58:14 | INFO | Processing Table: Login
|
|
2026-06-15 11:58:14 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 11:58:15 | INFO | Fetched 454 Login rows
|
|
2026-06-15 11:58:15 | INFO | Fetched 454 rows
|
|
2026-06-15 11:58:15 | INFO | Deleting from Login
|
|
2026-06-15 11:58:15 | ERROR | Failed processing table Login
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\test2.py", line 236, in main
|
|
delete_existing_data(
|
|
~~~~~~~~~~~~~~~~~~~~^
|
|
client=client,
|
|
^^^^^^^^^^^^^^
|
|
...<3 lines>...
|
|
emp_visit_df=emp_visit_df,
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\delete_task.py", line 83, in delete_existing_data
|
|
delete_rows(
|
|
~~~~~~~~~~~^
|
|
client,
|
|
^^^^^^^
|
|
table_name,
|
|
^^^^^^^^^^^
|
|
f"MID IN ({mids_str})",
|
|
^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\delete_task.py", line 46, in delete_rows
|
|
client.command(query)
|
|
~~~~~~~~~~~~~~^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 486, in command
|
|
response = self._raw_request(payload, params, headers, method, fields=fields, server_wait=False)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
|
|
self._error_handler(response)
|
|
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
|
|
raise err_type(err_str, code=code, name=name) from None
|
|
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 47, server response: Code: 47. DB::Exception: Missing columns: 'MID' while processing: 'isZeroOrNull(MID IN (1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650)), _database, _part_data_version, _part_granule_offset, _part_offset, _disk_name, _sample_factor, _block_offset, _table, login_date, _part_uuid, _part_starting_offset, first_store_in_time, _part_index, _part, unique_id, _block_number, login_time, _partition_id, last_store_out_time, _distance, employee_id', required columns: '_part_data_version' '_database' '_part_offset' '_part_granule_offset' 'MID' '_disk_name' '_sample_factor' 'login_date' '_block_offset' '_table' '_part_uuid' '_part' '_part_index' '_part_starting_offset' 'first_store_in_time' 'login_time' '_block_number' 'unique_id' '_partition_id' 'employee_id' '_distance' 'last_store_out_time', maybe you meant: 'login_date', 'first_store_in_time', 'login_time', 'unique_id', 'employee_id' or 'last_store_out_time'. (UNKNOWN_IDENTIFIER) (for url http://172.188.12.194:8123)
|
|
2026-06-15 12:03:38 | INFO | ================================================================================
|
|
2026-06-15 12:03:38 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 12:03:38 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 12:03:38 | INFO | Connecting to databases...
|
|
2026-06-15 12:03:39 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001ED17D575F0>
|
|
2026-06-15 12:03:40 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001ED18F29450>
|
|
2026-06-15 12:03:41 | INFO | Database connections established
|
|
2026-06-15 12:03:41 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 12:03:41 | INFO | Found 571 MIDs
|
|
2026-06-15 12:03:41 | INFO | ================================================================================
|
|
2026-06-15 12:03:41 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 12:03:41 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 12:03:42 | INFO | Fetched 1,995 rows from SQL Server
|
|
2026-06-15 12:03:42 | INFO | Fetched 1995 rows
|
|
2026-06-15 12:03:42 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 12:03:42 | INFO | SOS_OneApp: inserted 1,995 rows into ClickHouse
|
|
2026-06-15 12:03:42 | INFO | SOS_OneApp loaded successfully (1995 rows)
|
|
2026-06-15 12:03:42 | INFO | ================================================================================
|
|
2026-06-15 12:03:42 | INFO | Processing Table: OQaD
|
|
2026-06-15 12:03:42 | INFO | Fetching OQaD data for run_date=2026-06-14
|
|
2026-06-15 12:03:45 | INFO | Fetched 443 rows
|
|
2026-06-15 12:03:45 | INFO | Fetched 443 rows
|
|
2026-06-15 12:03:45 | INFO | Deleting from OQaD
|
|
2026-06-15 12:03:45 | INFO | OQaD: inserted 443 rows into ClickHouse
|
|
2026-06-15 12:03:45 | INFO | OQaD loaded successfully (443 rows)
|
|
2026-06-15 12:03:45 | INFO | ================================================================================
|
|
2026-06-15 12:03:45 | INFO | Processing Table: Survey
|
|
2026-06-15 12:03:45 | INFO | Fetching Survey data for 571 MIDs
|
|
2026-06-15 12:03:45 | INFO | Fetched 173 Survey rows
|
|
2026-06-15 12:03:45 | INFO | Fetched 173 rows
|
|
2026-06-15 12:03:45 | INFO | Deleting from Survey
|
|
2026-06-15 12:03:45 | INFO | Survey: inserted 173 rows into ClickHouse
|
|
2026-06-15 12:03:45 | INFO | Survey loaded successfully (173 rows)
|
|
2026-06-15 12:03:45 | INFO | ================================================================================
|
|
2026-06-15 12:03:45 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 12:03:45 | INFO | Fetching Additional Visibility data for 571 MIDs
|
|
2026-06-15 12:03:46 | INFO | Fetched 1,648 Additional Visibility rows
|
|
2026-06-15 12:03:46 | INFO | Fetched 1648 rows
|
|
2026-06-15 12:03:46 | INFO | Deleting from additional_visibility
|
|
2026-06-15 12:03:46 | INFO | additional_visibility: inserted 1,648 rows into ClickHouse
|
|
2026-06-15 12:03:46 | INFO | additional_visibility loaded successfully (1648 rows)
|
|
2026-06-15 12:03:46 | INFO | ================================================================================
|
|
2026-06-15 12:03:46 | INFO | Processing Table: Coverage
|
|
2026-06-15 12:03:46 | INFO | Fetching coverage data for 571 MIDs
|
|
2026-06-15 12:03:46 | INFO | Fetched 514 rows from SQL Server
|
|
2026-06-15 12:03:46 | INFO | Fetched 514 rows
|
|
2026-06-15 12:03:46 | INFO | Deleting from Coverage
|
|
2026-06-15 12:03:46 | INFO | Coverage: inserted 514 rows into ClickHouse
|
|
2026-06-15 12:03:46 | INFO | Coverage loaded successfully (514 rows)
|
|
2026-06-15 12:03:46 | INFO | ================================================================================
|
|
2026-06-15 12:03:46 | INFO | Processing Table: Login
|
|
2026-06-15 12:03:46 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 12:03:47 | INFO | Fetched 454 Login rows
|
|
2026-06-15 12:03:47 | INFO | Fetched 454 rows
|
|
2026-06-15 12:03:47 | INFO | Deleting from Login
|
|
2026-06-15 12:03:47 | ERROR | Failed processing table Login
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\test2.py", line 236, in main
|
|
delete_existing_data(
|
|
~~~~~~~~~~~~~~~~~~~~^
|
|
client=client,
|
|
^^^^^^^^^^^^^^
|
|
...<3 lines>...
|
|
emp_visit_df=emp_visit_df,
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\delete_task.py", line 83, in delete_existing_data
|
|
delete_rows(
|
|
~~~~~~~~~~~^
|
|
client,
|
|
^^^^^^^
|
|
table_name,
|
|
^^^^^^^^^^^
|
|
f"MID IN ({mids_str})",
|
|
^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\delete_task.py", line 46, in delete_rows
|
|
client.command(query)
|
|
~~~~~~~~~~~~~~^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 486, in command
|
|
response = self._raw_request(payload, params, headers, method, fields=fields, server_wait=False)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
|
|
self._error_handler(response)
|
|
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
|
|
raise err_type(err_str, code=code, name=name) from None
|
|
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 47, server response: Code: 47. DB::Exception: Missing columns: 'MID' while processing: 'isZeroOrNull(MID IN (1055918, 1057103, 1057202, 1057487, 1057606, 1058004, 1058337, 1058351, 1058352, 1058353, 1058423, 1058433, 1058456, 1058499, 1058500, 1058612, 1058627, 1058639, 1058640, 1058650, 1058726, 1058777, 1058782, 1058835, 1058845, 1058870, 1058967, 1059022, 1059036, 1059057, 1059058, 1059069, 1059097, 1059113, 1059114, 1059115, 1059116, 1059117, 1059118, 1059119, 1059120, 1059121, 1059122, 1059123, 1059124, 1059125, 1059126, 1059127, 1059128, 1059129, 1059130, 1059131, 1059132, 1059133, 1059134, 1059135, 1059136, 1059137, 1059138, 1059139, 1059140, 1059141, 1059142, 1059143, 1059144, 1059145, 1059146, 1059147, 1059148, 1059149, 1059150, 1059151, 1059152, 1059153, 1059154, 1059155, 1059156, 1059157, 1059158, 1059159, 1059160, 1059161, 1059162, 1059163, 1059164, 1059165, 1059166, 1059167, 1059168, 1059169, 1059170, 1059171, 1059172, 1059173, 1059174, 1059175, 1059176, 1059177, 1059178, 1059179, 1059180, 1059181, 1059182, 1059183, 1059184, 1059185, 1059186, 1059187, 1059188, 1059189, 1059190, 1059191, 1059192, 1059193, 1059194, 1059195, 1059196, 1059197, 1059198, 1059199, 1059200, 1059201, 1059202, 1059203, 1059204, 1059205, 1059206, 1059207, 1059208, 1059209, 1059210, 1059211, 1059212, 1059213, 1059214, 1059215, 1059216, 1059217, 1059218, 1059219, 1059220, 1059221, 1059222, 1059223, 1059224, 1059225, 1059226, 1059227, 1059228, 1059229, 1059230, 1059231, 1059232, 1059233, 1059234, 1059235, 1059236, 1059237, 1059238, 1059239, 1059240, 1059241, 1059242, 1059243, 1059244, 1059245, 1059246, 1059247, 1059248, 1059249, 1059250, 1059251, 1059252, 1059253, 1059254, 1059255, 1059256, 1059257, 1059258, 1059259, 1059260, 1059261, 1059262, 1059263, 1059264, 1059265, 1059266, 1059267, 1059268, 1059269, 1059270, 1059271, 1059272, 1059273, 1059274, 1059275, 1059276, 1059277, 1059278, 1059279, 1059280, 1059281, 1059282, 1059283, 1059284, 1059285, 1059286, 1059287, 1059288, 1059289, 1059290, 1059291, 1059292, 1059293, 1059294, 1059295, 1059296, 1059297, 1059298, 1059299, 1059300, 1059301, 1059302, 1059303, 1059304, 1059305, 1059306, 1059307, 1059308, 1059309, 1059310, 1059311, 1059312, 1059313, 1059314, 1059315, 1059316, 1059317, 1059318, 1059319, 1059320, 1059321, 1059322, 1059323, 1059324, 1059325, 1059326, 1059327, 1059328, 1059329, 1059330, 1059331, 1059332, 1059333, 1059334, 1059335, 1059336, 1059337, 1059338, 1059339, 1059340, 1059341, 1059342, 1059343, 1059344, 1059345, 1059346, 1059347, 1059348, 1059349, 1059350, 1059351, 1059352, 1059353, 1059354, 1059355, 1059356, 1059357, 1059358, 1059359, 1059360, 1059361, 1059362, 1059363, 1059364, 1059365, 1059366, 1059367, 1059368, 1059369, 1059370, 1059371, 1059372, 1059373, 1059374, 1059375, 1059376, 1059377, 1059378, 1059379, 1059380, 1059381, 1059382, 1059383, 1059384, 1059385, 1059386, 1059387, 1059388, 1059389, 1059390, 1059391, 1059392, 1059393, 1059394, 1059395, 1059396, 1059397, 1059398, 1059399, 1059400, 1059401, 1059402, 1059403, 1059404, 1059405, 1059406, 1059407, 1059408, 1059409, 1059410, 1059411, 1059412, 1059413, 1059414, 1059415, 1059416, 1059417, 1059418, 1059419, 1059420, 1059421, 1059422, 1059423, 1059424, 1059425, 1059426, 1059427, 1059428, 1059429, 1059430, 1059431, 1059432, 1059433, 1059434, 1059435, 1059436, 1059437, 1059438, 1059439, 1059440, 1059441, 1059442, 1059443, 1059444, 1059445, 1059446, 1059447, 1059448, 1059449, 1059450, 1059451, 1059452, 1059453, 1059454, 1059455, 1059456, 1059457, 1059458, 1059459, 1059460, 1059461, 1059462, 1059463, 1059464, 1059465, 1059466, 1059467, 1059468, 1059469, 1059470, 1059471, 1059472, 1059473, 1059474, 1059475, 1059476, 1059477, 1059478, 1059479, 1059480, 1059481, 1059482, 1059483, 1059484, 1059485, 1059486, 1059487, 1059488, 1059489, 1059490, 1059491, 1059492, 1059493, 1059494, 1059495, 1059496, 1059497, 1059498, 1059499, 1059500, 1059501, 1059502, 1059503, 1059504, 1059505, 1059506, 1059507, 1059508, 1059509, 1059510, 1059511, 1059512, 1059513, 1059514, 1059515, 1059516, 1059517, 1059518, 1059519, 1059520, 1059521, 1059522, 1059523, 1059524, 1059525, 1059526, 1059527, 1059528, 1059529, 1059530, 1059531, 1059532, 1059533, 1059534, 1059535, 1059536, 1059537, 1059538, 1059539, 1059540, 1059541, 1059542, 1059543, 1059544, 1059545, 1059546, 1059547, 1059548, 1059549, 1059550, 1059551, 1059552, 1059553, 1059554, 1059555, 1059556, 1059557, 1059558, 1059559, 1059560, 1059561, 1059562, 1059563, 1059564, 1059565, 1059566, 1059567, 1059568, 1059569, 1059570, 1059571, 1059572, 1059573, 1059574, 1059575, 1059576, 1059577, 1059578, 1059579, 1059580, 1059581, 1059582, 1059583, 1059584, 1059585, 1059586, 1059587, 1059588, 1059589, 1059590, 1059591, 1059592, 1059593, 1059594, 1059595, 1059596, 1059597, 1059598, 1059599, 1059600, 1059601, 1059602, 1059603, 1059604, 1059605, 1059606, 1059607, 1059608, 1059609, 1059610, 1059611, 1059612, 1059613, 1059614, 1059615, 1059616, 1059617, 1059618, 1059619, 1059620, 1059621, 1059622, 1059623, 1059624, 1059625, 1059626, 1059627, 1059628, 1059629, 1059630, 1059631, 1059632, 1059633, 1059634, 1059635, 1059636, 1059637, 1059638, 1059639, 1059640, 1059641, 1059642, 1059643, 1059644, 1059645, 1059646, 1059647, 1059648, 1059649, 1059650)), _database, _part_data_version, _part_granule_offset, _part_offset, _disk_name, _sample_factor, _block_offset, _table, login_date, _part_uuid, _part_starting_offset, first_store_in_time, _part_index, _part, unique_id, _block_number, login_time, _partition_id, last_store_out_time, _distance, employee_id', required columns: '_part_data_version' '_database' '_part_offset' '_part_granule_offset' 'MID' '_disk_name' '_sample_factor' 'login_date' '_block_offset' '_table' '_part_uuid' '_part' '_part_index' '_part_starting_offset' 'first_store_in_time' 'login_time' '_block_number' 'unique_id' '_partition_id' 'employee_id' '_distance' 'last_store_out_time', maybe you meant: 'login_date', 'first_store_in_time', 'login_time', 'unique_id', 'employee_id' or 'last_store_out_time'. (UNKNOWN_IDENTIFIER) (for url http://172.188.12.194:8123)
|
|
2026-06-15 12:53:14 | INFO | ================================================================================
|
|
2026-06-15 12:53:14 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 12:53:14 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 12:53:14 | INFO | Connecting to databases...
|
|
2026-06-15 12:53:17 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001A21C7875F0>
|
|
2026-06-15 12:53:18 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001A21D959450>
|
|
2026-06-15 12:53:19 | INFO | Database connections established
|
|
2026-06-15 12:53:19 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 12:53:21 | INFO | Found 571 MIDs
|
|
2026-06-15 12:53:21 | INFO | ================================================================================
|
|
2026-06-15 12:53:21 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 12:53:21 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 12:53:22 | INFO | Fetched 1,997 rows from SQL Server
|
|
2026-06-15 12:53:22 | INFO | Fetched 1997 rows
|
|
2026-06-15 12:53:22 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 12:53:23 | INFO | SOS_OneApp: inserted 1,997 rows into ClickHouse
|
|
2026-06-15 12:53:23 | INFO | SOS_OneApp loaded successfully (1997 rows)
|
|
2026-06-15 12:53:23 | INFO | ================================================================================
|
|
2026-06-15 12:53:23 | INFO | Processing Table: OQaD
|
|
2026-06-15 12:53:23 | INFO | Fetching OQaD data for run_date=2026-06-14
|
|
2026-06-15 12:53:26 | INFO | Fetched 443 rows
|
|
2026-06-15 12:53:26 | INFO | Fetched 443 rows
|
|
2026-06-15 12:53:26 | INFO | Deleting from OQaD
|
|
2026-06-15 12:53:26 | INFO | OQaD: inserted 443 rows into ClickHouse
|
|
2026-06-15 12:53:26 | INFO | OQaD loaded successfully (443 rows)
|
|
2026-06-15 12:53:26 | INFO | ================================================================================
|
|
2026-06-15 12:53:26 | INFO | Processing Table: Survey
|
|
2026-06-15 12:53:26 | INFO | Fetching Survey data for 571 MIDs
|
|
2026-06-15 12:53:27 | INFO | Fetched 174 Survey rows
|
|
2026-06-15 12:53:27 | INFO | Fetched 174 rows
|
|
2026-06-15 12:53:27 | INFO | Deleting from Survey
|
|
2026-06-15 12:53:27 | INFO | Survey: inserted 174 rows into ClickHouse
|
|
2026-06-15 12:53:27 | INFO | Survey loaded successfully (174 rows)
|
|
2026-06-15 12:53:27 | INFO | ================================================================================
|
|
2026-06-15 12:53:27 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 12:53:27 | INFO | Fetching Additional Visibility data for 571 MIDs
|
|
2026-06-15 12:53:28 | INFO | Fetched 1,651 Additional Visibility rows
|
|
2026-06-15 12:53:28 | INFO | Fetched 1651 rows
|
|
2026-06-15 12:53:28 | INFO | Deleting from additional_visibility
|
|
2026-06-15 12:53:28 | INFO | additional_visibility: inserted 1,651 rows into ClickHouse
|
|
2026-06-15 12:53:28 | INFO | additional_visibility loaded successfully (1651 rows)
|
|
2026-06-15 12:53:28 | INFO | ================================================================================
|
|
2026-06-15 12:53:28 | INFO | Processing Table: Coverage
|
|
2026-06-15 12:53:28 | INFO | Fetching coverage data for 571 MIDs
|
|
2026-06-15 12:53:28 | INFO | Fetched 514 rows from SQL Server
|
|
2026-06-15 12:53:28 | INFO | Fetched 514 rows
|
|
2026-06-15 12:53:29 | INFO | Deleting from Coverage
|
|
2026-06-15 12:53:29 | INFO | Coverage: inserted 514 rows into ClickHouse
|
|
2026-06-15 12:53:29 | INFO | Coverage loaded successfully (514 rows)
|
|
2026-06-15 12:53:29 | INFO | ================================================================================
|
|
2026-06-15 12:53:29 | INFO | Processing Table: Login
|
|
2026-06-15 12:53:29 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 12:53:31 | INFO | Fetched 454 Login rows
|
|
2026-06-15 12:53:31 | INFO | Fetched 454 rows
|
|
2026-06-15 12:53:31 | INFO | Deleting from Login
|
|
2026-06-15 12:53:31 | ERROR | Failed processing table Login
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\test2.py", line 236, in main
|
|
delete_existing_data(
|
|
~~~~~~~~~~~~~~~~~~~~^
|
|
client=client,
|
|
^^^^^^^^^^^^^^
|
|
...<3 lines>...
|
|
emp_visit_df=emp_visit_df,
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\delete_task.py", line 112, in delete_existing_data
|
|
delete_rows(
|
|
~~~~~~~~~~~^
|
|
client,
|
|
^^^^^^^
|
|
...<4 lines>...
|
|
"""
|
|
^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\delete_task.py", line 46, in delete_rows
|
|
client.command(query)
|
|
~~~~~~~~~~~~~~^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 486, in command
|
|
response = self._raw_request(payload, params, headers, method, fields=fields, server_wait=False)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
|
|
self._error_handler(response)
|
|
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
|
|
raise err_type(err_str, code=code, name=name) from None
|
|
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 47, server response: Code: 47. DB::Exception: Missing columns: 'project_id' while processing: 'isZeroOrNull((project_id = 40148) AND (toDate(login_date) = toDate('2026-06-14'))), _database, _part_data_version, _part_granule_offset, _part_offset, _disk_name, _sample_factor, _block_offset, _table, login_date, _part_uuid, _part_starting_offset, first_store_in_time, _part_index, _part, unique_id, _block_number, login_time, _partition_id, last_store_out_time, _distance, employee_id', required columns: '_part_offset' '_sample_factor' '_disk_name' '_part_data_version' 'project_id' '_database' '_part_granule_offset' '_part_uuid' '_block_offset' '_table' 'login_date' '_part' '_part_index' '_part_starting_offset' 'first_store_in_time' 'login_time' '_block_number' 'unique_id' '_partition_id' 'employee_id' '_distance' 'last_store_out_time', maybe you meant: 'login_date', 'first_store_in_time', 'login_time', 'unique_id', 'employee_id' or 'last_store_out_time'. (UNKNOWN_IDENTIFIER) (for url http://172.188.12.194:8123)
|
|
2026-06-15 13:19:07 | INFO | ================================================================================
|
|
2026-06-15 13:19:07 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:19:07 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 13:19:07 | INFO | Connecting to databases...
|
|
2026-06-15 13:19:07 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000027B7C2875F0>
|
|
2026-06-15 13:19:08 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000027B7D451310>
|
|
2026-06-15 13:19:09 | INFO | Database connections established
|
|
2026-06-15 13:19:09 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 13:19:09 | INFO | Found 571 MIDs
|
|
2026-06-15 13:19:09 | INFO | ================================================================================
|
|
2026-06-15 13:19:09 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:19:09 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 13:19:10 | INFO | Fetched 1,997 rows from SQL Server
|
|
2026-06-15 13:19:10 | INFO | Fetched 1997 rows
|
|
2026-06-15 13:19:10 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 13:19:10 | INFO | SOS_OneApp: inserted 1,997 rows into ClickHouse
|
|
2026-06-15 13:19:10 | INFO | SOS_OneApp loaded successfully (1997 rows)
|
|
2026-06-15 13:19:10 | INFO | ================================================================================
|
|
2026-06-15 13:19:10 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:19:10 | INFO | Fetching OQaD data for run_date=2026-06-14
|
|
2026-06-15 13:19:13 | INFO | Fetched 443 rows
|
|
2026-06-15 13:19:13 | INFO | Fetched 443 rows
|
|
2026-06-15 13:19:13 | INFO | Deleting from OQaD
|
|
2026-06-15 13:19:14 | INFO | OQaD: inserted 443 rows into ClickHouse
|
|
2026-06-15 13:19:14 | INFO | OQaD loaded successfully (443 rows)
|
|
2026-06-15 13:19:14 | INFO | ================================================================================
|
|
2026-06-15 13:19:14 | INFO | Processing Table: Survey
|
|
2026-06-15 13:19:14 | INFO | Fetching Survey data for 571 MIDs
|
|
2026-06-15 13:19:14 | INFO | Fetched 174 Survey rows
|
|
2026-06-15 13:19:14 | INFO | Fetched 174 rows
|
|
2026-06-15 13:19:14 | INFO | Deleting from Survey
|
|
2026-06-15 13:19:14 | INFO | Survey: inserted 174 rows into ClickHouse
|
|
2026-06-15 13:19:14 | INFO | Survey loaded successfully (174 rows)
|
|
2026-06-15 13:19:14 | INFO | ================================================================================
|
|
2026-06-15 13:19:14 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:19:14 | INFO | Fetching Additional Visibility data for 571 MIDs
|
|
2026-06-15 13:19:32 | INFO | Fetched 1,651 Additional Visibility rows
|
|
2026-06-15 13:19:32 | INFO | Fetched 1651 rows
|
|
2026-06-15 13:19:32 | INFO | Deleting from additional_visibility
|
|
2026-06-15 13:19:33 | INFO | additional_visibility: inserted 1,651 rows into ClickHouse
|
|
2026-06-15 13:19:33 | INFO | additional_visibility loaded successfully (1651 rows)
|
|
2026-06-15 13:19:33 | INFO | ================================================================================
|
|
2026-06-15 13:19:33 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:19:33 | INFO | Fetching coverage data for 571 MIDs
|
|
2026-06-15 13:19:34 | INFO | Fetched 514 rows from SQL Server
|
|
2026-06-15 13:19:34 | INFO | Fetched 514 rows
|
|
2026-06-15 13:19:34 | INFO | Deleting from Coverage
|
|
2026-06-15 13:19:34 | INFO | Coverage: inserted 514 rows into ClickHouse
|
|
2026-06-15 13:19:34 | INFO | Coverage loaded successfully (514 rows)
|
|
2026-06-15 13:19:34 | INFO | ================================================================================
|
|
2026-06-15 13:19:34 | INFO | Processing Table: Login
|
|
2026-06-15 13:19:34 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:19:36 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:19:36 | INFO | Fetched 454 rows
|
|
2026-06-15 13:19:36 | INFO | Deleting from Login
|
|
2026-06-15 13:19:36 | ERROR | Failed processing table Login
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\test2.py", line 236, in main
|
|
delete_existing_data(
|
|
~~~~~~~~~~~~~~~~~~~~^
|
|
client=client,
|
|
^^^^^^^^^^^^^^
|
|
...<3 lines>...
|
|
emp_visit_df=emp_visit_df,
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\delete_task.py", line 110, in delete_existing_data
|
|
delete_rows(
|
|
~~~~~~~~~~~^
|
|
client,
|
|
^^^^^^^
|
|
...<3 lines>...
|
|
"""
|
|
^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\delete_task.py", line 46, in delete_rows
|
|
client.command(query)
|
|
~~~~~~~~~~~~~~^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 486, in command
|
|
response = self._raw_request(payload, params, headers, method, fields=fields, server_wait=False)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
|
|
self._error_handler(response)
|
|
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
|
|
raise err_type(err_str, code=code, name=name) from None
|
|
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 62, server response: Code: 62. DB::Exception: Syntax error: failed at position 58 (toDate) (line 5, col 13): toDate(login_date) = toDate('2026-06-14')
|
|
|
|
. Expected one of: token sequence, Dot, token, OR, AND, IS NOT DISTINCT FROM, IS DISTINCT FROM, IS NULL, IS NOT NULL, BETWEEN, NOT BETWEEN, LIKE, ILIKE, NOT LIKE, NOT ILIKE, REGEXP, IN, NOT IN, GLOBAL IN, GLOBAL NOT IN, MOD, DIV, alias, AS, Comma, INTO OUTFILE, FORMAT, SETTINGS, ParallelWithClause, PARALLEL WITH, end of query. (SYNTAX_ERROR) (for url http://172.188.12.194:8123)
|
|
2026-06-15 13:21:54 | INFO | ================================================================================
|
|
2026-06-15 13:21:54 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:21:54 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 13:21:54 | INFO | Connecting to databases...
|
|
2026-06-15 13:21:55 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001F2EB5175F0>
|
|
2026-06-15 13:21:56 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001F2EC6E9310>
|
|
2026-06-15 13:21:56 | INFO | Database connections established
|
|
2026-06-15 13:21:56 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 13:21:57 | INFO | Found 571 MIDs
|
|
2026-06-15 13:21:57 | INFO | ================================================================================
|
|
2026-06-15 13:21:57 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:21:57 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 13:21:58 | INFO | Fetched 1,997 rows from SQL Server
|
|
2026-06-15 13:21:58 | INFO | Fetched 1997 rows
|
|
2026-06-15 13:21:58 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 13:21:58 | INFO | SOS_OneApp: inserted 1,997 rows into ClickHouse
|
|
2026-06-15 13:21:58 | INFO | SOS_OneApp loaded successfully (1997 rows)
|
|
2026-06-15 13:21:58 | INFO | ================================================================================
|
|
2026-06-15 13:21:58 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:21:58 | INFO | Fetching OQaD data for run_date=2026-06-14
|
|
2026-06-15 13:22:00 | INFO | Fetched 443 rows
|
|
2026-06-15 13:22:00 | INFO | Fetched 443 rows
|
|
2026-06-15 13:22:00 | INFO | Deleting from OQaD
|
|
2026-06-15 13:22:00 | INFO | OQaD: inserted 443 rows into ClickHouse
|
|
2026-06-15 13:22:00 | INFO | OQaD loaded successfully (443 rows)
|
|
2026-06-15 13:22:01 | INFO | ================================================================================
|
|
2026-06-15 13:22:01 | INFO | Processing Table: Survey
|
|
2026-06-15 13:22:01 | INFO | Fetching Survey data for 571 MIDs
|
|
2026-06-15 13:22:01 | INFO | Fetched 174 Survey rows
|
|
2026-06-15 13:22:01 | INFO | Fetched 174 rows
|
|
2026-06-15 13:22:01 | INFO | Deleting from Survey
|
|
2026-06-15 13:22:01 | INFO | Survey: inserted 174 rows into ClickHouse
|
|
2026-06-15 13:22:01 | INFO | Survey loaded successfully (174 rows)
|
|
2026-06-15 13:22:01 | INFO | ================================================================================
|
|
2026-06-15 13:22:01 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:22:01 | INFO | Fetching Additional Visibility data for 571 MIDs
|
|
2026-06-15 13:22:02 | INFO | Fetched 1,651 Additional Visibility rows
|
|
2026-06-15 13:22:02 | INFO | Fetched 1651 rows
|
|
2026-06-15 13:22:02 | INFO | Deleting from additional_visibility
|
|
2026-06-15 13:22:02 | INFO | additional_visibility: inserted 1,651 rows into ClickHouse
|
|
2026-06-15 13:22:02 | INFO | additional_visibility loaded successfully (1651 rows)
|
|
2026-06-15 13:22:02 | INFO | ================================================================================
|
|
2026-06-15 13:22:02 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:22:02 | INFO | Fetching coverage data for 571 MIDs
|
|
2026-06-15 13:22:02 | INFO | Fetched 514 rows from SQL Server
|
|
2026-06-15 13:22:02 | INFO | Fetched 514 rows
|
|
2026-06-15 13:22:02 | INFO | Deleting from Coverage
|
|
2026-06-15 13:22:02 | INFO | Coverage: inserted 514 rows into ClickHouse
|
|
2026-06-15 13:22:02 | INFO | Coverage loaded successfully (514 rows)
|
|
2026-06-15 13:22:02 | INFO | ================================================================================
|
|
2026-06-15 13:22:02 | INFO | Processing Table: Login
|
|
2026-06-15 13:22:02 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:22:03 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:22:03 | INFO | Fetched 454 rows
|
|
2026-06-15 13:22:03 | INFO | Deleting from Login
|
|
2026-06-15 13:22:03 | INFO | No delete logic required for Login
|
|
2026-06-15 13:22:03 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 13:22:03 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 13:22:03 | INFO | ================================================================================
|
|
2026-06-15 13:22:03 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 13:22:03 | INFO | Fetching Stock Details data for 571 MIDs
|
|
2026-06-15 13:22:12 | INFO | Fetched 28,574 Stock Details rows
|
|
2026-06-15 13:22:12 | INFO | Fetched 28574 rows
|
|
2026-06-15 13:22:12 | INFO | Deleting from Stock_Details
|
|
2026-06-15 13:22:13 | INFO | Stock_Details: inserted 28,574 rows into ClickHouse
|
|
2026-06-15 13:22:13 | INFO | Stock_Details loaded successfully (28574 rows)
|
|
2026-06-15 13:22:13 | INFO | ================================================================================
|
|
2026-06-15 13:22:13 | INFO | Processing Table: Attendance
|
|
2026-06-15 13:22:13 | INFO | Fetching Attendance data from 2026-05-30 to 2026-06-14
|
|
2026-06-15 13:22:14 | INFO | Fetched 11,543 attendance rows for 594 employees
|
|
2026-06-15 13:22:14 | INFO | Fetched 11543 rows
|
|
2026-06-15 13:22:14 | INFO | Truncated table Attendance
|
|
2026-06-15 13:22:15 | INFO | Attendance: inserted 11,543 rows into ClickHouse
|
|
2026-06-15 13:22:15 | INFO | Attendance loaded successfully (11543 rows)
|
|
2026-06-15 13:22:15 | INFO | ================================================================================
|
|
2026-06-15 13:22:15 | INFO | Processing Table: Store_Master
|
|
2026-06-15 13:22:15 | INFO | Fetching Store Master data
|
|
2026-06-15 13:22:17 | INFO | Fetched 5,984 stores
|
|
2026-06-15 13:22:17 | INFO | Fetched 5984 rows
|
|
2026-06-15 13:22:17 | INFO | Truncated table Store_Master
|
|
2026-06-15 13:22:18 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 13:22:18 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 13:22:18 | INFO | ================================================================================
|
|
2026-06-15 13:22:18 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 13:22:18 | INFO | Fetching SKU Master data
|
|
2026-06-15 13:22:18 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 13:22:18 | INFO | Fetched 160 rows
|
|
2026-06-15 13:22:18 | INFO | Truncated table SKU_Master
|
|
2026-06-15 13:22:18 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 13:22:18 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 13:22:18 | INFO | ================================================================================
|
|
2026-06-15 13:22:18 | INFO | Processing Table: display_master
|
|
2026-06-15 13:22:18 | INFO | Fetching Display Master data
|
|
2026-06-15 13:22:18 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 13:22:18 | INFO | Fetched 135 rows
|
|
2026-06-15 13:22:19 | INFO | Truncated table display_master
|
|
2026-06-15 13:22:19 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 13:22:19 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 13:22:19 | INFO | ================================================================================
|
|
2026-06-15 13:22:19 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 13:22:19 | INFO | Fetching Employee Master data
|
|
2026-06-15 13:22:19 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 13:22:19 | INFO | Fetched 2273 rows
|
|
2026-06-15 13:22:19 | INFO | Truncated table Employee_Master
|
|
2026-06-15 13:22:20 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 13:22:20 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 13:22:20 | INFO | ================================================================================
|
|
2026-06-15 13:22:20 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 13:22:20 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 13:22:20 | INFO | Fetched 21,141 Journey Plan records
|
|
2026-06-15 13:22:20 | INFO | Fetched 21141 rows
|
|
2026-06-15 13:22:20 | INFO | Deleting from Journey_Plan
|
|
2026-06-15 13:22:21 | INFO | Journey_Plan: inserted 21,141 rows into ClickHouse
|
|
2026-06-15 13:22:21 | INFO | Journey_Plan loaded successfully (21141 rows)
|
|
2026-06-15 13:22:21 | INFO | ================================================================================
|
|
2026-06-15 13:22:21 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 13:22:21 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 13:22:21 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 13:22:21 | INFO | Fetched 29 rows
|
|
2026-06-15 13:22:21 | INFO | Truncated table coverage_remarks
|
|
2026-06-15 13:22:21 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 13:22:21 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 13:22:21 | INFO | ================================================================================
|
|
2026-06-15 13:22:21 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 13:22:21 | INFO | Fetching Mapping Store Visibility for 2026-06-14
|
|
2026-06-15 13:22:21 | INFO | Fetched 0 Mapping Store Visibility records
|
|
2026-06-15 13:22:21 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-15 13:22:21 | INFO | ================================================================================
|
|
2026-06-15 13:22:21 | INFO | Processing Table: Master_VisibilityReason
|
|
2026-06-15 13:22:21 | INFO | Fetching Master Visibility Reason data
|
|
2026-06-15 13:22:21 | INFO | Fetched 17 Master Visibility Reason records
|
|
2026-06-15 13:22:21 | INFO | Fetched 17 rows
|
|
2026-06-15 13:22:21 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-15 13:22:21 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-15 13:22:21 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-15 13:22:21 | INFO | ================================================================================
|
|
2026-06-15 13:22:22 | INFO | Processing Table: Master_VisibilityDefinition
|
|
2026-06-15 13:22:22 | INFO | Fetching Master Visibility Definition data
|
|
2026-06-15 13:22:22 | INFO | Fetched 861 Master Visibility Definition records
|
|
2026-06-15 13:22:22 | INFO | Fetched 861 rows
|
|
2026-06-15 13:22:22 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-15 13:22:22 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-15 13:22:22 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-15 13:22:22 | INFO | ================================================================================
|
|
2026-06-15 13:22:22 | INFO | Processing Table: Web_Logins
|
|
2026-06-15 13:22:22 | INFO | Fetching Web Login data for 2026-06-14
|
|
2026-06-15 13:22:22 | INFO | Fetched 25 Web Login records
|
|
2026-06-15 13:22:22 | INFO | Fetched 25 rows
|
|
2026-06-15 13:22:22 | INFO | Deleting from Web_Logins
|
|
2026-06-15 13:22:22 | ERROR | Failed processing table Web_Logins
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\test2.py", line 236, in main
|
|
delete_existing_data(
|
|
~~~~~~~~~~~~~~~~~~~~^
|
|
client=client,
|
|
^^^^^^^^^^^^^^
|
|
...<3 lines>...
|
|
emp_visit_df=emp_visit_df,
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\delete_task.py", line 129, in delete_existing_data
|
|
delete_rows(
|
|
~~~~~~~~~~~^
|
|
client,
|
|
^^^^^^^
|
|
...<4 lines>...
|
|
""",
|
|
^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\delete_task.py", line 46, in delete_rows
|
|
client.command(query)
|
|
~~~~~~~~~~~~~~^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 486, in command
|
|
response = self._raw_request(payload, params, headers, method, fields=fields, server_wait=False)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
|
|
self._error_handler(response)
|
|
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
|
|
raise err_type(err_str, code=code, name=name) from None
|
|
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 47, server response: Code: 47. DB::Exception: Missing columns: 'date' while processing: 'isZeroOrNull(toDate(date) = toDate('2026-06-14')), _block_offset, _block_number, _table, _database, _distance, _part_starting_offset, _part_data_version, _part_granule_offset, _disk_name, _sample_factor, _partition_id, right_name, _part_index, _part, create_by, employee_name, activity_type, _part_uuid, login_time, create_date, project_id, activity_name, login_date, _part_offset, emp_id, designation, supervisor_name, supervisor_id', required columns: 'date' '_block_number' '_table' '_block_offset' '_part_uuid' 'activity_name' 'project_id' '_distance' '_database' '_part_starting_offset' '_part_data_version' '_part_granule_offset' '_disk_name' '_sample_factor' '_partition_id' 'create_date' 'login_time' 'designation' '_part_index' 'right_name' 'activity_type' 'employee_name' 'login_date' 'create_by' '_part_offset' 'emp_id' '_part' 'supervisor_name' 'supervisor_id', maybe you meant: 'activity_name', 'project_id', 'create_date', 'login_time', 'designation', 'right_name', 'activity_type', 'employee_name', 'login_date', 'create_by', 'emp_id', 'supervisor_name' or 'supervisor_id'. (UNKNOWN_IDENTIFIER) (for url http://172.188.12.194:8123)
|
|
2026-06-15 13:26:05 | INFO | ================================================================================
|
|
2026-06-15 13:26:05 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:26:05 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 13:26:05 | INFO | Connecting to databases...
|
|
2026-06-15 13:26:06 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000265710275F0>
|
|
2026-06-15 13:26:08 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000265721F9310>
|
|
2026-06-15 13:26:08 | INFO | Database connections established
|
|
2026-06-15 13:26:08 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 13:26:08 | INFO | Found 571 MIDs
|
|
2026-06-15 13:26:09 | INFO | ================================================================================
|
|
2026-06-15 13:26:09 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:26:09 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 13:26:09 | INFO | Fetched 1,997 rows from SQL Server
|
|
2026-06-15 13:26:09 | INFO | Fetched 1997 rows
|
|
2026-06-15 13:26:09 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 13:26:09 | INFO | SOS_OneApp: inserted 1,997 rows into ClickHouse
|
|
2026-06-15 13:26:09 | INFO | SOS_OneApp loaded successfully (1997 rows)
|
|
2026-06-15 13:26:09 | INFO | ================================================================================
|
|
2026-06-15 13:26:09 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:26:09 | INFO | Fetching OQaD data for run_date=2026-06-14
|
|
2026-06-15 13:26:11 | INFO | Fetched 443 rows
|
|
2026-06-15 13:26:11 | INFO | Fetched 443 rows
|
|
2026-06-15 13:26:12 | INFO | Deleting from OQaD
|
|
2026-06-15 13:26:12 | INFO | OQaD: inserted 443 rows into ClickHouse
|
|
2026-06-15 13:26:12 | INFO | OQaD loaded successfully (443 rows)
|
|
2026-06-15 13:26:12 | INFO | ================================================================================
|
|
2026-06-15 13:26:12 | INFO | Processing Table: Survey
|
|
2026-06-15 13:26:12 | INFO | Fetching Survey data for 571 MIDs
|
|
2026-06-15 13:26:12 | INFO | Fetched 174 Survey rows
|
|
2026-06-15 13:26:12 | INFO | Fetched 174 rows
|
|
2026-06-15 13:26:12 | INFO | Deleting from Survey
|
|
2026-06-15 13:26:12 | INFO | Survey: inserted 174 rows into ClickHouse
|
|
2026-06-15 13:26:12 | INFO | Survey loaded successfully (174 rows)
|
|
2026-06-15 13:26:12 | INFO | ================================================================================
|
|
2026-06-15 13:26:12 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:26:12 | INFO | Fetching Additional Visibility data for 571 MIDs
|
|
2026-06-15 13:26:13 | INFO | Fetched 1,651 Additional Visibility rows
|
|
2026-06-15 13:26:13 | INFO | Fetched 1651 rows
|
|
2026-06-15 13:26:13 | INFO | Deleting from additional_visibility
|
|
2026-06-15 13:26:13 | INFO | additional_visibility: inserted 1,651 rows into ClickHouse
|
|
2026-06-15 13:26:13 | INFO | additional_visibility loaded successfully (1651 rows)
|
|
2026-06-15 13:26:13 | INFO | ================================================================================
|
|
2026-06-15 13:26:13 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:26:13 | INFO | Fetching coverage data for 571 MIDs
|
|
2026-06-15 13:26:13 | INFO | Fetched 514 rows from SQL Server
|
|
2026-06-15 13:26:13 | INFO | Fetched 514 rows
|
|
2026-06-15 13:26:13 | INFO | Deleting from Coverage
|
|
2026-06-15 13:26:13 | INFO | Coverage: inserted 514 rows into ClickHouse
|
|
2026-06-15 13:26:13 | INFO | Coverage loaded successfully (514 rows)
|
|
2026-06-15 13:26:13 | INFO | ================================================================================
|
|
2026-06-15 13:26:13 | INFO | Processing Table: Login
|
|
2026-06-15 13:26:13 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:26:14 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:26:14 | INFO | Fetched 454 rows
|
|
2026-06-15 13:26:14 | INFO | Deleting from Login
|
|
2026-06-15 13:26:14 | INFO | No delete logic required for Login
|
|
2026-06-15 13:26:14 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 13:26:14 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 13:26:14 | INFO | ================================================================================
|
|
2026-06-15 13:26:14 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 13:26:14 | INFO | Fetching Stock Details data for 571 MIDs
|
|
2026-06-15 13:26:17 | INFO | Fetched 28,574 Stock Details rows
|
|
2026-06-15 13:26:17 | INFO | Fetched 28574 rows
|
|
2026-06-15 13:26:17 | INFO | Deleting from Stock_Details
|
|
2026-06-15 13:26:18 | INFO | Stock_Details: inserted 28,574 rows into ClickHouse
|
|
2026-06-15 13:26:18 | INFO | Stock_Details loaded successfully (28574 rows)
|
|
2026-06-15 13:26:18 | INFO | ================================================================================
|
|
2026-06-15 13:26:18 | INFO | Processing Table: Attendance
|
|
2026-06-15 13:26:18 | INFO | Fetching Attendance data from 2026-05-30 to 2026-06-14
|
|
2026-06-15 13:26:18 | INFO | Fetched 11,543 attendance rows for 594 employees
|
|
2026-06-15 13:26:18 | INFO | Fetched 11543 rows
|
|
2026-06-15 13:26:19 | INFO | Truncated table Attendance
|
|
2026-06-15 13:26:19 | INFO | Attendance: inserted 11,543 rows into ClickHouse
|
|
2026-06-15 13:26:19 | INFO | Attendance loaded successfully (11543 rows)
|
|
2026-06-15 13:26:19 | INFO | ================================================================================
|
|
2026-06-15 13:26:19 | INFO | Processing Table: Store_Master
|
|
2026-06-15 13:26:19 | INFO | Fetching Store Master data
|
|
2026-06-15 13:26:21 | INFO | Fetched 5,984 stores
|
|
2026-06-15 13:26:21 | INFO | Fetched 5984 rows
|
|
2026-06-15 13:26:22 | INFO | Truncated table Store_Master
|
|
2026-06-15 13:26:22 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 13:26:22 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 13:26:22 | INFO | ================================================================================
|
|
2026-06-15 13:26:22 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 13:26:22 | INFO | Fetching SKU Master data
|
|
2026-06-15 13:26:22 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 13:26:22 | INFO | Fetched 160 rows
|
|
2026-06-15 13:26:22 | INFO | Truncated table SKU_Master
|
|
2026-06-15 13:26:22 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 13:26:22 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 13:26:22 | INFO | ================================================================================
|
|
2026-06-15 13:26:22 | INFO | Processing Table: display_master
|
|
2026-06-15 13:26:22 | INFO | Fetching Display Master data
|
|
2026-06-15 13:26:23 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 13:26:23 | INFO | Fetched 135 rows
|
|
2026-06-15 13:26:23 | INFO | Truncated table display_master
|
|
2026-06-15 13:26:23 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 13:26:23 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 13:26:23 | INFO | ================================================================================
|
|
2026-06-15 13:26:23 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 13:26:23 | INFO | Fetching Employee Master data
|
|
2026-06-15 13:26:24 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 13:26:24 | INFO | Fetched 2273 rows
|
|
2026-06-15 13:26:24 | INFO | Truncated table Employee_Master
|
|
2026-06-15 13:26:24 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 13:26:24 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 13:26:24 | INFO | ================================================================================
|
|
2026-06-15 13:26:24 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 13:26:24 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 13:26:24 | INFO | Fetched 21,141 Journey Plan records
|
|
2026-06-15 13:26:24 | INFO | Fetched 21141 rows
|
|
2026-06-15 13:26:24 | INFO | Deleting from Journey_Plan
|
|
2026-06-15 13:26:25 | INFO | Journey_Plan: inserted 21,141 rows into ClickHouse
|
|
2026-06-15 13:26:25 | INFO | Journey_Plan loaded successfully (21141 rows)
|
|
2026-06-15 13:26:25 | INFO | ================================================================================
|
|
2026-06-15 13:26:25 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 13:26:25 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 13:26:25 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 13:26:25 | INFO | Fetched 29 rows
|
|
2026-06-15 13:26:25 | INFO | Truncated table coverage_remarks
|
|
2026-06-15 13:26:25 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 13:26:25 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 13:26:25 | INFO | ================================================================================
|
|
2026-06-15 13:26:25 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 13:26:25 | INFO | Fetching Mapping Store Visibility for 2026-06-14
|
|
2026-06-15 13:26:25 | INFO | Fetched 0 Mapping Store Visibility records
|
|
2026-06-15 13:26:25 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-15 13:26:25 | INFO | ================================================================================
|
|
2026-06-15 13:26:25 | INFO | Processing Table: Master_VisibilityReason
|
|
2026-06-15 13:26:25 | INFO | Fetching Master Visibility Reason data
|
|
2026-06-15 13:26:25 | INFO | Fetched 17 Master Visibility Reason records
|
|
2026-06-15 13:26:25 | INFO | Fetched 17 rows
|
|
2026-06-15 13:26:25 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-15 13:26:25 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-15 13:26:25 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-15 13:26:25 | INFO | ================================================================================
|
|
2026-06-15 13:26:25 | INFO | Processing Table: Master_VisibilityDefinition
|
|
2026-06-15 13:26:25 | INFO | Fetching Master Visibility Definition data
|
|
2026-06-15 13:26:25 | INFO | Fetched 861 Master Visibility Definition records
|
|
2026-06-15 13:26:25 | INFO | Fetched 861 rows
|
|
2026-06-15 13:26:26 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-15 13:26:26 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-15 13:26:26 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-15 13:26:26 | INFO | ================================================================================
|
|
2026-06-15 13:26:26 | INFO | Processing Table: Web_Logins
|
|
2026-06-15 13:26:26 | INFO | Fetching Web Login data for 2026-06-14
|
|
2026-06-15 13:26:26 | INFO | Fetched 25 Web Login records
|
|
2026-06-15 13:26:26 | INFO | Fetched 25 rows
|
|
2026-06-15 13:26:26 | INFO | Deleting from Web_Logins
|
|
2026-06-15 13:26:26 | ERROR | Failed processing table Web_Logins
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\test2.py", line 236, in main
|
|
delete_existing_data(
|
|
~~~~~~~~~~~~~~~~~~~~^
|
|
client=client,
|
|
^^^^^^^^^^^^^^
|
|
...<3 lines>...
|
|
emp_visit_df=emp_visit_df,
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\delete_task.py", line 129, in delete_existing_data
|
|
delete_rows(
|
|
~~~~~~~~~~~^
|
|
client,
|
|
^^^^^^^
|
|
...<4 lines>...
|
|
""",
|
|
^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\delete_task.py", line 46, in delete_rows
|
|
client.command(query)
|
|
~~~~~~~~~~~~~~^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 486, in command
|
|
response = self._raw_request(payload, params, headers, method, fields=fields, server_wait=False)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
|
|
self._error_handler(response)
|
|
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
|
|
raise err_type(err_str, code=code, name=name) from None
|
|
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 47, server response: Code: 47. DB::Exception: Missing columns: 'date' while processing: 'isZeroOrNull(toDate(date) = toDate('2026-06-14')), _block_offset, _block_number, _table, _database, _distance, _part_starting_offset, _part_data_version, _part_granule_offset, _disk_name, _sample_factor, _partition_id, right_name, _part_index, _part, create_by, employee_name, activity_type, _part_uuid, login_time, create_date, project_id, activity_name, login_date, _part_offset, emp_id, designation, supervisor_name, supervisor_id', required columns: 'date' '_block_number' '_table' '_block_offset' '_part_uuid' 'activity_name' 'project_id' '_distance' '_database' '_part_starting_offset' '_part_data_version' '_part_granule_offset' '_disk_name' '_sample_factor' '_partition_id' 'create_date' 'login_time' 'designation' '_part_index' 'right_name' 'activity_type' 'employee_name' 'login_date' 'create_by' '_part_offset' 'emp_id' '_part' 'supervisor_name' 'supervisor_id', maybe you meant: 'activity_name', 'project_id', 'create_date', 'login_time', 'designation', 'right_name', 'activity_type', 'employee_name', 'login_date', 'create_by', 'emp_id', 'supervisor_name' or 'supervisor_id'. (UNKNOWN_IDENTIFIER) (for url http://172.188.12.194:8123)
|
|
2026-06-15 13:27:51 | INFO | ================================================================================
|
|
2026-06-15 13:27:51 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:27:51 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 13:27:51 | INFO | Connecting to databases...
|
|
2026-06-15 13:27:52 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000026359EC75F0>
|
|
2026-06-15 13:27:53 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002635B099310>
|
|
2026-06-15 13:27:53 | INFO | Database connections established
|
|
2026-06-15 13:27:53 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 13:27:54 | INFO | Found 571 MIDs
|
|
2026-06-15 13:27:54 | INFO | ================================================================================
|
|
2026-06-15 13:27:54 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:27:54 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 13:27:54 | INFO | Fetched 1,997 rows from SQL Server
|
|
2026-06-15 13:27:54 | INFO | Fetched 1997 rows
|
|
2026-06-15 13:27:54 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 13:27:55 | INFO | SOS_OneApp: inserted 1,997 rows into ClickHouse
|
|
2026-06-15 13:27:55 | INFO | SOS_OneApp loaded successfully (1997 rows)
|
|
2026-06-15 13:27:55 | INFO | ================================================================================
|
|
2026-06-15 13:27:55 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:27:55 | INFO | Fetching OQaD data for run_date=2026-06-14
|
|
2026-06-15 13:27:57 | INFO | Fetched 443 rows
|
|
2026-06-15 13:27:57 | INFO | Fetched 443 rows
|
|
2026-06-15 13:27:57 | INFO | Deleting from OQaD
|
|
2026-06-15 13:27:57 | INFO | OQaD: inserted 443 rows into ClickHouse
|
|
2026-06-15 13:27:57 | INFO | OQaD loaded successfully (443 rows)
|
|
2026-06-15 13:27:57 | INFO | ================================================================================
|
|
2026-06-15 13:27:57 | INFO | Processing Table: Survey
|
|
2026-06-15 13:27:57 | INFO | Fetching Survey data for 571 MIDs
|
|
2026-06-15 13:27:57 | INFO | Fetched 174 Survey rows
|
|
2026-06-15 13:27:57 | INFO | Fetched 174 rows
|
|
2026-06-15 13:27:57 | INFO | Deleting from Survey
|
|
2026-06-15 13:27:58 | INFO | Survey: inserted 174 rows into ClickHouse
|
|
2026-06-15 13:27:58 | INFO | Survey loaded successfully (174 rows)
|
|
2026-06-15 13:27:58 | INFO | ================================================================================
|
|
2026-06-15 13:27:58 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:27:58 | INFO | Fetching Additional Visibility data for 571 MIDs
|
|
2026-06-15 13:27:58 | INFO | Fetched 1,651 Additional Visibility rows
|
|
2026-06-15 13:27:58 | INFO | Fetched 1651 rows
|
|
2026-06-15 13:27:58 | INFO | Deleting from additional_visibility
|
|
2026-06-15 13:27:58 | INFO | additional_visibility: inserted 1,651 rows into ClickHouse
|
|
2026-06-15 13:27:58 | INFO | additional_visibility loaded successfully (1651 rows)
|
|
2026-06-15 13:27:58 | INFO | ================================================================================
|
|
2026-06-15 13:27:58 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:27:58 | INFO | Fetching coverage data for 571 MIDs
|
|
2026-06-15 13:27:58 | INFO | Fetched 514 rows from SQL Server
|
|
2026-06-15 13:27:58 | INFO | Fetched 514 rows
|
|
2026-06-15 13:27:59 | INFO | Deleting from Coverage
|
|
2026-06-15 13:27:59 | INFO | Coverage: inserted 514 rows into ClickHouse
|
|
2026-06-15 13:27:59 | INFO | Coverage loaded successfully (514 rows)
|
|
2026-06-15 13:27:59 | INFO | ================================================================================
|
|
2026-06-15 13:27:59 | INFO | Processing Table: Login
|
|
2026-06-15 13:27:59 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:27:59 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:27:59 | INFO | Fetched 454 rows
|
|
2026-06-15 13:27:59 | INFO | Deleting from Login
|
|
2026-06-15 13:28:00 | INFO | No delete logic required for Login
|
|
2026-06-15 13:28:00 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 13:28:00 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 13:28:00 | INFO | ================================================================================
|
|
2026-06-15 13:28:00 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 13:28:00 | INFO | Fetching Stock Details data for 571 MIDs
|
|
2026-06-15 13:28:03 | INFO | Fetched 28,574 Stock Details rows
|
|
2026-06-15 13:28:03 | INFO | Fetched 28574 rows
|
|
2026-06-15 13:28:03 | INFO | Deleting from Stock_Details
|
|
2026-06-15 13:28:03 | INFO | Stock_Details: inserted 28,574 rows into ClickHouse
|
|
2026-06-15 13:28:03 | INFO | Stock_Details loaded successfully (28574 rows)
|
|
2026-06-15 13:28:03 | INFO | ================================================================================
|
|
2026-06-15 13:28:03 | INFO | Processing Table: Attendance
|
|
2026-06-15 13:28:03 | INFO | Fetching Attendance data from 2026-05-30 to 2026-06-14
|
|
2026-06-15 13:28:04 | INFO | Fetched 11,543 attendance rows for 594 employees
|
|
2026-06-15 13:28:04 | INFO | Fetched 11543 rows
|
|
2026-06-15 13:28:04 | INFO | Truncated table Attendance
|
|
2026-06-15 13:28:05 | INFO | Attendance: inserted 11,543 rows into ClickHouse
|
|
2026-06-15 13:28:05 | INFO | Attendance loaded successfully (11543 rows)
|
|
2026-06-15 13:28:05 | INFO | ================================================================================
|
|
2026-06-15 13:28:05 | INFO | Processing Table: Store_Master
|
|
2026-06-15 13:28:05 | INFO | Fetching Store Master data
|
|
2026-06-15 13:28:06 | INFO | Fetched 5,984 stores
|
|
2026-06-15 13:28:06 | INFO | Fetched 5984 rows
|
|
2026-06-15 13:28:06 | INFO | Truncated table Store_Master
|
|
2026-06-15 13:28:07 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 13:28:07 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 13:28:07 | INFO | ================================================================================
|
|
2026-06-15 13:28:07 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 13:28:07 | INFO | Fetching SKU Master data
|
|
2026-06-15 13:28:07 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 13:28:07 | INFO | Fetched 160 rows
|
|
2026-06-15 13:28:07 | INFO | Truncated table SKU_Master
|
|
2026-06-15 13:28:07 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 13:28:07 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 13:28:07 | INFO | ================================================================================
|
|
2026-06-15 13:28:07 | INFO | Processing Table: display_master
|
|
2026-06-15 13:28:07 | INFO | Fetching Display Master data
|
|
2026-06-15 13:28:08 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 13:28:08 | INFO | Fetched 135 rows
|
|
2026-06-15 13:28:08 | INFO | Truncated table display_master
|
|
2026-06-15 13:28:08 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 13:28:08 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 13:28:08 | INFO | ================================================================================
|
|
2026-06-15 13:28:08 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 13:28:08 | INFO | Fetching Employee Master data
|
|
2026-06-15 13:28:08 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 13:28:08 | INFO | Fetched 2273 rows
|
|
2026-06-15 13:28:08 | INFO | Truncated table Employee_Master
|
|
2026-06-15 13:28:09 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 13:28:09 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 13:28:09 | INFO | ================================================================================
|
|
2026-06-15 13:28:09 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 13:28:09 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 13:28:09 | INFO | Fetched 21,141 Journey Plan records
|
|
2026-06-15 13:28:09 | INFO | Fetched 21141 rows
|
|
2026-06-15 13:28:09 | INFO | Deleting from Journey_Plan
|
|
2026-06-15 13:28:09 | INFO | Journey_Plan: inserted 21,141 rows into ClickHouse
|
|
2026-06-15 13:28:09 | INFO | Journey_Plan loaded successfully (21141 rows)
|
|
2026-06-15 13:28:09 | INFO | ================================================================================
|
|
2026-06-15 13:28:09 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 13:28:09 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 13:28:09 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 13:28:09 | INFO | Fetched 29 rows
|
|
2026-06-15 13:28:10 | INFO | Truncated table coverage_remarks
|
|
2026-06-15 13:28:10 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 13:28:10 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 13:28:10 | INFO | ================================================================================
|
|
2026-06-15 13:28:10 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 13:28:10 | INFO | Fetching Mapping Store Visibility for 2026-06-14
|
|
2026-06-15 13:28:10 | INFO | Fetched 0 Mapping Store Visibility records
|
|
2026-06-15 13:28:10 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-15 13:28:10 | INFO | ================================================================================
|
|
2026-06-15 13:28:10 | INFO | Processing Table: Master_VisibilityReason
|
|
2026-06-15 13:28:10 | INFO | Fetching Master Visibility Reason data
|
|
2026-06-15 13:28:10 | INFO | Fetched 17 Master Visibility Reason records
|
|
2026-06-15 13:28:10 | INFO | Fetched 17 rows
|
|
2026-06-15 13:28:10 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-15 13:28:10 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-15 13:28:10 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-15 13:28:10 | INFO | ================================================================================
|
|
2026-06-15 13:28:10 | INFO | Processing Table: Master_VisibilityDefinition
|
|
2026-06-15 13:28:10 | INFO | Fetching Master Visibility Definition data
|
|
2026-06-15 13:28:10 | INFO | Fetched 861 Master Visibility Definition records
|
|
2026-06-15 13:28:10 | INFO | Fetched 861 rows
|
|
2026-06-15 13:28:10 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-15 13:28:11 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-15 13:28:11 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-15 13:28:11 | INFO | ================================================================================
|
|
2026-06-15 13:28:11 | INFO | Processing Table: Web_Logins
|
|
2026-06-15 13:28:11 | INFO | Fetching Web Login data for 2026-06-14
|
|
2026-06-15 13:28:11 | INFO | Fetched 25 Web Login records
|
|
2026-06-15 13:28:11 | INFO | Fetched 25 rows
|
|
2026-06-15 13:28:11 | INFO | Deleting from Web_Logins
|
|
2026-06-15 13:28:11 | ERROR | Failed processing table Web_Logins
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\test2.py", line 248, in main
|
|
load_to_clickhouse(
|
|
~~~~~~~~~~~~~~~~~~^
|
|
client=client,
|
|
^^^^^^^^^^^^^^
|
|
table_name=table_name,
|
|
^^^^^^^^^^^^^^^^^^^^^^
|
|
df=df,
|
|
^^^^^^
|
|
)
|
|
^
|
|
File "D:\data_move\clickhouse_task\load_table.py", line 25, in load_to_clickhouse
|
|
client.insert_arrow(
|
|
~~~~~~~~~~~~~~~~~~~^
|
|
table=table_name,
|
|
^^^^^^^^^^^^^^^^^
|
|
arrow_table=arrow_table,
|
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
|
)
|
|
^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\client.py", line 1054, in insert_arrow
|
|
return self.raw_insert(full_table, column_names, insert_block, settings, "Arrow", transport_settings)
|
|
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 432, in raw_insert
|
|
response = self._raw_request(insert_block, params, headers, server_wait=False)
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 645, in _raw_request
|
|
self._error_handler(response)
|
|
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
|
|
File "C:\Users\ankitm\AppData\Local\uv\cache\environments-v2\test2-86cf58bc5ffcfdee\Lib\site-packages\clickhouse_connect\driver\httpclient.py", line 537, in _error_handler
|
|
raise err_type(err_str, code=code, name=name) from None
|
|
clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 16, server response: Code: 16. DB::Exception: No such column date in table kelloggs_1.Web_Logins (c845112d-d41d-478b-ba61-6c113b115c9d). (NO_SUCH_COLUMN_IN_TABLE) (for url http://172.188.12.194:8123)
|
|
2026-06-15 13:34:41 | INFO | ================================================================================
|
|
2026-06-15 13:34:41 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:34:41 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 13:34:41 | INFO | Connecting to databases...
|
|
2026-06-15 13:34:41 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000204A13675F0>
|
|
2026-06-15 13:34:42 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000204A25D9310>
|
|
2026-06-15 13:34:43 | INFO | Database connections established
|
|
2026-06-15 13:34:43 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 13:34:43 | INFO | Found 571 MIDs
|
|
2026-06-15 13:34:43 | INFO | ================================================================================
|
|
2026-06-15 13:34:43 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:34:43 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 13:34:44 | INFO | Fetched 1,997 rows from SQL Server
|
|
2026-06-15 13:34:44 | INFO | Fetched 1997 rows
|
|
2026-06-15 13:34:44 | INFO | Creating table SOS_OneApp
|
|
2026-06-15 13:34:44 | INFO | Table ready: SOS_OneApp
|
|
2026-06-15 13:34:44 | INFO | SOS_OneApp: inserted 1,997 rows into ClickHouse
|
|
2026-06-15 13:34:44 | INFO | SOS_OneApp loaded successfully (1997 rows)
|
|
2026-06-15 13:34:44 | INFO | ================================================================================
|
|
2026-06-15 13:34:44 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:34:44 | INFO | Fetching OQaD data for run_date=2026-06-14
|
|
2026-06-15 13:34:47 | INFO | Fetched 443 rows
|
|
2026-06-15 13:34:47 | INFO | Fetched 443 rows
|
|
2026-06-15 13:34:47 | INFO | Creating table OQaD
|
|
2026-06-15 13:34:47 | INFO | Table ready: OQaD
|
|
2026-06-15 13:34:47 | INFO | OQaD: inserted 443 rows into ClickHouse
|
|
2026-06-15 13:34:47 | INFO | OQaD loaded successfully (443 rows)
|
|
2026-06-15 13:34:47 | INFO | ================================================================================
|
|
2026-06-15 13:34:47 | INFO | Processing Table: Survey
|
|
2026-06-15 13:34:47 | INFO | Fetching Survey data for 571 MIDs
|
|
2026-06-15 13:34:47 | INFO | Fetched 174 Survey rows
|
|
2026-06-15 13:34:47 | INFO | Fetched 174 rows
|
|
2026-06-15 13:34:47 | INFO | Creating table Survey
|
|
2026-06-15 13:34:47 | INFO | Table ready: Survey
|
|
2026-06-15 13:34:48 | INFO | Survey: inserted 174 rows into ClickHouse
|
|
2026-06-15 13:34:48 | INFO | Survey loaded successfully (174 rows)
|
|
2026-06-15 13:34:48 | INFO | ================================================================================
|
|
2026-06-15 13:34:48 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:34:48 | INFO | Fetching Additional Visibility data for 571 MIDs
|
|
2026-06-15 13:34:48 | INFO | Fetched 1,651 Additional Visibility rows
|
|
2026-06-15 13:34:48 | INFO | Fetched 1651 rows
|
|
2026-06-15 13:34:48 | INFO | Creating table additional_visibility
|
|
2026-06-15 13:34:48 | INFO | Table ready: additional_visibility
|
|
2026-06-15 13:34:48 | INFO | additional_visibility: inserted 1,651 rows into ClickHouse
|
|
2026-06-15 13:34:48 | INFO | additional_visibility loaded successfully (1651 rows)
|
|
2026-06-15 13:34:48 | INFO | ================================================================================
|
|
2026-06-15 13:34:48 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:34:48 | INFO | Fetching coverage data for 571 MIDs
|
|
2026-06-15 13:34:48 | INFO | Fetched 514 rows from SQL Server
|
|
2026-06-15 13:34:48 | INFO | Fetched 514 rows
|
|
2026-06-15 13:34:48 | INFO | Creating table Coverage
|
|
2026-06-15 13:34:49 | INFO | Table ready: Coverage
|
|
2026-06-15 13:34:49 | INFO | Coverage: inserted 514 rows into ClickHouse
|
|
2026-06-15 13:34:49 | INFO | Coverage loaded successfully (514 rows)
|
|
2026-06-15 13:34:49 | INFO | ================================================================================
|
|
2026-06-15 13:34:49 | INFO | Processing Table: Login
|
|
2026-06-15 13:34:49 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:34:49 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:34:49 | INFO | Fetched 454 rows
|
|
2026-06-15 13:34:49 | INFO | Creating table Login
|
|
2026-06-15 13:34:49 | INFO | Table ready: Login
|
|
2026-06-15 13:34:50 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 13:34:50 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 13:34:50 | INFO | ================================================================================
|
|
2026-06-15 13:34:50 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 13:34:50 | INFO | Fetching Stock Details data for 571 MIDs
|
|
2026-06-15 13:34:52 | INFO | Fetched 28,574 Stock Details rows
|
|
2026-06-15 13:34:52 | INFO | Fetched 28574 rows
|
|
2026-06-15 13:34:52 | INFO | Creating table Stock_Details
|
|
2026-06-15 13:34:52 | INFO | Table ready: Stock_Details
|
|
2026-06-15 13:34:52 | INFO | Stock_Details: inserted 28,574 rows into ClickHouse
|
|
2026-06-15 13:34:52 | INFO | Stock_Details loaded successfully (28574 rows)
|
|
2026-06-15 13:34:52 | INFO | ================================================================================
|
|
2026-06-15 13:34:52 | INFO | Processing Table: Attendance
|
|
2026-06-15 13:34:52 | INFO | Fetching Attendance data from 2026-05-30 to 2026-06-14
|
|
2026-06-15 13:34:53 | INFO | Fetched 11,543 attendance rows for 594 employees
|
|
2026-06-15 13:34:53 | INFO | Fetched 11543 rows
|
|
2026-06-15 13:34:53 | INFO | Creating table Attendance
|
|
2026-06-15 13:34:54 | INFO | Table ready: Attendance
|
|
2026-06-15 13:34:54 | INFO | Attendance: inserted 11,543 rows into ClickHouse
|
|
2026-06-15 13:34:54 | INFO | Attendance loaded successfully (11543 rows)
|
|
2026-06-15 13:34:54 | INFO | ================================================================================
|
|
2026-06-15 13:34:54 | INFO | Processing Table: Store_Master
|
|
2026-06-15 13:34:54 | INFO | Fetching Store Master data
|
|
2026-06-15 13:34:56 | INFO | Fetched 5,984 stores
|
|
2026-06-15 13:34:56 | INFO | Fetched 5984 rows
|
|
2026-06-15 13:34:56 | INFO | Creating table Store_Master
|
|
2026-06-15 13:34:56 | INFO | Table ready: Store_Master
|
|
2026-06-15 13:34:57 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 13:34:57 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 13:34:57 | INFO | ================================================================================
|
|
2026-06-15 13:34:57 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 13:34:57 | INFO | Fetching SKU Master data
|
|
2026-06-15 13:34:57 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 13:34:57 | INFO | Fetched 160 rows
|
|
2026-06-15 13:34:57 | INFO | Creating table SKU_Master
|
|
2026-06-15 13:34:57 | INFO | Table ready: SKU_Master
|
|
2026-06-15 13:34:57 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 13:34:57 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 13:34:57 | INFO | ================================================================================
|
|
2026-06-15 13:34:57 | INFO | Processing Table: display_master
|
|
2026-06-15 13:34:57 | INFO | Fetching Display Master data
|
|
2026-06-15 13:34:57 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 13:34:57 | INFO | Fetched 135 rows
|
|
2026-06-15 13:34:57 | INFO | Creating table display_master
|
|
2026-06-15 13:34:58 | INFO | Table ready: display_master
|
|
2026-06-15 13:34:58 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 13:34:58 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 13:34:58 | INFO | ================================================================================
|
|
2026-06-15 13:34:58 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 13:34:58 | INFO | Fetching Employee Master data
|
|
2026-06-15 13:34:58 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 13:34:58 | INFO | Fetched 2273 rows
|
|
2026-06-15 13:34:58 | INFO | Creating table Employee_Master
|
|
2026-06-15 13:34:58 | INFO | Table ready: Employee_Master
|
|
2026-06-15 13:34:59 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 13:34:59 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 13:34:59 | INFO | ================================================================================
|
|
2026-06-15 13:34:59 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 13:34:59 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 13:35:00 | INFO | Fetched 21,141 Journey Plan records
|
|
2026-06-15 13:35:00 | INFO | Fetched 21141 rows
|
|
2026-06-15 13:35:00 | INFO | Creating table Journey_Plan
|
|
2026-06-15 13:35:00 | INFO | Table ready: Journey_Plan
|
|
2026-06-15 13:35:00 | INFO | Journey_Plan: inserted 21,141 rows into ClickHouse
|
|
2026-06-15 13:35:00 | INFO | Journey_Plan loaded successfully (21141 rows)
|
|
2026-06-15 13:35:00 | INFO | ================================================================================
|
|
2026-06-15 13:35:00 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 13:35:00 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 13:35:00 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 13:35:00 | INFO | Fetched 29 rows
|
|
2026-06-15 13:35:00 | INFO | Creating table coverage_remarks
|
|
2026-06-15 13:35:00 | INFO | Table ready: coverage_remarks
|
|
2026-06-15 13:35:01 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 13:35:01 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 13:35:01 | INFO | ================================================================================
|
|
2026-06-15 13:35:01 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 13:35:01 | INFO | Fetching Mapping Store Visibility for 2026-06-14
|
|
2026-06-15 13:35:01 | INFO | Fetched 0 Mapping Store Visibility records
|
|
2026-06-15 13:35:01 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-15 13:35:01 | INFO | ================================================================================
|
|
2026-06-15 13:35:01 | INFO | Processing Table: Master_VisibilityReason
|
|
2026-06-15 13:35:01 | INFO | Fetching Master Visibility Reason data
|
|
2026-06-15 13:35:01 | INFO | Fetched 17 Master Visibility Reason records
|
|
2026-06-15 13:35:01 | INFO | Fetched 17 rows
|
|
2026-06-15 13:35:01 | INFO | Creating table Master_VisibilityReason
|
|
2026-06-15 13:35:01 | INFO | Table ready: Master_VisibilityReason
|
|
2026-06-15 13:35:01 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-15 13:35:01 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-15 13:35:01 | INFO | ================================================================================
|
|
2026-06-15 13:35:01 | INFO | Processing Table: Master_VisibilityDefinition
|
|
2026-06-15 13:35:01 | INFO | Fetching Master Visibility Definition data
|
|
2026-06-15 13:35:01 | INFO | Fetched 861 Master Visibility Definition records
|
|
2026-06-15 13:35:01 | INFO | Fetched 861 rows
|
|
2026-06-15 13:35:01 | INFO | Creating table Master_VisibilityDefinition
|
|
2026-06-15 13:35:02 | INFO | Table ready: Master_VisibilityDefinition
|
|
2026-06-15 13:35:02 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-15 13:35:02 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-15 13:35:02 | INFO | ================================================================================
|
|
2026-06-15 13:35:02 | INFO | Processing Table: Web_Logins
|
|
2026-06-15 13:35:02 | INFO | Fetching Web Login data for 2026-06-14
|
|
2026-06-15 13:35:02 | INFO | Fetched 25 Web Login records
|
|
2026-06-15 13:35:02 | INFO | Fetched 25 rows
|
|
2026-06-15 13:35:02 | INFO | Creating table Web_Logins
|
|
2026-06-15 13:35:02 | INFO | Table ready: Web_Logins
|
|
2026-06-15 13:35:02 | INFO | Web_Logins: inserted 25 rows into ClickHouse
|
|
2026-06-15 13:35:02 | INFO | Web_Logins loaded successfully (25 rows)
|
|
2026-06-15 13:35:02 | INFO | ================================================================================
|
|
2026-06-15 13:35:02 | INFO | Processing Table: Promotion
|
|
2026-06-15 13:35:02 | INFO | Fetching Promotion data for 571 MIDs
|
|
2026-06-15 13:35:05 | INFO | Fetched 4,426 Promotion records
|
|
2026-06-15 13:35:05 | INFO | Fetched 4426 rows
|
|
2026-06-15 13:35:05 | INFO | Creating table Promotion
|
|
2026-06-15 13:35:06 | INFO | Table ready: Promotion
|
|
2026-06-15 13:35:06 | INFO | Promotion: inserted 4,426 rows into ClickHouse
|
|
2026-06-15 13:35:06 | INFO | Promotion loaded successfully (4426 rows)
|
|
2026-06-15 13:35:06 | INFO | ================================================================================
|
|
2026-06-15 13:35:06 | INFO | Processing Table: PaidVisibility
|
|
2026-06-15 13:35:06 | INFO | Fetching Paid Visibility data for 571 MIDs
|
|
2026-06-15 13:35:07 | INFO | Fetched 1,296 Paid Visibility records
|
|
2026-06-15 13:35:07 | INFO | Fetched 1296 rows
|
|
2026-06-15 13:35:07 | INFO | Creating table PaidVisibility
|
|
2026-06-15 13:35:07 | INFO | Table ready: PaidVisibility
|
|
2026-06-15 13:35:07 | INFO | PaidVisibility: inserted 1,296 rows into ClickHouse
|
|
2026-06-15 13:35:07 | INFO | PaidVisibility loaded successfully (1296 rows)
|
|
2026-06-15 13:35:07 | INFO | ================================================================================
|
|
2026-06-15 13:35:07 | INFO | Processing Table: Master_Salesterritorylayer
|
|
2026-06-15 13:35:07 | INFO | Fetching Master Sales Territory Layer data
|
|
2026-06-15 13:35:08 | INFO | Fetched 33 Master Sales Territory Layer records
|
|
2026-06-15 13:35:08 | INFO | Fetched 33 rows
|
|
2026-06-15 13:35:08 | INFO | Creating table Master_Salesterritorylayer
|
|
2026-06-15 13:35:08 | INFO | Table ready: Master_Salesterritorylayer
|
|
2026-06-15 13:35:08 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-15 13:35:08 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-15 13:35:08 | INFO | ================================================================================
|
|
2026-06-15 13:35:08 | INFO | Pipeline Completed Successfully
|
|
2026-06-15 13:35:08 | INFO | ================================================================================
|
|
2026-06-15 13:35:25 | INFO | ================================================================================
|
|
2026-06-15 13:35:25 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:35:25 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 13:35:25 | INFO | Connecting to databases...
|
|
2026-06-15 13:35:26 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000123715175F0>
|
|
2026-06-15 13:35:27 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000123726D9310>
|
|
2026-06-15 13:35:27 | INFO | Database connections established
|
|
2026-06-15 13:35:27 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 13:35:27 | INFO | Found 571 MIDs
|
|
2026-06-15 13:35:28 | INFO | ================================================================================
|
|
2026-06-15 13:35:28 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:35:28 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 13:35:28 | INFO | Fetched 1,997 rows from SQL Server
|
|
2026-06-15 13:35:28 | INFO | Fetched 1997 rows
|
|
2026-06-15 13:35:28 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 13:35:29 | INFO | SOS_OneApp: inserted 1,997 rows into ClickHouse
|
|
2026-06-15 13:35:29 | INFO | SOS_OneApp loaded successfully (1997 rows)
|
|
2026-06-15 13:35:29 | INFO | ================================================================================
|
|
2026-06-15 13:35:29 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:35:29 | INFO | Fetching OQaD data for run_date=2026-06-14
|
|
2026-06-15 13:35:31 | INFO | Fetched 443 rows
|
|
2026-06-15 13:35:31 | INFO | Fetched 443 rows
|
|
2026-06-15 13:35:31 | INFO | Deleting from OQaD
|
|
2026-06-15 13:35:31 | INFO | OQaD: inserted 443 rows into ClickHouse
|
|
2026-06-15 13:35:31 | INFO | OQaD loaded successfully (443 rows)
|
|
2026-06-15 13:35:31 | INFO | ================================================================================
|
|
2026-06-15 13:35:31 | INFO | Processing Table: Survey
|
|
2026-06-15 13:35:31 | INFO | Fetching Survey data for 571 MIDs
|
|
2026-06-15 13:35:31 | INFO | Fetched 174 Survey rows
|
|
2026-06-15 13:35:31 | INFO | Fetched 174 rows
|
|
2026-06-15 13:35:32 | INFO | Deleting from Survey
|
|
2026-06-15 13:35:32 | INFO | Survey: inserted 174 rows into ClickHouse
|
|
2026-06-15 13:35:32 | INFO | Survey loaded successfully (174 rows)
|
|
2026-06-15 13:35:32 | INFO | ================================================================================
|
|
2026-06-15 13:35:32 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:35:32 | INFO | Fetching Additional Visibility data for 571 MIDs
|
|
2026-06-15 13:35:32 | INFO | Fetched 1,651 Additional Visibility rows
|
|
2026-06-15 13:35:32 | INFO | Fetched 1651 rows
|
|
2026-06-15 13:35:32 | INFO | Deleting from additional_visibility
|
|
2026-06-15 13:35:32 | INFO | additional_visibility: inserted 1,651 rows into ClickHouse
|
|
2026-06-15 13:35:32 | INFO | additional_visibility loaded successfully (1651 rows)
|
|
2026-06-15 13:35:32 | INFO | ================================================================================
|
|
2026-06-15 13:35:32 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:35:32 | INFO | Fetching coverage data for 571 MIDs
|
|
2026-06-15 13:35:33 | INFO | Fetched 514 rows from SQL Server
|
|
2026-06-15 13:35:33 | INFO | Fetched 514 rows
|
|
2026-06-15 13:35:33 | INFO | Deleting from Coverage
|
|
2026-06-15 13:35:33 | INFO | Coverage: inserted 514 rows into ClickHouse
|
|
2026-06-15 13:35:33 | INFO | Coverage loaded successfully (514 rows)
|
|
2026-06-15 13:35:33 | INFO | ================================================================================
|
|
2026-06-15 13:35:33 | INFO | Processing Table: Login
|
|
2026-06-15 13:35:33 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:35:33 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:35:33 | INFO | Fetched 454 rows
|
|
2026-06-15 13:35:34 | INFO | Deleting from Login
|
|
2026-06-15 13:35:34 | INFO | No delete logic required for Login
|
|
2026-06-15 13:35:34 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 13:35:34 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 13:35:34 | INFO | ================================================================================
|
|
2026-06-15 13:35:34 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 13:35:34 | INFO | Fetching Stock Details data for 571 MIDs
|
|
2026-06-15 13:35:36 | INFO | Fetched 28,574 Stock Details rows
|
|
2026-06-15 13:35:36 | INFO | Fetched 28574 rows
|
|
2026-06-15 13:35:36 | INFO | Deleting from Stock_Details
|
|
2026-06-15 13:35:36 | INFO | Stock_Details: inserted 28,574 rows into ClickHouse
|
|
2026-06-15 13:35:36 | INFO | Stock_Details loaded successfully (28574 rows)
|
|
2026-06-15 13:35:36 | INFO | ================================================================================
|
|
2026-06-15 13:35:36 | INFO | Processing Table: Attendance
|
|
2026-06-15 13:35:36 | INFO | Fetching Attendance data from 2026-05-30 to 2026-06-14
|
|
2026-06-15 13:35:37 | INFO | Fetched 11,543 attendance rows for 594 employees
|
|
2026-06-15 13:35:37 | INFO | Fetched 11543 rows
|
|
2026-06-15 13:35:37 | INFO | Truncated table Attendance
|
|
2026-06-15 13:35:37 | INFO | Attendance: inserted 11,543 rows into ClickHouse
|
|
2026-06-15 13:35:37 | INFO | Attendance loaded successfully (11543 rows)
|
|
2026-06-15 13:35:37 | INFO | ================================================================================
|
|
2026-06-15 13:35:37 | INFO | Processing Table: Store_Master
|
|
2026-06-15 13:35:37 | INFO | Fetching Store Master data
|
|
2026-06-15 13:35:39 | INFO | Fetched 5,984 stores
|
|
2026-06-15 13:35:39 | INFO | Fetched 5984 rows
|
|
2026-06-15 13:35:39 | INFO | Truncated table Store_Master
|
|
2026-06-15 13:35:40 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 13:35:40 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 13:35:40 | INFO | ================================================================================
|
|
2026-06-15 13:35:40 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 13:35:40 | INFO | Fetching SKU Master data
|
|
2026-06-15 13:35:40 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 13:35:40 | INFO | Fetched 160 rows
|
|
2026-06-15 13:35:40 | INFO | Truncated table SKU_Master
|
|
2026-06-15 13:35:40 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 13:35:40 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 13:35:40 | INFO | ================================================================================
|
|
2026-06-15 13:35:40 | INFO | Processing Table: display_master
|
|
2026-06-15 13:35:40 | INFO | Fetching Display Master data
|
|
2026-06-15 13:35:40 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 13:35:40 | INFO | Fetched 135 rows
|
|
2026-06-15 13:35:41 | INFO | Truncated table display_master
|
|
2026-06-15 13:35:41 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 13:35:41 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 13:35:41 | INFO | ================================================================================
|
|
2026-06-15 13:35:41 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 13:35:41 | INFO | Fetching Employee Master data
|
|
2026-06-15 13:35:41 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 13:35:41 | INFO | Fetched 2273 rows
|
|
2026-06-15 13:35:41 | INFO | Truncated table Employee_Master
|
|
2026-06-15 13:35:42 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 13:35:42 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 13:35:42 | INFO | ================================================================================
|
|
2026-06-15 13:35:42 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 13:35:42 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 13:35:42 | INFO | Fetched 21,141 Journey Plan records
|
|
2026-06-15 13:35:42 | INFO | Fetched 21141 rows
|
|
2026-06-15 13:35:42 | INFO | Deleting from Journey_Plan
|
|
2026-06-15 13:35:42 | INFO | Journey_Plan: inserted 21,141 rows into ClickHouse
|
|
2026-06-15 13:35:42 | INFO | Journey_Plan loaded successfully (21141 rows)
|
|
2026-06-15 13:35:42 | INFO | ================================================================================
|
|
2026-06-15 13:35:42 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 13:35:42 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 13:35:43 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 13:35:43 | INFO | Fetched 29 rows
|
|
2026-06-15 13:35:43 | INFO | Truncated table coverage_remarks
|
|
2026-06-15 13:35:43 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 13:35:43 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 13:35:43 | INFO | ================================================================================
|
|
2026-06-15 13:35:43 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 13:35:43 | INFO | Fetching Mapping Store Visibility for 2026-06-14
|
|
2026-06-15 13:35:43 | INFO | Fetched 0 Mapping Store Visibility records
|
|
2026-06-15 13:35:43 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-15 13:35:43 | INFO | ================================================================================
|
|
2026-06-15 13:35:43 | INFO | Processing Table: Master_VisibilityReason
|
|
2026-06-15 13:35:43 | INFO | Fetching Master Visibility Reason data
|
|
2026-06-15 13:35:43 | INFO | Fetched 17 Master Visibility Reason records
|
|
2026-06-15 13:35:43 | INFO | Fetched 17 rows
|
|
2026-06-15 13:35:43 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-15 13:35:43 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-15 13:35:43 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-15 13:35:43 | INFO | ================================================================================
|
|
2026-06-15 13:35:43 | INFO | Processing Table: Master_VisibilityDefinition
|
|
2026-06-15 13:35:43 | INFO | Fetching Master Visibility Definition data
|
|
2026-06-15 13:35:44 | INFO | Fetched 861 Master Visibility Definition records
|
|
2026-06-15 13:35:44 | INFO | Fetched 861 rows
|
|
2026-06-15 13:35:44 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-15 13:35:44 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-15 13:35:44 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-15 13:35:44 | INFO | ================================================================================
|
|
2026-06-15 13:35:44 | INFO | Processing Table: Web_Logins
|
|
2026-06-15 13:35:44 | INFO | Fetching Web Login data for 2026-06-14
|
|
2026-06-15 13:35:44 | INFO | Fetched 25 Web Login records
|
|
2026-06-15 13:35:44 | INFO | Fetched 25 rows
|
|
2026-06-15 13:35:44 | INFO | Deleting from Web_Logins
|
|
2026-06-15 13:35:45 | INFO | Web_Logins: inserted 25 rows into ClickHouse
|
|
2026-06-15 13:35:45 | INFO | Web_Logins loaded successfully (25 rows)
|
|
2026-06-15 13:35:45 | INFO | ================================================================================
|
|
2026-06-15 13:35:45 | INFO | Processing Table: Promotion
|
|
2026-06-15 13:35:45 | INFO | Fetching Promotion data for 571 MIDs
|
|
2026-06-15 13:35:46 | INFO | Fetched 4,426 Promotion records
|
|
2026-06-15 13:35:46 | INFO | Fetched 4426 rows
|
|
2026-06-15 13:35:46 | INFO | Deleting from Promotion
|
|
2026-06-15 13:35:47 | INFO | Promotion: inserted 4,426 rows into ClickHouse
|
|
2026-06-15 13:35:47 | INFO | Promotion loaded successfully (4426 rows)
|
|
2026-06-15 13:35:47 | INFO | ================================================================================
|
|
2026-06-15 13:35:47 | INFO | Processing Table: PaidVisibility
|
|
2026-06-15 13:35:47 | INFO | Fetching Paid Visibility data for 571 MIDs
|
|
2026-06-15 13:35:48 | INFO | Fetched 1,296 Paid Visibility records
|
|
2026-06-15 13:35:48 | INFO | Fetched 1296 rows
|
|
2026-06-15 13:35:48 | INFO | Deleting from PaidVisibility
|
|
2026-06-15 13:35:48 | INFO | PaidVisibility: inserted 1,296 rows into ClickHouse
|
|
2026-06-15 13:35:48 | INFO | PaidVisibility loaded successfully (1296 rows)
|
|
2026-06-15 13:35:48 | INFO | ================================================================================
|
|
2026-06-15 13:35:48 | INFO | Processing Table: Master_Salesterritorylayer
|
|
2026-06-15 13:35:48 | INFO | Fetching Master Sales Territory Layer data
|
|
2026-06-15 13:35:48 | INFO | Fetched 33 Master Sales Territory Layer records
|
|
2026-06-15 13:35:48 | INFO | Fetched 33 rows
|
|
2026-06-15 13:35:48 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-15 13:35:48 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-15 13:35:48 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-15 13:35:48 | INFO | ================================================================================
|
|
2026-06-15 13:35:48 | INFO | Pipeline Completed Successfully
|
|
2026-06-15 13:35:48 | INFO | ================================================================================
|
|
2026-06-15 13:35:54 | INFO | ================================================================================
|
|
2026-06-15 13:35:54 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:35:54 | INFO | Pipeline Run Date: 2026-06-14
|
|
2026-06-15 13:35:54 | INFO | Connecting to databases...
|
|
2026-06-15 13:35:55 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000238FAE375F0>
|
|
2026-06-15 13:35:56 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000238FC059310>
|
|
2026-06-15 13:35:56 | INFO | Database connections established
|
|
2026-06-15 13:35:56 | INFO | Collecting MIDs for: 2026-06-14
|
|
2026-06-15 13:35:56 | INFO | Found 571 MIDs
|
|
2026-06-15 13:35:57 | INFO | ================================================================================
|
|
2026-06-15 13:35:57 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:35:57 | INFO | Fetching data for 571 MIDs
|
|
2026-06-15 13:35:57 | INFO | Fetched 1,997 rows from SQL Server
|
|
2026-06-15 13:35:57 | INFO | Fetched 1997 rows
|
|
2026-06-15 13:35:57 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 13:35:58 | INFO | SOS_OneApp: inserted 1,997 rows into ClickHouse
|
|
2026-06-15 13:35:58 | INFO | SOS_OneApp loaded successfully (1997 rows)
|
|
2026-06-15 13:35:58 | INFO | ================================================================================
|
|
2026-06-15 13:35:58 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:35:58 | INFO | Fetching OQaD data for run_date=2026-06-14
|
|
2026-06-15 13:36:01 | INFO | Fetched 443 rows
|
|
2026-06-15 13:36:01 | INFO | Fetched 443 rows
|
|
2026-06-15 13:36:01 | INFO | Deleting from OQaD
|
|
2026-06-15 13:36:01 | INFO | OQaD: inserted 443 rows into ClickHouse
|
|
2026-06-15 13:36:01 | INFO | OQaD loaded successfully (443 rows)
|
|
2026-06-15 13:36:01 | INFO | ================================================================================
|
|
2026-06-15 13:36:01 | INFO | Processing Table: Survey
|
|
2026-06-15 13:36:01 | INFO | Fetching Survey data for 571 MIDs
|
|
2026-06-15 13:36:01 | INFO | Fetched 174 Survey rows
|
|
2026-06-15 13:36:01 | INFO | Fetched 174 rows
|
|
2026-06-15 13:36:01 | INFO | Deleting from Survey
|
|
2026-06-15 13:36:01 | INFO | Survey: inserted 174 rows into ClickHouse
|
|
2026-06-15 13:36:01 | INFO | Survey loaded successfully (174 rows)
|
|
2026-06-15 13:36:01 | INFO | ================================================================================
|
|
2026-06-15 13:36:01 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:36:01 | INFO | Fetching Additional Visibility data for 571 MIDs
|
|
2026-06-15 13:36:02 | INFO | Fetched 1,651 Additional Visibility rows
|
|
2026-06-15 13:36:02 | INFO | Fetched 1651 rows
|
|
2026-06-15 13:36:02 | INFO | Deleting from additional_visibility
|
|
2026-06-15 13:36:02 | INFO | additional_visibility: inserted 1,651 rows into ClickHouse
|
|
2026-06-15 13:36:02 | INFO | additional_visibility loaded successfully (1651 rows)
|
|
2026-06-15 13:36:02 | INFO | ================================================================================
|
|
2026-06-15 13:36:02 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:36:02 | INFO | Fetching coverage data for 571 MIDs
|
|
2026-06-15 13:36:02 | INFO | Fetched 514 rows from SQL Server
|
|
2026-06-15 13:36:02 | INFO | Fetched 514 rows
|
|
2026-06-15 13:36:02 | INFO | Deleting from Coverage
|
|
2026-06-15 13:36:02 | INFO | Coverage: inserted 514 rows into ClickHouse
|
|
2026-06-15 13:36:02 | INFO | Coverage loaded successfully (514 rows)
|
|
2026-06-15 13:36:02 | INFO | ================================================================================
|
|
2026-06-15 13:36:02 | INFO | Processing Table: Login
|
|
2026-06-15 13:36:02 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:36:03 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:36:03 | INFO | Fetched 454 rows
|
|
2026-06-15 13:36:03 | INFO | Deleting from Login
|
|
2026-06-15 13:36:03 | INFO | No delete logic required for Login
|
|
2026-06-15 13:36:03 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 13:36:03 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 13:36:03 | INFO | ================================================================================
|
|
2026-06-15 13:36:03 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 13:36:03 | INFO | Fetching Stock Details data for 571 MIDs
|
|
2026-06-15 13:36:09 | INFO | Fetched 28,574 Stock Details rows
|
|
2026-06-15 13:36:09 | INFO | Fetched 28574 rows
|
|
2026-06-15 13:36:09 | INFO | Deleting from Stock_Details
|
|
2026-06-15 13:36:09 | INFO | Stock_Details: inserted 28,574 rows into ClickHouse
|
|
2026-06-15 13:36:09 | INFO | Stock_Details loaded successfully (28574 rows)
|
|
2026-06-15 13:36:09 | INFO | ================================================================================
|
|
2026-06-15 13:36:09 | INFO | Processing Table: Attendance
|
|
2026-06-15 13:36:09 | INFO | Fetching Attendance data from 2026-05-30 to 2026-06-14
|
|
2026-06-15 13:36:10 | INFO | Fetched 11,543 attendance rows for 594 employees
|
|
2026-06-15 13:36:10 | INFO | Fetched 11543 rows
|
|
2026-06-15 13:36:10 | INFO | Truncated table Attendance
|
|
2026-06-15 13:36:10 | INFO | Attendance: inserted 11,543 rows into ClickHouse
|
|
2026-06-15 13:36:10 | INFO | Attendance loaded successfully (11543 rows)
|
|
2026-06-15 13:36:10 | INFO | ================================================================================
|
|
2026-06-15 13:36:10 | INFO | Processing Table: Store_Master
|
|
2026-06-15 13:36:10 | INFO | Fetching Store Master data
|
|
2026-06-15 13:36:12 | INFO | Fetched 5,984 stores
|
|
2026-06-15 13:36:12 | INFO | Fetched 5984 rows
|
|
2026-06-15 13:36:12 | INFO | Truncated table Store_Master
|
|
2026-06-15 13:36:13 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 13:36:13 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 13:36:13 | INFO | ================================================================================
|
|
2026-06-15 13:36:13 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 13:36:13 | INFO | Fetching SKU Master data
|
|
2026-06-15 13:36:13 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 13:36:13 | INFO | Fetched 160 rows
|
|
2026-06-15 13:36:13 | INFO | Truncated table SKU_Master
|
|
2026-06-15 13:36:13 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 13:36:13 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 13:36:13 | INFO | ================================================================================
|
|
2026-06-15 13:36:13 | INFO | Processing Table: display_master
|
|
2026-06-15 13:36:13 | INFO | Fetching Display Master data
|
|
2026-06-15 13:36:13 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 13:36:13 | INFO | Fetched 135 rows
|
|
2026-06-15 13:36:13 | INFO | Truncated table display_master
|
|
2026-06-15 13:36:13 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 13:36:13 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 13:36:13 | INFO | ================================================================================
|
|
2026-06-15 13:36:13 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 13:36:13 | INFO | Fetching Employee Master data
|
|
2026-06-15 13:36:14 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 13:36:14 | INFO | Fetched 2273 rows
|
|
2026-06-15 13:36:14 | INFO | Truncated table Employee_Master
|
|
2026-06-15 13:36:14 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 13:36:14 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 13:36:14 | INFO | ================================================================================
|
|
2026-06-15 13:36:14 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 13:36:14 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 13:36:15 | INFO | Fetched 21,141 Journey Plan records
|
|
2026-06-15 13:36:15 | INFO | Fetched 21141 rows
|
|
2026-06-15 13:36:15 | INFO | Deleting from Journey_Plan
|
|
2026-06-15 13:36:15 | INFO | Journey_Plan: inserted 21,141 rows into ClickHouse
|
|
2026-06-15 13:36:15 | INFO | Journey_Plan loaded successfully (21141 rows)
|
|
2026-06-15 13:36:15 | INFO | ================================================================================
|
|
2026-06-15 13:36:15 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 13:36:15 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 13:36:15 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 13:36:15 | INFO | Fetched 29 rows
|
|
2026-06-15 13:36:15 | INFO | Truncated table coverage_remarks
|
|
2026-06-15 13:36:15 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 13:36:15 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 13:36:15 | INFO | ================================================================================
|
|
2026-06-15 13:36:15 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 13:36:15 | INFO | Fetching Mapping Store Visibility for 2026-06-14
|
|
2026-06-15 13:36:15 | INFO | Fetched 0 Mapping Store Visibility records
|
|
2026-06-15 13:36:15 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-15 13:36:15 | INFO | ================================================================================
|
|
2026-06-15 13:36:15 | INFO | Processing Table: Master_VisibilityReason
|
|
2026-06-15 13:36:15 | INFO | Fetching Master Visibility Reason data
|
|
2026-06-15 13:36:15 | INFO | Fetched 17 Master Visibility Reason records
|
|
2026-06-15 13:36:15 | INFO | Fetched 17 rows
|
|
2026-06-15 13:36:16 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-15 13:36:16 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-15 13:36:16 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-15 13:36:16 | INFO | ================================================================================
|
|
2026-06-15 13:36:16 | INFO | Processing Table: Master_VisibilityDefinition
|
|
2026-06-15 13:36:16 | INFO | Fetching Master Visibility Definition data
|
|
2026-06-15 13:36:16 | INFO | Fetched 861 Master Visibility Definition records
|
|
2026-06-15 13:36:16 | INFO | Fetched 861 rows
|
|
2026-06-15 13:36:16 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-15 13:36:16 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-15 13:36:16 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-15 13:36:16 | INFO | ================================================================================
|
|
2026-06-15 13:36:16 | INFO | Processing Table: Web_Logins
|
|
2026-06-15 13:36:16 | INFO | Fetching Web Login data for 2026-06-14
|
|
2026-06-15 13:36:16 | INFO | Fetched 25 Web Login records
|
|
2026-06-15 13:36:16 | INFO | Fetched 25 rows
|
|
2026-06-15 13:36:16 | INFO | Deleting from Web_Logins
|
|
2026-06-15 13:36:16 | INFO | Web_Logins: inserted 25 rows into ClickHouse
|
|
2026-06-15 13:36:16 | INFO | Web_Logins loaded successfully (25 rows)
|
|
2026-06-15 13:36:16 | INFO | ================================================================================
|
|
2026-06-15 13:36:16 | INFO | Processing Table: Promotion
|
|
2026-06-15 13:36:16 | INFO | Fetching Promotion data for 571 MIDs
|
|
2026-06-15 13:36:18 | INFO | Fetched 4,426 Promotion records
|
|
2026-06-15 13:36:18 | INFO | Fetched 4426 rows
|
|
2026-06-15 13:36:18 | INFO | Deleting from Promotion
|
|
2026-06-15 13:36:18 | INFO | Promotion: inserted 4,426 rows into ClickHouse
|
|
2026-06-15 13:36:18 | INFO | Promotion loaded successfully (4426 rows)
|
|
2026-06-15 13:36:18 | INFO | ================================================================================
|
|
2026-06-15 13:36:18 | INFO | Processing Table: PaidVisibility
|
|
2026-06-15 13:36:18 | INFO | Fetching Paid Visibility data for 571 MIDs
|
|
2026-06-15 13:36:19 | INFO | Fetched 1,296 Paid Visibility records
|
|
2026-06-15 13:36:19 | INFO | Fetched 1296 rows
|
|
2026-06-15 13:36:19 | INFO | Deleting from PaidVisibility
|
|
2026-06-15 13:36:20 | INFO | PaidVisibility: inserted 1,296 rows into ClickHouse
|
|
2026-06-15 13:36:20 | INFO | PaidVisibility loaded successfully (1296 rows)
|
|
2026-06-15 13:36:20 | INFO | ================================================================================
|
|
2026-06-15 13:36:20 | INFO | Processing Table: Master_Salesterritorylayer
|
|
2026-06-15 13:36:20 | INFO | Fetching Master Sales Territory Layer data
|
|
2026-06-15 13:36:20 | INFO | Fetched 33 Master Sales Territory Layer records
|
|
2026-06-15 13:36:20 | INFO | Fetched 33 rows
|
|
2026-06-15 13:36:20 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-15 13:36:20 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-15 13:36:20 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-15 13:36:20 | INFO | ================================================================================
|
|
2026-06-15 13:36:20 | INFO | Pipeline Completed Successfully
|
|
2026-06-15 13:36:20 | INFO | ================================================================================
|
|
2026-06-15 13:37:03 | INFO | ================================================================================
|
|
2026-06-15 13:37:03 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:37:03 | INFO | Pipeline Run Date: 2026-06-03
|
|
2026-06-15 13:37:03 | INFO | Connecting to databases...
|
|
2026-06-15 13:37:04 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001CAC7EF7A10>
|
|
2026-06-15 13:37:05 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001CAC90DD310>
|
|
2026-06-15 13:37:06 | INFO | Database connections established
|
|
2026-06-15 13:37:06 | INFO | Collecting MIDs for: 2026-06-03
|
|
2026-06-15 13:37:06 | INFO | Found 808 MIDs
|
|
2026-06-15 13:37:06 | INFO | ================================================================================
|
|
2026-06-15 13:37:06 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:37:06 | INFO | Fetching data for 808 MIDs
|
|
2026-06-15 13:37:07 | INFO | Fetched 3,565 rows from SQL Server
|
|
2026-06-15 13:37:07 | INFO | Fetched 3565 rows
|
|
2026-06-15 13:37:08 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 13:37:08 | INFO | SOS_OneApp: inserted 3,565 rows into ClickHouse
|
|
2026-06-15 13:37:08 | INFO | SOS_OneApp loaded successfully (3565 rows)
|
|
2026-06-15 13:37:08 | INFO | ================================================================================
|
|
2026-06-15 13:37:08 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:37:08 | INFO | Fetching OQaD data for run_date=2026-06-03
|
|
2026-06-15 13:37:12 | INFO | Fetched 455 rows
|
|
2026-06-15 13:37:12 | INFO | Fetched 455 rows
|
|
2026-06-15 13:37:12 | INFO | Deleting from OQaD
|
|
2026-06-15 13:37:12 | INFO | OQaD: inserted 455 rows into ClickHouse
|
|
2026-06-15 13:37:12 | INFO | OQaD loaded successfully (455 rows)
|
|
2026-06-15 13:37:12 | INFO | ================================================================================
|
|
2026-06-15 13:37:12 | INFO | Processing Table: Survey
|
|
2026-06-15 13:37:12 | INFO | Fetching Survey data for 808 MIDs
|
|
2026-06-15 13:37:13 | INFO | Fetched 139 Survey rows
|
|
2026-06-15 13:37:13 | INFO | Fetched 139 rows
|
|
2026-06-15 13:37:13 | INFO | Deleting from Survey
|
|
2026-06-15 13:37:13 | INFO | Survey: inserted 139 rows into ClickHouse
|
|
2026-06-15 13:37:13 | INFO | Survey loaded successfully (139 rows)
|
|
2026-06-15 13:37:13 | INFO | ================================================================================
|
|
2026-06-15 13:37:13 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:37:13 | INFO | Fetching Additional Visibility data for 808 MIDs
|
|
2026-06-15 13:37:14 | INFO | Fetched 1,884 Additional Visibility rows
|
|
2026-06-15 13:37:14 | INFO | Fetched 1884 rows
|
|
2026-06-15 13:37:14 | INFO | Deleting from additional_visibility
|
|
2026-06-15 13:37:14 | INFO | additional_visibility: inserted 1,884 rows into ClickHouse
|
|
2026-06-15 13:37:14 | INFO | additional_visibility loaded successfully (1884 rows)
|
|
2026-06-15 13:37:14 | INFO | ================================================================================
|
|
2026-06-15 13:37:14 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:37:14 | INFO | Fetching coverage data for 808 MIDs
|
|
2026-06-15 13:37:15 | INFO | Fetched 751 rows from SQL Server
|
|
2026-06-15 13:37:15 | INFO | Fetched 751 rows
|
|
2026-06-15 13:37:15 | INFO | Deleting from Coverage
|
|
2026-06-15 13:37:15 | INFO | Coverage: inserted 751 rows into ClickHouse
|
|
2026-06-15 13:37:15 | INFO | Coverage loaded successfully (751 rows)
|
|
2026-06-15 13:37:15 | INFO | ================================================================================
|
|
2026-06-15 13:37:15 | INFO | Processing Table: Login
|
|
2026-06-15 13:37:15 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:37:16 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:37:16 | INFO | Fetched 454 rows
|
|
2026-06-15 13:37:16 | INFO | Deleting from Login
|
|
2026-06-15 13:37:16 | INFO | No delete logic required for Login
|
|
2026-06-15 13:37:16 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 13:37:16 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 13:37:16 | INFO | ================================================================================
|
|
2026-06-15 13:37:16 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 13:37:16 | INFO | Fetching Stock Details data for 808 MIDs
|
|
2026-06-15 13:37:19 | INFO | Fetched 40,840 Stock Details rows
|
|
2026-06-15 13:37:19 | INFO | Fetched 40840 rows
|
|
2026-06-15 13:37:19 | INFO | Deleting from Stock_Details
|
|
2026-06-15 13:37:20 | INFO | Stock_Details: inserted 40,840 rows into ClickHouse
|
|
2026-06-15 13:37:20 | INFO | Stock_Details loaded successfully (40840 rows)
|
|
2026-06-15 13:37:20 | INFO | ================================================================================
|
|
2026-06-15 13:37:20 | INFO | Processing Table: Attendance
|
|
2026-06-15 13:37:20 | INFO | Fetching Attendance data from 2026-05-19 to 2026-06-03
|
|
2026-06-15 13:37:21 | INFO | Fetched 12,432 attendance rows for 579 employees
|
|
2026-06-15 13:37:21 | INFO | Fetched 12432 rows
|
|
2026-06-15 13:37:21 | INFO | Truncated table Attendance
|
|
2026-06-15 13:37:21 | INFO | Attendance: inserted 12,432 rows into ClickHouse
|
|
2026-06-15 13:37:21 | INFO | Attendance loaded successfully (12432 rows)
|
|
2026-06-15 13:37:21 | INFO | ================================================================================
|
|
2026-06-15 13:37:21 | INFO | Processing Table: Store_Master
|
|
2026-06-15 13:37:21 | INFO | Fetching Store Master data
|
|
2026-06-15 13:37:22 | INFO | Fetched 5,984 stores
|
|
2026-06-15 13:37:22 | INFO | Fetched 5984 rows
|
|
2026-06-15 13:37:22 | INFO | Truncated table Store_Master
|
|
2026-06-15 13:37:22 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 13:37:22 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 13:37:22 | INFO | ================================================================================
|
|
2026-06-15 13:37:22 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 13:37:22 | INFO | Fetching SKU Master data
|
|
2026-06-15 13:37:22 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 13:37:22 | INFO | Fetched 160 rows
|
|
2026-06-15 13:37:23 | INFO | Truncated table SKU_Master
|
|
2026-06-15 13:37:23 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 13:37:23 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 13:37:23 | INFO | ================================================================================
|
|
2026-06-15 13:37:23 | INFO | Processing Table: display_master
|
|
2026-06-15 13:37:23 | INFO | Fetching Display Master data
|
|
2026-06-15 13:37:23 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 13:37:23 | INFO | Fetched 135 rows
|
|
2026-06-15 13:37:23 | INFO | Truncated table display_master
|
|
2026-06-15 13:37:23 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 13:37:23 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 13:37:23 | INFO | ================================================================================
|
|
2026-06-15 13:37:23 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 13:37:23 | INFO | Fetching Employee Master data
|
|
2026-06-15 13:37:24 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 13:37:24 | INFO | Fetched 2273 rows
|
|
2026-06-15 13:37:24 | INFO | Truncated table Employee_Master
|
|
2026-06-15 13:37:25 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 13:37:25 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 13:37:25 | INFO | ================================================================================
|
|
2026-06-15 13:37:25 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 13:37:25 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 13:37:25 | INFO | Fetched 21,141 Journey Plan records
|
|
2026-06-15 13:37:25 | INFO | Fetched 21141 rows
|
|
2026-06-15 13:37:25 | INFO | Deleting from Journey_Plan
|
|
2026-06-15 13:37:25 | INFO | Journey_Plan: inserted 21,141 rows into ClickHouse
|
|
2026-06-15 13:37:25 | INFO | Journey_Plan loaded successfully (21141 rows)
|
|
2026-06-15 13:37:25 | INFO | ================================================================================
|
|
2026-06-15 13:37:25 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 13:37:25 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 13:37:25 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 13:37:25 | INFO | Fetched 29 rows
|
|
2026-06-15 13:37:25 | INFO | Truncated table coverage_remarks
|
|
2026-06-15 13:37:26 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 13:37:26 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 13:37:26 | INFO | ================================================================================
|
|
2026-06-15 13:37:26 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 13:37:26 | INFO | Fetching Mapping Store Visibility for 2026-06-03
|
|
2026-06-15 13:37:26 | INFO | Fetched 0 Mapping Store Visibility records
|
|
2026-06-15 13:37:26 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-15 13:37:26 | INFO | ================================================================================
|
|
2026-06-15 13:37:26 | INFO | Processing Table: Master_VisibilityReason
|
|
2026-06-15 13:37:26 | INFO | Fetching Master Visibility Reason data
|
|
2026-06-15 13:37:26 | INFO | Fetched 17 Master Visibility Reason records
|
|
2026-06-15 13:37:26 | INFO | Fetched 17 rows
|
|
2026-06-15 13:37:26 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-15 13:37:26 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-15 13:37:26 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-15 13:37:26 | INFO | ================================================================================
|
|
2026-06-15 13:37:26 | INFO | Processing Table: Master_VisibilityDefinition
|
|
2026-06-15 13:37:26 | INFO | Fetching Master Visibility Definition data
|
|
2026-06-15 13:37:26 | INFO | Fetched 861 Master Visibility Definition records
|
|
2026-06-15 13:37:26 | INFO | Fetched 861 rows
|
|
2026-06-15 13:37:26 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-15 13:37:26 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-15 13:37:26 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-15 13:37:26 | INFO | ================================================================================
|
|
2026-06-15 13:37:26 | INFO | Processing Table: Web_Logins
|
|
2026-06-15 13:37:26 | INFO | Fetching Web Login data for 2026-06-03
|
|
2026-06-15 13:37:27 | INFO | Fetched 200 Web Login records
|
|
2026-06-15 13:37:27 | INFO | Fetched 200 rows
|
|
2026-06-15 13:37:27 | INFO | Deleting from Web_Logins
|
|
2026-06-15 13:37:27 | INFO | Web_Logins: inserted 200 rows into ClickHouse
|
|
2026-06-15 13:37:27 | INFO | Web_Logins loaded successfully (200 rows)
|
|
2026-06-15 13:37:27 | INFO | ================================================================================
|
|
2026-06-15 13:37:27 | INFO | Processing Table: Promotion
|
|
2026-06-15 13:37:27 | INFO | Fetching Promotion data for 808 MIDs
|
|
2026-06-15 13:37:32 | INFO | Fetched 8,162 Promotion records
|
|
2026-06-15 13:37:32 | INFO | Fetched 8162 rows
|
|
2026-06-15 13:37:32 | INFO | Deleting from Promotion
|
|
2026-06-15 13:37:32 | INFO | Promotion: inserted 8,162 rows into ClickHouse
|
|
2026-06-15 13:37:32 | INFO | Promotion loaded successfully (8162 rows)
|
|
2026-06-15 13:37:32 | INFO | ================================================================================
|
|
2026-06-15 13:37:32 | INFO | Processing Table: PaidVisibility
|
|
2026-06-15 13:37:32 | INFO | Fetching Paid Visibility data for 808 MIDs
|
|
2026-06-15 13:37:34 | INFO | Fetched 1,064 Paid Visibility records
|
|
2026-06-15 13:37:34 | INFO | Fetched 1064 rows
|
|
2026-06-15 13:37:34 | INFO | Deleting from PaidVisibility
|
|
2026-06-15 13:37:34 | INFO | PaidVisibility: inserted 1,064 rows into ClickHouse
|
|
2026-06-15 13:37:34 | INFO | PaidVisibility loaded successfully (1064 rows)
|
|
2026-06-15 13:37:34 | INFO | ================================================================================
|
|
2026-06-15 13:37:34 | INFO | Processing Table: Master_Salesterritorylayer
|
|
2026-06-15 13:37:34 | INFO | Fetching Master Sales Territory Layer data
|
|
2026-06-15 13:37:34 | INFO | Fetched 33 Master Sales Territory Layer records
|
|
2026-06-15 13:37:34 | INFO | Fetched 33 rows
|
|
2026-06-15 13:37:34 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-15 13:37:34 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-15 13:37:34 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-15 13:37:34 | INFO | ================================================================================
|
|
2026-06-15 13:37:34 | INFO | Pipeline Completed Successfully
|
|
2026-06-15 13:37:34 | INFO | ================================================================================
|
|
2026-06-15 13:37:43 | INFO | ================================================================================
|
|
2026-06-15 13:37:43 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:37:43 | INFO | Pipeline Run Date: 2026-06-03
|
|
2026-06-15 13:37:43 | INFO | Connecting to databases...
|
|
2026-06-15 13:37:44 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001914F877A10>
|
|
2026-06-15 13:37:44 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000019150A4D310>
|
|
2026-06-15 13:37:45 | INFO | Database connections established
|
|
2026-06-15 13:37:45 | INFO | Collecting MIDs for: 2026-06-03
|
|
2026-06-15 13:37:45 | INFO | Found 808 MIDs
|
|
2026-06-15 13:37:45 | INFO | ================================================================================
|
|
2026-06-15 13:37:45 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:37:45 | INFO | Fetching data for 808 MIDs
|
|
2026-06-15 13:37:46 | INFO | Fetched 3,565 rows from SQL Server
|
|
2026-06-15 13:37:46 | INFO | Fetched 3565 rows
|
|
2026-06-15 13:37:46 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 13:37:46 | INFO | SOS_OneApp: inserted 3,565 rows into ClickHouse
|
|
2026-06-15 13:37:46 | INFO | SOS_OneApp loaded successfully (3565 rows)
|
|
2026-06-15 13:37:46 | INFO | ================================================================================
|
|
2026-06-15 13:37:46 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:37:46 | INFO | Fetching OQaD data for run_date=2026-06-03
|
|
2026-06-15 13:37:49 | INFO | Fetched 455 rows
|
|
2026-06-15 13:37:49 | INFO | Fetched 455 rows
|
|
2026-06-15 13:37:49 | INFO | Deleting from OQaD
|
|
2026-06-15 13:37:49 | INFO | OQaD: inserted 455 rows into ClickHouse
|
|
2026-06-15 13:37:49 | INFO | OQaD loaded successfully (455 rows)
|
|
2026-06-15 13:37:49 | INFO | ================================================================================
|
|
2026-06-15 13:37:49 | INFO | Processing Table: Survey
|
|
2026-06-15 13:37:49 | INFO | Fetching Survey data for 808 MIDs
|
|
2026-06-15 13:37:49 | INFO | Fetched 139 Survey rows
|
|
2026-06-15 13:37:49 | INFO | Fetched 139 rows
|
|
2026-06-15 13:37:49 | INFO | Deleting from Survey
|
|
2026-06-15 13:37:49 | INFO | Survey: inserted 139 rows into ClickHouse
|
|
2026-06-15 13:37:49 | INFO | Survey loaded successfully (139 rows)
|
|
2026-06-15 13:37:49 | INFO | ================================================================================
|
|
2026-06-15 13:37:49 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:37:49 | INFO | Fetching Additional Visibility data for 808 MIDs
|
|
2026-06-15 13:37:50 | INFO | Fetched 1,884 Additional Visibility rows
|
|
2026-06-15 13:37:50 | INFO | Fetched 1884 rows
|
|
2026-06-15 13:37:50 | INFO | Deleting from additional_visibility
|
|
2026-06-15 13:37:50 | INFO | additional_visibility: inserted 1,884 rows into ClickHouse
|
|
2026-06-15 13:37:50 | INFO | additional_visibility loaded successfully (1884 rows)
|
|
2026-06-15 13:37:50 | INFO | ================================================================================
|
|
2026-06-15 13:37:50 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:37:50 | INFO | Fetching coverage data for 808 MIDs
|
|
2026-06-15 13:37:50 | INFO | Fetched 751 rows from SQL Server
|
|
2026-06-15 13:37:50 | INFO | Fetched 751 rows
|
|
2026-06-15 13:37:50 | INFO | Deleting from Coverage
|
|
2026-06-15 13:37:50 | INFO | Coverage: inserted 751 rows into ClickHouse
|
|
2026-06-15 13:37:50 | INFO | Coverage loaded successfully (751 rows)
|
|
2026-06-15 13:37:50 | INFO | ================================================================================
|
|
2026-06-15 13:37:50 | INFO | Processing Table: Login
|
|
2026-06-15 13:37:50 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:37:51 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:37:51 | INFO | Fetched 454 rows
|
|
2026-06-15 13:37:51 | INFO | Deleting from Login
|
|
2026-06-15 13:37:51 | INFO | No delete logic required for Login
|
|
2026-06-15 13:37:51 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 13:37:51 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 13:37:51 | INFO | ================================================================================
|
|
2026-06-15 13:37:51 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 13:37:51 | INFO | Fetching Stock Details data for 808 MIDs
|
|
2026-06-15 13:37:55 | INFO | Fetched 40,840 Stock Details rows
|
|
2026-06-15 13:37:55 | INFO | Fetched 40840 rows
|
|
2026-06-15 13:37:55 | INFO | Deleting from Stock_Details
|
|
2026-06-15 13:37:55 | INFO | Stock_Details: inserted 40,840 rows into ClickHouse
|
|
2026-06-15 13:37:55 | INFO | Stock_Details loaded successfully (40840 rows)
|
|
2026-06-15 13:37:55 | INFO | ================================================================================
|
|
2026-06-15 13:37:55 | INFO | Processing Table: Attendance
|
|
2026-06-15 13:37:55 | INFO | Fetching Attendance data from 2026-05-19 to 2026-06-03
|
|
2026-06-15 13:37:56 | INFO | Fetched 12,432 attendance rows for 579 employees
|
|
2026-06-15 13:37:56 | INFO | Fetched 12432 rows
|
|
2026-06-15 13:37:57 | INFO | Truncated table Attendance
|
|
2026-06-15 13:37:57 | INFO | Attendance: inserted 12,432 rows into ClickHouse
|
|
2026-06-15 13:37:57 | INFO | Attendance loaded successfully (12432 rows)
|
|
2026-06-15 13:37:57 | INFO | ================================================================================
|
|
2026-06-15 13:37:57 | INFO | Processing Table: Store_Master
|
|
2026-06-15 13:37:57 | INFO | Fetching Store Master data
|
|
2026-06-15 13:37:58 | INFO | Fetched 5,984 stores
|
|
2026-06-15 13:37:58 | INFO | Fetched 5984 rows
|
|
2026-06-15 13:37:58 | INFO | Truncated table Store_Master
|
|
2026-06-15 13:37:59 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 13:37:59 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 13:37:59 | INFO | ================================================================================
|
|
2026-06-15 13:37:59 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 13:37:59 | INFO | Fetching SKU Master data
|
|
2026-06-15 13:37:59 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 13:37:59 | INFO | Fetched 160 rows
|
|
2026-06-15 13:37:59 | INFO | Truncated table SKU_Master
|
|
2026-06-15 13:37:59 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 13:37:59 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 13:37:59 | INFO | ================================================================================
|
|
2026-06-15 13:37:59 | INFO | Processing Table: display_master
|
|
2026-06-15 13:37:59 | INFO | Fetching Display Master data
|
|
2026-06-15 13:37:59 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 13:37:59 | INFO | Fetched 135 rows
|
|
2026-06-15 13:37:59 | INFO | Truncated table display_master
|
|
2026-06-15 13:37:59 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 13:37:59 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 13:37:59 | INFO | ================================================================================
|
|
2026-06-15 13:37:59 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 13:37:59 | INFO | Fetching Employee Master data
|
|
2026-06-15 13:38:00 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 13:38:00 | INFO | Fetched 2273 rows
|
|
2026-06-15 13:38:00 | INFO | Truncated table Employee_Master
|
|
2026-06-15 13:38:00 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 13:38:00 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 13:38:00 | INFO | ================================================================================
|
|
2026-06-15 13:38:00 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 13:38:00 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 13:38:00 | INFO | Fetched 21,141 Journey Plan records
|
|
2026-06-15 13:38:00 | INFO | Fetched 21141 rows
|
|
2026-06-15 13:38:00 | INFO | Deleting from Journey_Plan
|
|
2026-06-15 13:38:01 | INFO | Journey_Plan: inserted 21,141 rows into ClickHouse
|
|
2026-06-15 13:38:01 | INFO | Journey_Plan loaded successfully (21141 rows)
|
|
2026-06-15 13:38:01 | INFO | ================================================================================
|
|
2026-06-15 13:38:01 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 13:38:01 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 13:38:01 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 13:38:01 | INFO | Fetched 29 rows
|
|
2026-06-15 13:38:01 | INFO | Truncated table coverage_remarks
|
|
2026-06-15 13:38:01 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 13:38:01 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 13:38:01 | INFO | ================================================================================
|
|
2026-06-15 13:38:01 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 13:38:01 | INFO | Fetching Mapping Store Visibility for 2026-06-03
|
|
2026-06-15 13:38:01 | INFO | Fetched 0 Mapping Store Visibility records
|
|
2026-06-15 13:38:01 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-15 13:38:01 | INFO | ================================================================================
|
|
2026-06-15 13:38:01 | INFO | Processing Table: Master_VisibilityReason
|
|
2026-06-15 13:38:01 | INFO | Fetching Master Visibility Reason data
|
|
2026-06-15 13:38:01 | INFO | Fetched 17 Master Visibility Reason records
|
|
2026-06-15 13:38:01 | INFO | Fetched 17 rows
|
|
2026-06-15 13:38:01 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-15 13:38:01 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-15 13:38:01 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-15 13:38:01 | INFO | ================================================================================
|
|
2026-06-15 13:38:01 | INFO | Processing Table: Master_VisibilityDefinition
|
|
2026-06-15 13:38:01 | INFO | Fetching Master Visibility Definition data
|
|
2026-06-15 13:38:02 | INFO | Fetched 861 Master Visibility Definition records
|
|
2026-06-15 13:38:02 | INFO | Fetched 861 rows
|
|
2026-06-15 13:38:02 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-15 13:38:02 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-15 13:38:02 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-15 13:38:02 | INFO | ================================================================================
|
|
2026-06-15 13:38:02 | INFO | Processing Table: Web_Logins
|
|
2026-06-15 13:38:02 | INFO | Fetching Web Login data for 2026-06-03
|
|
2026-06-15 13:38:02 | INFO | Fetched 200 Web Login records
|
|
2026-06-15 13:38:02 | INFO | Fetched 200 rows
|
|
2026-06-15 13:38:02 | INFO | Deleting from Web_Logins
|
|
2026-06-15 13:38:02 | INFO | Web_Logins: inserted 200 rows into ClickHouse
|
|
2026-06-15 13:38:02 | INFO | Web_Logins loaded successfully (200 rows)
|
|
2026-06-15 13:38:02 | INFO | ================================================================================
|
|
2026-06-15 13:38:02 | INFO | Processing Table: Promotion
|
|
2026-06-15 13:38:02 | INFO | Fetching Promotion data for 808 MIDs
|
|
2026-06-15 13:38:04 | INFO | Fetched 8,162 Promotion records
|
|
2026-06-15 13:38:04 | INFO | Fetched 8162 rows
|
|
2026-06-15 13:38:04 | INFO | Deleting from Promotion
|
|
2026-06-15 13:38:04 | INFO | Promotion: inserted 8,162 rows into ClickHouse
|
|
2026-06-15 13:38:04 | INFO | Promotion loaded successfully (8162 rows)
|
|
2026-06-15 13:38:04 | INFO | ================================================================================
|
|
2026-06-15 13:38:04 | INFO | Processing Table: PaidVisibility
|
|
2026-06-15 13:38:04 | INFO | Fetching Paid Visibility data for 808 MIDs
|
|
2026-06-15 13:38:05 | INFO | Fetched 1,064 Paid Visibility records
|
|
2026-06-15 13:38:05 | INFO | Fetched 1064 rows
|
|
2026-06-15 13:38:06 | INFO | Deleting from PaidVisibility
|
|
2026-06-15 13:38:06 | INFO | PaidVisibility: inserted 1,064 rows into ClickHouse
|
|
2026-06-15 13:38:06 | INFO | PaidVisibility loaded successfully (1064 rows)
|
|
2026-06-15 13:38:06 | INFO | ================================================================================
|
|
2026-06-15 13:38:06 | INFO | Processing Table: Master_Salesterritorylayer
|
|
2026-06-15 13:38:06 | INFO | Fetching Master Sales Territory Layer data
|
|
2026-06-15 13:38:06 | INFO | Fetched 33 Master Sales Territory Layer records
|
|
2026-06-15 13:38:06 | INFO | Fetched 33 rows
|
|
2026-06-15 13:38:06 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-15 13:38:07 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-15 13:38:07 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-15 13:38:07 | INFO | ================================================================================
|
|
2026-06-15 13:38:07 | INFO | Pipeline Completed Successfully
|
|
2026-06-15 13:38:07 | INFO | ================================================================================
|
|
2026-06-15 13:40:53 | INFO | ================================================================================
|
|
2026-06-15 13:40:53 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:40:53 | INFO | Pipeline Run Date: 2026-06-03
|
|
2026-06-15 13:40:53 | INFO | Connecting to databases...
|
|
2026-06-15 13:40:53 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002105349B8B0>
|
|
2026-06-15 13:40:55 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000210546D91D0>
|
|
2026-06-15 13:40:56 | INFO | Database connections established
|
|
2026-06-15 13:40:56 | INFO | Collecting MIDs for: 2026-06-03
|
|
2026-06-15 13:40:56 | INFO | Found 808 MIDs
|
|
2026-06-15 13:40:56 | INFO | ================================================================================
|
|
2026-06-15 13:40:56 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:40:56 | INFO | Fetching data for 808 MIDs
|
|
2026-06-15 13:40:57 | INFO | Fetched 3,565 rows from SQL Server
|
|
2026-06-15 13:40:57 | INFO | Fetched 3565 rows
|
|
2026-06-15 13:40:57 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 13:40:57 | INFO | SOS_OneApp: inserted 3,565 rows into ClickHouse
|
|
2026-06-15 13:40:57 | INFO | SOS_OneApp loaded successfully (3565 rows)
|
|
2026-06-15 13:40:57 | INFO | ================================================================================
|
|
2026-06-15 13:40:57 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:40:57 | INFO | Fetching OQaD data for run_date=2026-06-03
|
|
2026-06-15 13:41:00 | INFO | Fetched 455 rows
|
|
2026-06-15 13:41:00 | INFO | Fetched 455 rows
|
|
2026-06-15 13:41:00 | INFO | Deleting from OQaD
|
|
2026-06-15 13:41:00 | INFO | OQaD: inserted 455 rows into ClickHouse
|
|
2026-06-15 13:41:00 | INFO | OQaD loaded successfully (455 rows)
|
|
2026-06-15 13:41:00 | INFO | ================================================================================
|
|
2026-06-15 13:41:00 | INFO | Processing Table: Survey
|
|
2026-06-15 13:41:00 | INFO | Fetching Survey data for 808 MIDs
|
|
2026-06-15 13:41:00 | INFO | Fetched 139 Survey rows
|
|
2026-06-15 13:41:00 | INFO | Fetched 139 rows
|
|
2026-06-15 13:41:00 | INFO | Deleting from Survey
|
|
2026-06-15 13:41:00 | INFO | Survey: inserted 139 rows into ClickHouse
|
|
2026-06-15 13:41:00 | INFO | Survey loaded successfully (139 rows)
|
|
2026-06-15 13:41:00 | INFO | ================================================================================
|
|
2026-06-15 13:41:00 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:41:00 | INFO | Fetching Additional Visibility data for 808 MIDs
|
|
2026-06-15 13:41:01 | INFO | Fetched 1,884 Additional Visibility rows
|
|
2026-06-15 13:41:01 | INFO | Fetched 1884 rows
|
|
2026-06-15 13:41:01 | INFO | Deleting from additional_visibility
|
|
2026-06-15 13:41:01 | INFO | additional_visibility: inserted 1,884 rows into ClickHouse
|
|
2026-06-15 13:41:01 | INFO | additional_visibility loaded successfully (1884 rows)
|
|
2026-06-15 13:41:01 | INFO | ================================================================================
|
|
2026-06-15 13:41:01 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:41:01 | INFO | Fetching coverage data for 808 MIDs
|
|
2026-06-15 13:41:01 | INFO | Fetched 751 rows from SQL Server
|
|
2026-06-15 13:41:01 | INFO | Fetched 751 rows
|
|
2026-06-15 13:41:01 | INFO | Deleting from Coverage
|
|
2026-06-15 13:41:02 | INFO | Coverage: inserted 751 rows into ClickHouse
|
|
2026-06-15 13:41:02 | INFO | Coverage loaded successfully (751 rows)
|
|
2026-06-15 13:41:02 | INFO | ================================================================================
|
|
2026-06-15 13:41:02 | INFO | Processing Table: Login
|
|
2026-06-15 13:41:02 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:41:02 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:41:02 | INFO | Fetched 454 rows
|
|
2026-06-15 13:41:02 | INFO | Deleting from Login
|
|
2026-06-15 13:41:02 | INFO | No delete logic required for Login
|
|
2026-06-15 13:41:02 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 13:41:02 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 13:41:02 | INFO | ================================================================================
|
|
2026-06-15 13:41:02 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 13:41:02 | INFO | Fetching Stock Details data for 808 MIDs
|
|
2026-06-15 13:41:07 | INFO | Fetched 40,840 Stock Details rows
|
|
2026-06-15 13:41:07 | INFO | Fetched 40840 rows
|
|
2026-06-15 13:41:08 | INFO | Deleting from Stock_Details
|
|
2026-06-15 13:41:08 | INFO | Stock_Details: inserted 40,840 rows into ClickHouse
|
|
2026-06-15 13:41:08 | INFO | Stock_Details loaded successfully (40840 rows)
|
|
2026-06-15 13:41:08 | INFO | ================================================================================
|
|
2026-06-15 13:41:08 | INFO | Processing Table: Attendance
|
|
2026-06-15 13:41:08 | INFO | Fetching Attendance data from 2026-05-19 to 2026-06-03
|
|
2026-06-15 13:41:10 | INFO | Fetched 12,432 attendance rows for 579 employees
|
|
2026-06-15 13:41:10 | INFO | Fetched 12432 rows
|
|
2026-06-15 13:41:10 | INFO | Truncated table Attendance
|
|
2026-06-15 13:41:10 | INFO | Attendance: inserted 12,432 rows into ClickHouse
|
|
2026-06-15 13:41:10 | INFO | Attendance loaded successfully (12432 rows)
|
|
2026-06-15 13:41:10 | INFO | ================================================================================
|
|
2026-06-15 13:41:10 | INFO | Processing Table: Store_Master
|
|
2026-06-15 13:41:10 | INFO | Fetching Store Master data
|
|
2026-06-15 13:41:18 | INFO | Fetched 5,984 stores
|
|
2026-06-15 13:41:18 | INFO | Fetched 5984 rows
|
|
2026-06-15 13:41:18 | INFO | Truncated table Store_Master
|
|
2026-06-15 13:41:18 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 13:41:18 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 13:41:18 | INFO | ================================================================================
|
|
2026-06-15 13:41:18 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 13:41:18 | INFO | Fetching SKU Master data
|
|
2026-06-15 13:41:18 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 13:41:18 | INFO | Fetched 160 rows
|
|
2026-06-15 13:41:18 | INFO | Truncated table SKU_Master
|
|
2026-06-15 13:41:19 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 13:41:19 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 13:41:19 | INFO | ================================================================================
|
|
2026-06-15 13:41:19 | INFO | Processing Table: display_master
|
|
2026-06-15 13:41:19 | INFO | Fetching Display Master data
|
|
2026-06-15 13:41:19 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 13:41:19 | INFO | Fetched 135 rows
|
|
2026-06-15 13:41:19 | INFO | Truncated table display_master
|
|
2026-06-15 13:41:19 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 13:41:19 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 13:41:19 | INFO | ================================================================================
|
|
2026-06-15 13:41:19 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 13:41:19 | INFO | Fetching Employee Master data
|
|
2026-06-15 13:41:20 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 13:41:20 | INFO | Fetched 2273 rows
|
|
2026-06-15 13:41:20 | INFO | Truncated table Employee_Master
|
|
2026-06-15 13:41:20 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 13:41:20 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 13:41:20 | INFO | ================================================================================
|
|
2026-06-15 13:41:20 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 13:41:20 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 13:41:21 | INFO | Fetched 21,141 Journey Plan records
|
|
2026-06-15 13:41:21 | INFO | Fetched 21141 rows
|
|
2026-06-15 13:41:21 | INFO | Deleting from Journey_Plan
|
|
2026-06-15 13:41:21 | INFO | Journey_Plan: inserted 21,141 rows into ClickHouse
|
|
2026-06-15 13:41:21 | INFO | Journey_Plan loaded successfully (21141 rows)
|
|
2026-06-15 13:41:21 | INFO | ================================================================================
|
|
2026-06-15 13:41:21 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 13:41:21 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 13:41:21 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 13:41:21 | INFO | Fetched 29 rows
|
|
2026-06-15 13:41:21 | INFO | Truncated table coverage_remarks
|
|
2026-06-15 13:41:22 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 13:41:22 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 13:41:22 | INFO | ================================================================================
|
|
2026-06-15 13:41:22 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 13:41:22 | INFO | Fetching Mapping Store Visibility for 2026-06-03
|
|
2026-06-15 13:41:22 | INFO | Fetched 0 Mapping Store Visibility records
|
|
2026-06-15 13:41:22 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-15 13:41:22 | INFO | ================================================================================
|
|
2026-06-15 13:41:22 | INFO | Processing Table: Master_VisibilityReason
|
|
2026-06-15 13:41:22 | INFO | Fetching Master Visibility Reason data
|
|
2026-06-15 13:41:22 | INFO | Fetched 17 Master Visibility Reason records
|
|
2026-06-15 13:41:22 | INFO | Fetched 17 rows
|
|
2026-06-15 13:41:22 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-15 13:41:22 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-15 13:41:22 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-15 13:41:22 | INFO | ================================================================================
|
|
2026-06-15 13:41:22 | INFO | Processing Table: Master_VisibilityDefinition
|
|
2026-06-15 13:41:22 | INFO | Fetching Master Visibility Definition data
|
|
2026-06-15 13:41:22 | INFO | Fetched 861 Master Visibility Definition records
|
|
2026-06-15 13:41:22 | INFO | Fetched 861 rows
|
|
2026-06-15 13:41:22 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-15 13:41:22 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-15 13:41:22 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-15 13:41:22 | INFO | ================================================================================
|
|
2026-06-15 13:41:22 | INFO | Processing Table: Web_Logins
|
|
2026-06-15 13:41:22 | INFO | Fetching Web Login data for 2026-06-03
|
|
2026-06-15 13:41:23 | INFO | Fetched 200 Web Login records
|
|
2026-06-15 13:41:23 | INFO | Fetched 200 rows
|
|
2026-06-15 13:41:23 | INFO | Deleting from Web_Logins
|
|
2026-06-15 13:41:23 | INFO | Web_Logins: inserted 200 rows into ClickHouse
|
|
2026-06-15 13:41:23 | INFO | Web_Logins loaded successfully (200 rows)
|
|
2026-06-15 13:41:23 | INFO | ================================================================================
|
|
2026-06-15 13:41:23 | INFO | Processing Table: Promotion
|
|
2026-06-15 13:41:23 | INFO | Fetching Promotion data for 808 MIDs
|
|
2026-06-15 13:41:28 | INFO | Fetched 8,162 Promotion records
|
|
2026-06-15 13:41:28 | INFO | Fetched 8162 rows
|
|
2026-06-15 13:41:28 | INFO | Deleting from Promotion
|
|
2026-06-15 13:41:28 | INFO | Promotion: inserted 8,162 rows into ClickHouse
|
|
2026-06-15 13:41:28 | INFO | Promotion loaded successfully (8162 rows)
|
|
2026-06-15 13:41:28 | INFO | ================================================================================
|
|
2026-06-15 13:41:28 | INFO | Processing Table: PaidVisibility
|
|
2026-06-15 13:41:28 | INFO | Fetching Paid Visibility data for 808 MIDs
|
|
2026-06-15 13:41:29 | INFO | Fetched 1,064 Paid Visibility records
|
|
2026-06-15 13:41:29 | INFO | Fetched 1064 rows
|
|
2026-06-15 13:41:29 | INFO | Deleting from PaidVisibility
|
|
2026-06-15 13:41:30 | INFO | PaidVisibility: inserted 1,064 rows into ClickHouse
|
|
2026-06-15 13:41:30 | INFO | PaidVisibility loaded successfully (1064 rows)
|
|
2026-06-15 13:41:30 | INFO | ================================================================================
|
|
2026-06-15 13:41:30 | INFO | Processing Table: Master_Salesterritorylayer
|
|
2026-06-15 13:41:30 | INFO | Fetching Master Sales Territory Layer data
|
|
2026-06-15 13:41:30 | INFO | Fetched 33 Master Sales Territory Layer records
|
|
2026-06-15 13:41:30 | INFO | Fetched 33 rows
|
|
2026-06-15 13:41:30 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-15 13:41:30 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-15 13:41:30 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-15 13:41:30 | INFO | ================================================================================
|
|
2026-06-15 13:41:30 | INFO | Pipeline Completed Successfully
|
|
2026-06-15 13:41:30 | INFO | ================================================================================
|
|
2026-06-15 13:46:09 | INFO | ================================================================================
|
|
2026-06-15 13:46:09 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:46:09 | INFO | Pipeline Run Date: 2026-06-03
|
|
2026-06-15 13:46:09 | INFO | Connecting to databases...
|
|
2026-06-15 13:46:10 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x0000023606FAB8B0>
|
|
2026-06-15 13:46:11 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x00000236081B91D0>
|
|
2026-06-15 13:46:11 | INFO | Database connections established
|
|
2026-06-15 13:46:11 | INFO | Collecting MIDs for: 2026-06-03
|
|
2026-06-15 13:46:11 | INFO | Found 808 MIDs
|
|
2026-06-15 13:46:11 | INFO | ================================================================================
|
|
2026-06-15 13:46:11 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:46:11 | INFO | Fetching data for 808 MIDs
|
|
2026-06-15 13:46:12 | INFO | Fetched 3,565 rows from SQL Server
|
|
2026-06-15 13:46:12 | INFO | Fetched 3565 rows
|
|
2026-06-15 13:46:12 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 13:46:13 | INFO | SOS_OneApp: inserted 3,565 rows into ClickHouse
|
|
2026-06-15 13:46:13 | INFO | SOS_OneApp loaded successfully (3565 rows)
|
|
2026-06-15 13:46:13 | INFO | ================================================================================
|
|
2026-06-15 13:46:13 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:46:13 | INFO | Fetching OQaD data for run_date=2026-06-03
|
|
2026-06-15 13:46:15 | INFO | Fetched 455 rows
|
|
2026-06-15 13:46:15 | INFO | Fetched 455 rows
|
|
2026-06-15 13:46:15 | INFO | Deleting from OQaD
|
|
2026-06-15 13:46:15 | INFO | OQaD: inserted 455 rows into ClickHouse
|
|
2026-06-15 13:46:15 | INFO | OQaD loaded successfully (455 rows)
|
|
2026-06-15 13:46:15 | INFO | ================================================================================
|
|
2026-06-15 13:46:15 | INFO | Processing Table: Survey
|
|
2026-06-15 13:46:15 | INFO | Fetching Survey data for 808 MIDs
|
|
2026-06-15 13:46:15 | INFO | Fetched 139 Survey rows
|
|
2026-06-15 13:46:15 | INFO | Fetched 139 rows
|
|
2026-06-15 13:46:15 | INFO | Deleting from Survey
|
|
2026-06-15 13:46:16 | INFO | Survey: inserted 139 rows into ClickHouse
|
|
2026-06-15 13:46:16 | INFO | Survey loaded successfully (139 rows)
|
|
2026-06-15 13:46:16 | INFO | ================================================================================
|
|
2026-06-15 13:46:16 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:46:16 | INFO | Fetching Additional Visibility data for 808 MIDs
|
|
2026-06-15 13:46:16 | INFO | Fetched 1,884 Additional Visibility rows
|
|
2026-06-15 13:46:16 | INFO | Fetched 1884 rows
|
|
2026-06-15 13:46:16 | INFO | Deleting from additional_visibility
|
|
2026-06-15 13:46:16 | INFO | additional_visibility: inserted 1,884 rows into ClickHouse
|
|
2026-06-15 13:46:16 | INFO | additional_visibility loaded successfully (1884 rows)
|
|
2026-06-15 13:46:16 | INFO | ================================================================================
|
|
2026-06-15 13:46:16 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:46:16 | INFO | Fetching coverage data for 808 MIDs
|
|
2026-06-15 13:46:16 | INFO | Fetched 751 rows from SQL Server
|
|
2026-06-15 13:46:16 | INFO | Fetched 751 rows
|
|
2026-06-15 13:46:16 | INFO | Deleting from Coverage
|
|
2026-06-15 13:46:17 | INFO | Coverage: inserted 751 rows into ClickHouse
|
|
2026-06-15 13:46:17 | INFO | Coverage loaded successfully (751 rows)
|
|
2026-06-15 13:46:17 | INFO | ================================================================================
|
|
2026-06-15 13:46:17 | INFO | Processing Table: Login
|
|
2026-06-15 13:46:17 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:46:17 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:46:17 | INFO | Fetched 454 rows
|
|
2026-06-15 13:46:17 | INFO | Deleting from Login
|
|
2026-06-15 13:46:17 | INFO | No delete logic required for Login
|
|
2026-06-15 13:46:18 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 13:46:18 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 13:46:18 | INFO | ================================================================================
|
|
2026-06-15 13:46:18 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 13:46:18 | INFO | Fetching Stock Details data for 808 MIDs
|
|
2026-06-15 13:46:22 | INFO | Fetched 40,840 Stock Details rows
|
|
2026-06-15 13:46:22 | INFO | Fetched 40840 rows
|
|
2026-06-15 13:46:22 | INFO | Deleting from Stock_Details
|
|
2026-06-15 13:46:23 | INFO | Stock_Details: inserted 40,840 rows into ClickHouse
|
|
2026-06-15 13:46:23 | INFO | Stock_Details loaded successfully (40840 rows)
|
|
2026-06-15 13:46:23 | INFO | ================================================================================
|
|
2026-06-15 13:46:23 | INFO | Processing Table: Attendance
|
|
2026-06-15 13:46:23 | INFO | Fetching Attendance data from 2026-05-19 to 2026-06-03
|
|
2026-06-15 13:46:25 | INFO | Fetched 12,432 attendance rows for 579 employees
|
|
2026-06-15 13:46:25 | INFO | Fetched 12432 rows
|
|
2026-06-15 13:46:25 | INFO | Truncated table Attendance
|
|
2026-06-15 13:46:25 | INFO | Attendance: inserted 12,432 rows into ClickHouse
|
|
2026-06-15 13:46:25 | INFO | Attendance loaded successfully (12432 rows)
|
|
2026-06-15 13:46:25 | INFO | ================================================================================
|
|
2026-06-15 13:46:25 | INFO | Processing Table: Store_Master
|
|
2026-06-15 13:46:25 | INFO | Fetching Store Master data
|
|
2026-06-15 13:46:29 | INFO | Fetched 5,984 stores
|
|
2026-06-15 13:46:29 | INFO | Fetched 5984 rows
|
|
2026-06-15 13:46:29 | INFO | Truncated table Store_Master
|
|
2026-06-15 13:46:29 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 13:46:29 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 13:46:29 | INFO | ================================================================================
|
|
2026-06-15 13:46:29 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 13:46:29 | INFO | Fetching SKU Master data
|
|
2026-06-15 13:46:29 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 13:46:29 | INFO | Fetched 160 rows
|
|
2026-06-15 13:46:30 | INFO | Truncated table SKU_Master
|
|
2026-06-15 13:46:30 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 13:46:30 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 13:46:30 | INFO | ================================================================================
|
|
2026-06-15 13:46:30 | INFO | Processing Table: display_master
|
|
2026-06-15 13:46:30 | INFO | Fetching Display Master data
|
|
2026-06-15 13:46:30 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 13:46:30 | INFO | Fetched 135 rows
|
|
2026-06-15 13:46:30 | INFO | Truncated table display_master
|
|
2026-06-15 13:46:30 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 13:46:30 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 13:46:30 | INFO | ================================================================================
|
|
2026-06-15 13:46:30 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 13:46:30 | INFO | Fetching Employee Master data
|
|
2026-06-15 13:46:31 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 13:46:31 | INFO | Fetched 2273 rows
|
|
2026-06-15 13:46:31 | INFO | Truncated table Employee_Master
|
|
2026-06-15 13:46:31 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 13:46:31 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 13:46:31 | INFO | ================================================================================
|
|
2026-06-15 13:46:31 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 13:46:31 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 13:46:31 | INFO | Fetched 21,142 Journey Plan records
|
|
2026-06-15 13:46:31 | INFO | Fetched 21142 rows
|
|
2026-06-15 13:46:32 | INFO | Deleting from Journey_Plan
|
|
2026-06-15 13:46:32 | INFO | Journey_Plan: inserted 21,142 rows into ClickHouse
|
|
2026-06-15 13:46:32 | INFO | Journey_Plan loaded successfully (21142 rows)
|
|
2026-06-15 13:46:32 | INFO | ================================================================================
|
|
2026-06-15 13:46:32 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 13:46:32 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 13:46:32 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 13:46:32 | INFO | Fetched 29 rows
|
|
2026-06-15 13:46:32 | INFO | Truncated table coverage_remarks
|
|
2026-06-15 13:46:33 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 13:46:33 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 13:46:33 | INFO | ================================================================================
|
|
2026-06-15 13:46:33 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 13:46:33 | INFO | Fetching Mapping Store Visibility for 2026-06-03
|
|
2026-06-15 13:46:33 | INFO | Fetched 0 Mapping Store Visibility records
|
|
2026-06-15 13:46:33 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-15 13:46:33 | INFO | ================================================================================
|
|
2026-06-15 13:46:33 | INFO | Processing Table: Master_VisibilityReason
|
|
2026-06-15 13:46:33 | INFO | Fetching Master Visibility Reason data
|
|
2026-06-15 13:46:33 | INFO | Fetched 17 Master Visibility Reason records
|
|
2026-06-15 13:46:33 | INFO | Fetched 17 rows
|
|
2026-06-15 13:46:33 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-15 13:46:33 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-15 13:46:33 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-15 13:46:33 | INFO | ================================================================================
|
|
2026-06-15 13:46:33 | INFO | Processing Table: Master_VisibilityDefinition
|
|
2026-06-15 13:46:33 | INFO | Fetching Master Visibility Definition data
|
|
2026-06-15 13:46:33 | INFO | Fetched 861 Master Visibility Definition records
|
|
2026-06-15 13:46:33 | INFO | Fetched 861 rows
|
|
2026-06-15 13:46:34 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-15 13:46:34 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-15 13:46:34 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-15 13:46:34 | INFO | ================================================================================
|
|
2026-06-15 13:46:34 | INFO | Processing Table: Web_Logins
|
|
2026-06-15 13:46:34 | INFO | Fetching Web Login data for 2026-06-03
|
|
2026-06-15 13:46:34 | INFO | Fetched 200 Web Login records
|
|
2026-06-15 13:46:34 | INFO | Fetched 200 rows
|
|
2026-06-15 13:46:34 | INFO | Deleting from Web_Logins
|
|
2026-06-15 13:46:34 | INFO | Web_Logins: inserted 200 rows into ClickHouse
|
|
2026-06-15 13:46:34 | INFO | Web_Logins loaded successfully (200 rows)
|
|
2026-06-15 13:46:34 | INFO | ================================================================================
|
|
2026-06-15 13:46:34 | INFO | Processing Table: Promotion
|
|
2026-06-15 13:46:34 | INFO | Fetching Promotion data for 808 MIDs
|
|
2026-06-15 13:46:37 | INFO | Fetched 8,162 Promotion records
|
|
2026-06-15 13:46:37 | INFO | Fetched 8162 rows
|
|
2026-06-15 13:46:37 | INFO | Deleting from Promotion
|
|
2026-06-15 13:46:37 | INFO | Promotion: inserted 8,162 rows into ClickHouse
|
|
2026-06-15 13:46:37 | INFO | Promotion loaded successfully (8162 rows)
|
|
2026-06-15 13:46:37 | INFO | ================================================================================
|
|
2026-06-15 13:46:37 | INFO | Processing Table: PaidVisibility
|
|
2026-06-15 13:46:37 | INFO | Fetching Paid Visibility data for 808 MIDs
|
|
2026-06-15 13:46:38 | INFO | Fetched 1,064 Paid Visibility records
|
|
2026-06-15 13:46:38 | INFO | Fetched 1064 rows
|
|
2026-06-15 13:46:38 | INFO | Deleting from PaidVisibility
|
|
2026-06-15 13:46:38 | INFO | PaidVisibility: inserted 1,064 rows into ClickHouse
|
|
2026-06-15 13:46:38 | INFO | PaidVisibility loaded successfully (1064 rows)
|
|
2026-06-15 13:46:38 | INFO | ================================================================================
|
|
2026-06-15 13:46:38 | INFO | Processing Table: Master_Salesterritorylayer
|
|
2026-06-15 13:46:38 | INFO | Fetching Master Sales Territory Layer data
|
|
2026-06-15 13:46:38 | INFO | Fetched 33 Master Sales Territory Layer records
|
|
2026-06-15 13:46:38 | INFO | Fetched 33 rows
|
|
2026-06-15 13:46:38 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-15 13:46:38 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-15 13:46:38 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-15 13:46:38 | INFO | ================================================================================
|
|
2026-06-15 13:46:38 | INFO | Pipeline Completed Successfully
|
|
2026-06-15 13:46:38 | INFO | ================================================================================
|
|
2026-06-15 13:46:43 | INFO | ================================================================================
|
|
2026-06-15 13:46:43 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:46:43 | INFO | Pipeline Run Date: 2026-06-03
|
|
2026-06-15 13:46:43 | INFO | Connecting to databases...
|
|
2026-06-15 13:46:43 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001CBF519B8B0>
|
|
2026-06-15 13:46:44 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001CBF63991D0>
|
|
2026-06-15 13:46:45 | INFO | Database connections established
|
|
2026-06-15 13:46:45 | INFO | Collecting MIDs for: 2026-06-03
|
|
2026-06-15 13:46:45 | INFO | Found 808 MIDs
|
|
2026-06-15 13:46:45 | INFO | ================================================================================
|
|
2026-06-15 13:46:45 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:46:45 | INFO | Fetching data for 808 MIDs
|
|
2026-06-15 13:46:45 | INFO | Fetched 3,565 rows from SQL Server
|
|
2026-06-15 13:46:45 | INFO | Fetched 3565 rows
|
|
2026-06-15 13:46:46 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 13:46:46 | INFO | SOS_OneApp: inserted 3,565 rows into ClickHouse
|
|
2026-06-15 13:46:46 | INFO | SOS_OneApp loaded successfully (3565 rows)
|
|
2026-06-15 13:46:46 | INFO | ================================================================================
|
|
2026-06-15 13:46:46 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:46:46 | INFO | Fetching OQaD data for run_date=2026-06-03
|
|
2026-06-15 13:46:48 | INFO | Fetched 455 rows
|
|
2026-06-15 13:46:48 | INFO | Fetched 455 rows
|
|
2026-06-15 13:46:48 | INFO | Deleting from OQaD
|
|
2026-06-15 13:46:48 | INFO | OQaD: inserted 455 rows into ClickHouse
|
|
2026-06-15 13:46:48 | INFO | OQaD loaded successfully (455 rows)
|
|
2026-06-15 13:46:48 | INFO | ================================================================================
|
|
2026-06-15 13:46:48 | INFO | Processing Table: Survey
|
|
2026-06-15 13:46:48 | INFO | Fetching Survey data for 808 MIDs
|
|
2026-06-15 13:46:48 | INFO | Fetched 139 Survey rows
|
|
2026-06-15 13:46:48 | INFO | Fetched 139 rows
|
|
2026-06-15 13:46:49 | INFO | Deleting from Survey
|
|
2026-06-15 13:46:49 | INFO | Survey: inserted 139 rows into ClickHouse
|
|
2026-06-15 13:46:49 | INFO | Survey loaded successfully (139 rows)
|
|
2026-06-15 13:46:49 | INFO | ================================================================================
|
|
2026-06-15 13:46:49 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:46:49 | INFO | Fetching Additional Visibility data for 808 MIDs
|
|
2026-06-15 13:46:50 | INFO | Fetched 1,884 Additional Visibility rows
|
|
2026-06-15 13:46:50 | INFO | Fetched 1884 rows
|
|
2026-06-15 13:46:50 | INFO | Deleting from additional_visibility
|
|
2026-06-15 13:46:50 | INFO | additional_visibility: inserted 1,884 rows into ClickHouse
|
|
2026-06-15 13:46:50 | INFO | additional_visibility loaded successfully (1884 rows)
|
|
2026-06-15 13:46:50 | INFO | ================================================================================
|
|
2026-06-15 13:46:50 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:46:50 | INFO | Fetching coverage data for 808 MIDs
|
|
2026-06-15 13:46:50 | INFO | Fetched 751 rows from SQL Server
|
|
2026-06-15 13:46:50 | INFO | Fetched 751 rows
|
|
2026-06-15 13:46:50 | INFO | Deleting from Coverage
|
|
2026-06-15 13:46:50 | INFO | Coverage: inserted 751 rows into ClickHouse
|
|
2026-06-15 13:46:50 | INFO | Coverage loaded successfully (751 rows)
|
|
2026-06-15 13:46:50 | INFO | ================================================================================
|
|
2026-06-15 13:46:50 | INFO | Processing Table: Login
|
|
2026-06-15 13:46:50 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:46:51 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:46:51 | INFO | Fetched 454 rows
|
|
2026-06-15 13:46:51 | INFO | Deleting from Login
|
|
2026-06-15 13:46:51 | INFO | No delete logic required for Login
|
|
2026-06-15 13:46:51 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 13:46:51 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 13:46:51 | INFO | ================================================================================
|
|
2026-06-15 13:46:51 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 13:46:51 | INFO | Fetching Stock Details data for 808 MIDs
|
|
2026-06-15 13:46:53 | INFO | Fetched 40,840 Stock Details rows
|
|
2026-06-15 13:46:53 | INFO | Fetched 40840 rows
|
|
2026-06-15 13:46:53 | INFO | Deleting from Stock_Details
|
|
2026-06-15 13:46:54 | INFO | Stock_Details: inserted 40,840 rows into ClickHouse
|
|
2026-06-15 13:46:54 | INFO | Stock_Details loaded successfully (40840 rows)
|
|
2026-06-15 13:46:54 | INFO | ================================================================================
|
|
2026-06-15 13:46:54 | INFO | Processing Table: Attendance
|
|
2026-06-15 13:46:54 | INFO | Fetching Attendance data from 2026-05-19 to 2026-06-03
|
|
2026-06-15 13:46:54 | INFO | Fetched 12,432 attendance rows for 579 employees
|
|
2026-06-15 13:46:54 | INFO | Fetched 12432 rows
|
|
2026-06-15 13:46:55 | INFO | Truncated table Attendance
|
|
2026-06-15 13:46:55 | INFO | Attendance: inserted 12,432 rows into ClickHouse
|
|
2026-06-15 13:46:55 | INFO | Attendance loaded successfully (12432 rows)
|
|
2026-06-15 13:46:55 | INFO | ================================================================================
|
|
2026-06-15 13:46:55 | INFO | Processing Table: Store_Master
|
|
2026-06-15 13:46:55 | INFO | Fetching Store Master data
|
|
2026-06-15 13:46:56 | INFO | Fetched 5,984 stores
|
|
2026-06-15 13:46:56 | INFO | Fetched 5984 rows
|
|
2026-06-15 13:46:56 | INFO | Truncated table Store_Master
|
|
2026-06-15 13:46:56 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 13:46:56 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 13:46:56 | INFO | ================================================================================
|
|
2026-06-15 13:46:56 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 13:46:56 | INFO | Fetching SKU Master data
|
|
2026-06-15 13:46:57 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 13:46:57 | INFO | Fetched 160 rows
|
|
2026-06-15 13:46:57 | INFO | Truncated table SKU_Master
|
|
2026-06-15 13:46:57 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 13:46:57 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 13:46:57 | INFO | ================================================================================
|
|
2026-06-15 13:46:57 | INFO | Processing Table: display_master
|
|
2026-06-15 13:46:57 | INFO | Fetching Display Master data
|
|
2026-06-15 13:46:57 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 13:46:57 | INFO | Fetched 135 rows
|
|
2026-06-15 13:46:57 | INFO | Truncated table display_master
|
|
2026-06-15 13:46:57 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 13:46:57 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 13:46:57 | INFO | ================================================================================
|
|
2026-06-15 13:46:57 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 13:46:57 | INFO | Fetching Employee Master data
|
|
2026-06-15 13:46:58 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 13:46:58 | INFO | Fetched 2273 rows
|
|
2026-06-15 13:46:58 | INFO | Truncated table Employee_Master
|
|
2026-06-15 13:46:58 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 13:46:58 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 13:46:58 | INFO | ================================================================================
|
|
2026-06-15 13:46:58 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 13:46:58 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 13:46:58 | INFO | Fetched 21,142 Journey Plan records
|
|
2026-06-15 13:46:58 | INFO | Fetched 21142 rows
|
|
2026-06-15 13:46:58 | INFO | Deleting from Journey_Plan
|
|
2026-06-15 13:46:58 | INFO | Journey_Plan: inserted 21,142 rows into ClickHouse
|
|
2026-06-15 13:46:58 | INFO | Journey_Plan loaded successfully (21142 rows)
|
|
2026-06-15 13:46:58 | INFO | ================================================================================
|
|
2026-06-15 13:46:58 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 13:46:58 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 13:46:58 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 13:46:59 | INFO | Fetched 29 rows
|
|
2026-06-15 13:46:59 | INFO | Truncated table coverage_remarks
|
|
2026-06-15 13:46:59 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 13:46:59 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 13:46:59 | INFO | ================================================================================
|
|
2026-06-15 13:46:59 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 13:46:59 | INFO | Fetching Mapping Store Visibility for 2026-06-03
|
|
2026-06-15 13:46:59 | INFO | Fetched 0 Mapping Store Visibility records
|
|
2026-06-15 13:46:59 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-15 13:46:59 | INFO | ================================================================================
|
|
2026-06-15 13:46:59 | INFO | Processing Table: Master_VisibilityReason
|
|
2026-06-15 13:46:59 | INFO | Fetching Master Visibility Reason data
|
|
2026-06-15 13:46:59 | INFO | Fetched 17 Master Visibility Reason records
|
|
2026-06-15 13:46:59 | INFO | Fetched 17 rows
|
|
2026-06-15 13:46:59 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-15 13:46:59 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-15 13:46:59 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-15 13:46:59 | INFO | ================================================================================
|
|
2026-06-15 13:46:59 | INFO | Processing Table: Master_VisibilityDefinition
|
|
2026-06-15 13:46:59 | INFO | Fetching Master Visibility Definition data
|
|
2026-06-15 13:46:59 | INFO | Fetched 861 Master Visibility Definition records
|
|
2026-06-15 13:46:59 | INFO | Fetched 861 rows
|
|
2026-06-15 13:46:59 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-15 13:47:00 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-15 13:47:00 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-15 13:47:00 | INFO | ================================================================================
|
|
2026-06-15 13:47:00 | INFO | Processing Table: Web_Logins
|
|
2026-06-15 13:47:00 | INFO | Fetching Web Login data for 2026-06-03
|
|
2026-06-15 13:47:00 | INFO | Fetched 200 Web Login records
|
|
2026-06-15 13:47:00 | INFO | Fetched 200 rows
|
|
2026-06-15 13:47:00 | INFO | Deleting from Web_Logins
|
|
2026-06-15 13:47:00 | INFO | Web_Logins: inserted 200 rows into ClickHouse
|
|
2026-06-15 13:47:00 | INFO | Web_Logins loaded successfully (200 rows)
|
|
2026-06-15 13:47:00 | INFO | ================================================================================
|
|
2026-06-15 13:47:00 | INFO | Processing Table: Promotion
|
|
2026-06-15 13:47:00 | INFO | Fetching Promotion data for 808 MIDs
|
|
2026-06-15 13:47:02 | INFO | Fetched 8,162 Promotion records
|
|
2026-06-15 13:47:02 | INFO | Fetched 8162 rows
|
|
2026-06-15 13:47:02 | INFO | Deleting from Promotion
|
|
2026-06-15 13:47:02 | INFO | Promotion: inserted 8,162 rows into ClickHouse
|
|
2026-06-15 13:47:02 | INFO | Promotion loaded successfully (8162 rows)
|
|
2026-06-15 13:47:02 | INFO | ================================================================================
|
|
2026-06-15 13:47:02 | INFO | Processing Table: PaidVisibility
|
|
2026-06-15 13:47:02 | INFO | Fetching Paid Visibility data for 808 MIDs
|
|
2026-06-15 13:47:03 | INFO | Fetched 1,064 Paid Visibility records
|
|
2026-06-15 13:47:03 | INFO | Fetched 1064 rows
|
|
2026-06-15 13:47:03 | INFO | Deleting from PaidVisibility
|
|
2026-06-15 13:47:03 | INFO | PaidVisibility: inserted 1,064 rows into ClickHouse
|
|
2026-06-15 13:47:03 | INFO | PaidVisibility loaded successfully (1064 rows)
|
|
2026-06-15 13:47:03 | INFO | ================================================================================
|
|
2026-06-15 13:47:03 | INFO | Processing Table: Master_Salesterritorylayer
|
|
2026-06-15 13:47:03 | INFO | Fetching Master Sales Territory Layer data
|
|
2026-06-15 13:47:03 | INFO | Fetched 33 Master Sales Territory Layer records
|
|
2026-06-15 13:47:03 | INFO | Fetched 33 rows
|
|
2026-06-15 13:47:03 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-15 13:47:03 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-15 13:47:03 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-15 13:47:03 | INFO | ================================================================================
|
|
2026-06-15 13:47:03 | INFO | Pipeline Completed Successfully
|
|
2026-06-15 13:47:03 | INFO | ================================================================================
|
|
2026-06-15 13:50:19 | INFO | ================================================================================
|
|
2026-06-15 13:50:19 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:50:19 | INFO | Pipeline Run Date: 2026-06-03
|
|
2026-06-15 13:50:19 | INFO | Connecting to databases...
|
|
2026-06-15 13:50:20 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001C6ED80B8B0>
|
|
2026-06-15 13:50:21 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000001C6EEAA91D0>
|
|
2026-06-15 13:50:21 | INFO | Database connections established
|
|
2026-06-15 13:50:21 | INFO | Collecting MIDs for: 2026-06-03
|
|
2026-06-15 13:50:21 | INFO | Found 808 MIDs
|
|
2026-06-15 13:50:22 | INFO | ================================================================================
|
|
2026-06-15 13:50:22 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:50:22 | INFO | Fetching data for 808 MIDs
|
|
2026-06-15 13:50:22 | INFO | Fetched 3,565 rows from SQL Server
|
|
2026-06-15 13:50:22 | INFO | Fetched 3565 rows
|
|
2026-06-15 13:50:22 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 13:50:23 | INFO | SOS_OneApp: inserted 3,565 rows into ClickHouse
|
|
2026-06-15 13:50:23 | INFO | SOS_OneApp loaded successfully (3565 rows)
|
|
2026-06-15 13:50:23 | INFO | ================================================================================
|
|
2026-06-15 13:50:23 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:50:23 | INFO | Fetching OQaD data for run_date=2026-06-03
|
|
2026-06-15 13:50:27 | INFO | Fetched 455 rows
|
|
2026-06-15 13:50:27 | INFO | Fetched 455 rows
|
|
2026-06-15 13:50:27 | INFO | Deleting from OQaD
|
|
2026-06-15 13:50:27 | INFO | OQaD: inserted 455 rows into ClickHouse
|
|
2026-06-15 13:50:27 | INFO | OQaD loaded successfully (455 rows)
|
|
2026-06-15 13:50:27 | INFO | ================================================================================
|
|
2026-06-15 13:50:27 | INFO | Processing Table: Survey
|
|
2026-06-15 13:50:27 | INFO | Fetching Survey data for 808 MIDs
|
|
2026-06-15 13:50:27 | INFO | Fetched 139 Survey rows
|
|
2026-06-15 13:50:27 | INFO | Fetched 139 rows
|
|
2026-06-15 13:50:27 | INFO | Deleting from Survey
|
|
2026-06-15 13:50:27 | INFO | Survey: inserted 139 rows into ClickHouse
|
|
2026-06-15 13:50:27 | INFO | Survey loaded successfully (139 rows)
|
|
2026-06-15 13:50:27 | INFO | ================================================================================
|
|
2026-06-15 13:50:27 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:50:27 | INFO | Fetching Additional Visibility data for 808 MIDs
|
|
2026-06-15 13:50:28 | INFO | Fetched 1,884 Additional Visibility rows
|
|
2026-06-15 13:50:28 | INFO | Fetched 1884 rows
|
|
2026-06-15 13:50:28 | INFO | Deleting from additional_visibility
|
|
2026-06-15 13:50:28 | INFO | additional_visibility: inserted 1,884 rows into ClickHouse
|
|
2026-06-15 13:50:28 | INFO | additional_visibility loaded successfully (1884 rows)
|
|
2026-06-15 13:50:28 | INFO | ================================================================================
|
|
2026-06-15 13:50:28 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:50:28 | INFO | Fetching coverage data for 808 MIDs
|
|
2026-06-15 13:50:28 | INFO | Fetched 751 rows from SQL Server
|
|
2026-06-15 13:50:28 | INFO | Fetched 751 rows
|
|
2026-06-15 13:50:28 | INFO | Deleting from Coverage
|
|
2026-06-15 13:50:29 | INFO | Coverage: inserted 751 rows into ClickHouse
|
|
2026-06-15 13:50:29 | INFO | Coverage loaded successfully (751 rows)
|
|
2026-06-15 13:50:29 | INFO | ================================================================================
|
|
2026-06-15 13:50:29 | INFO | Processing Table: Login
|
|
2026-06-15 13:50:29 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:50:29 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:50:29 | INFO | Fetched 454 rows
|
|
2026-06-15 13:50:29 | INFO | Deleting from Login
|
|
2026-06-15 13:50:29 | INFO | No delete logic required for Login
|
|
2026-06-15 13:50:29 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 13:50:29 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 13:50:29 | INFO | ================================================================================
|
|
2026-06-15 13:50:29 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 13:50:29 | INFO | Fetching Stock Details data for 808 MIDs
|
|
2026-06-15 13:50:33 | INFO | Fetched 40,840 Stock Details rows
|
|
2026-06-15 13:50:33 | INFO | Fetched 40840 rows
|
|
2026-06-15 13:50:33 | INFO | Deleting from Stock_Details
|
|
2026-06-15 13:50:34 | INFO | Stock_Details: inserted 40,840 rows into ClickHouse
|
|
2026-06-15 13:50:34 | INFO | Stock_Details loaded successfully (40840 rows)
|
|
2026-06-15 13:50:34 | INFO | ================================================================================
|
|
2026-06-15 13:50:34 | INFO | Processing Table: Attendance
|
|
2026-06-15 13:50:34 | INFO | Fetching Attendance data from 2026-05-19 to 2026-06-03
|
|
2026-06-15 13:50:35 | INFO | Fetched 12,432 attendance rows for 579 employees
|
|
2026-06-15 13:50:35 | INFO | Fetched 12432 rows
|
|
2026-06-15 13:50:35 | INFO | Truncated table Attendance
|
|
2026-06-15 13:50:35 | INFO | Attendance: inserted 12,432 rows into ClickHouse
|
|
2026-06-15 13:50:35 | INFO | Attendance loaded successfully (12432 rows)
|
|
2026-06-15 13:50:35 | INFO | ================================================================================
|
|
2026-06-15 13:50:35 | INFO | Processing Table: Store_Master
|
|
2026-06-15 13:50:35 | INFO | Fetching Store Master data
|
|
2026-06-15 13:50:37 | INFO | Fetched 5,984 stores
|
|
2026-06-15 13:50:37 | INFO | Fetched 5984 rows
|
|
2026-06-15 13:50:37 | INFO | Truncated table Store_Master
|
|
2026-06-15 13:50:38 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 13:50:38 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 13:50:38 | INFO | ================================================================================
|
|
2026-06-15 13:50:38 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 13:50:38 | INFO | Fetching SKU Master data
|
|
2026-06-15 13:50:38 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 13:50:38 | INFO | Fetched 160 rows
|
|
2026-06-15 13:50:38 | INFO | Truncated table SKU_Master
|
|
2026-06-15 13:50:38 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 13:50:38 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 13:50:38 | INFO | ================================================================================
|
|
2026-06-15 13:50:38 | INFO | Processing Table: display_master
|
|
2026-06-15 13:50:38 | INFO | Fetching Display Master data
|
|
2026-06-15 13:50:38 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 13:50:38 | INFO | Fetched 135 rows
|
|
2026-06-15 13:50:38 | INFO | Truncated table display_master
|
|
2026-06-15 13:50:39 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 13:50:39 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 13:50:39 | INFO | ================================================================================
|
|
2026-06-15 13:50:39 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 13:50:39 | INFO | Fetching Employee Master data
|
|
2026-06-15 13:50:39 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 13:50:39 | INFO | Fetched 2273 rows
|
|
2026-06-15 13:50:39 | INFO | Truncated table Employee_Master
|
|
2026-06-15 13:50:39 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 13:50:39 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 13:50:39 | INFO | ================================================================================
|
|
2026-06-15 13:50:39 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 13:50:39 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 13:50:40 | INFO | Fetched 21,142 Journey Plan records
|
|
2026-06-15 13:50:40 | INFO | Fetched 21142 rows
|
|
2026-06-15 13:50:40 | INFO | Deleting from Journey_Plan
|
|
2026-06-15 13:50:40 | INFO | Journey_Plan: inserted 21,142 rows into ClickHouse
|
|
2026-06-15 13:50:40 | INFO | Journey_Plan loaded successfully (21142 rows)
|
|
2026-06-15 13:50:40 | INFO | ================================================================================
|
|
2026-06-15 13:50:40 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 13:50:40 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 13:50:40 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 13:50:40 | INFO | Fetched 29 rows
|
|
2026-06-15 13:50:40 | INFO | Truncated table coverage_remarks
|
|
2026-06-15 13:50:40 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 13:50:40 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 13:50:40 | INFO | ================================================================================
|
|
2026-06-15 13:50:40 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 13:50:40 | ERROR | Failed processing table mapping_storevisibility
|
|
Traceback (most recent call last):
|
|
File "D:\data_move\main.py", line 177, in main
|
|
df = get_dataframe(
|
|
fn_name=fn_name,
|
|
...<3 lines>...
|
|
run_date=run_date,
|
|
)
|
|
File "D:\data_move\main.py", line 83, in get_dataframe
|
|
return fn(sql_engine)
|
|
TypeError: fetch_mapping_storevisibility() missing 1 required positional argument: 'run_date'
|
|
2026-06-15 13:51:03 | INFO | ================================================================================
|
|
2026-06-15 13:51:03 | INFO | Hello from data-move Python data pipeline!
|
|
2026-06-15 13:51:03 | INFO | Pipeline Run Date: 2026-06-03
|
|
2026-06-15 13:51:03 | INFO | Connecting to databases...
|
|
2026-06-15 13:51:03 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002632A0BB8B0>
|
|
2026-06-15 13:51:04 | INFO | <sqlalchemy.engine.cursor.CursorResult object at 0x000002632B2991D0>
|
|
2026-06-15 13:51:05 | INFO | Database connections established
|
|
2026-06-15 13:51:05 | INFO | Collecting MIDs for: 2026-06-03
|
|
2026-06-15 13:51:05 | INFO | Found 808 MIDs
|
|
2026-06-15 13:51:05 | INFO | ================================================================================
|
|
2026-06-15 13:51:05 | INFO | Processing Table: SOS_OneApp
|
|
2026-06-15 13:51:05 | INFO | Fetching data for 808 MIDs
|
|
2026-06-15 13:51:06 | INFO | Fetched 3,565 rows from SQL Server
|
|
2026-06-15 13:51:06 | INFO | Fetched 3565 rows
|
|
2026-06-15 13:51:06 | INFO | Deleting from SOS_OneApp
|
|
2026-06-15 13:51:06 | INFO | SOS_OneApp: inserted 3,565 rows into ClickHouse
|
|
2026-06-15 13:51:06 | INFO | SOS_OneApp loaded successfully (3565 rows)
|
|
2026-06-15 13:51:06 | INFO | ================================================================================
|
|
2026-06-15 13:51:06 | INFO | Processing Table: OQaD
|
|
2026-06-15 13:51:06 | INFO | Fetching OQaD data for run_date=2026-06-03
|
|
2026-06-15 13:51:09 | INFO | Fetched 455 rows
|
|
2026-06-15 13:51:09 | INFO | Fetched 455 rows
|
|
2026-06-15 13:51:09 | INFO | Deleting from OQaD
|
|
2026-06-15 13:51:09 | INFO | OQaD: inserted 455 rows into ClickHouse
|
|
2026-06-15 13:51:09 | INFO | OQaD loaded successfully (455 rows)
|
|
2026-06-15 13:51:09 | INFO | ================================================================================
|
|
2026-06-15 13:51:09 | INFO | Processing Table: Survey
|
|
2026-06-15 13:51:09 | INFO | Fetching Survey data for 808 MIDs
|
|
2026-06-15 13:51:09 | INFO | Fetched 139 Survey rows
|
|
2026-06-15 13:51:09 | INFO | Fetched 139 rows
|
|
2026-06-15 13:51:09 | INFO | Deleting from Survey
|
|
2026-06-15 13:51:09 | INFO | Survey: inserted 139 rows into ClickHouse
|
|
2026-06-15 13:51:09 | INFO | Survey loaded successfully (139 rows)
|
|
2026-06-15 13:51:09 | INFO | ================================================================================
|
|
2026-06-15 13:51:09 | INFO | Processing Table: additional_visibility
|
|
2026-06-15 13:51:09 | INFO | Fetching Additional Visibility data for 808 MIDs
|
|
2026-06-15 13:51:10 | INFO | Fetched 1,884 Additional Visibility rows
|
|
2026-06-15 13:51:10 | INFO | Fetched 1884 rows
|
|
2026-06-15 13:51:10 | INFO | Deleting from additional_visibility
|
|
2026-06-15 13:51:10 | INFO | additional_visibility: inserted 1,884 rows into ClickHouse
|
|
2026-06-15 13:51:10 | INFO | additional_visibility loaded successfully (1884 rows)
|
|
2026-06-15 13:51:10 | INFO | ================================================================================
|
|
2026-06-15 13:51:10 | INFO | Processing Table: Coverage
|
|
2026-06-15 13:51:10 | INFO | Fetching coverage data for 808 MIDs
|
|
2026-06-15 13:51:10 | INFO | Fetched 751 rows from SQL Server
|
|
2026-06-15 13:51:10 | INFO | Fetched 751 rows
|
|
2026-06-15 13:51:10 | INFO | Deleting from Coverage
|
|
2026-06-15 13:51:11 | INFO | Coverage: inserted 751 rows into ClickHouse
|
|
2026-06-15 13:51:11 | INFO | Coverage loaded successfully (751 rows)
|
|
2026-06-15 13:51:11 | INFO | ================================================================================
|
|
2026-06-15 13:51:11 | INFO | Processing Table: Login
|
|
2026-06-15 13:51:11 | INFO | Fetching Login data for yesterday
|
|
2026-06-15 13:51:11 | INFO | Fetched 454 Login rows
|
|
2026-06-15 13:51:11 | INFO | Fetched 454 rows
|
|
2026-06-15 13:51:11 | INFO | Deleting from Login
|
|
2026-06-15 13:51:11 | INFO | No delete logic required for Login
|
|
2026-06-15 13:51:11 | INFO | Login: inserted 454 rows into ClickHouse
|
|
2026-06-15 13:51:11 | INFO | Login loaded successfully (454 rows)
|
|
2026-06-15 13:51:11 | INFO | ================================================================================
|
|
2026-06-15 13:51:11 | INFO | Processing Table: Stock_Details
|
|
2026-06-15 13:51:11 | INFO | Fetching Stock Details data for 808 MIDs
|
|
2026-06-15 13:51:15 | INFO | Fetched 40,840 Stock Details rows
|
|
2026-06-15 13:51:15 | INFO | Fetched 40840 rows
|
|
2026-06-15 13:51:15 | INFO | Deleting from Stock_Details
|
|
2026-06-15 13:51:15 | INFO | Stock_Details: inserted 40,840 rows into ClickHouse
|
|
2026-06-15 13:51:15 | INFO | Stock_Details loaded successfully (40840 rows)
|
|
2026-06-15 13:51:15 | INFO | ================================================================================
|
|
2026-06-15 13:51:15 | INFO | Processing Table: Attendance
|
|
2026-06-15 13:51:15 | INFO | Fetching Attendance data from 2026-05-19 to 2026-06-03
|
|
2026-06-15 13:51:16 | INFO | Fetched 12,432 attendance rows for 579 employees
|
|
2026-06-15 13:51:16 | INFO | Fetched 12432 rows
|
|
2026-06-15 13:51:17 | INFO | Truncated table Attendance
|
|
2026-06-15 13:51:17 | INFO | Attendance: inserted 12,432 rows into ClickHouse
|
|
2026-06-15 13:51:17 | INFO | Attendance loaded successfully (12432 rows)
|
|
2026-06-15 13:51:17 | INFO | ================================================================================
|
|
2026-06-15 13:51:17 | INFO | Processing Table: Store_Master
|
|
2026-06-15 13:51:17 | INFO | Fetching Store Master data
|
|
2026-06-15 13:51:20 | INFO | Fetched 5,984 stores
|
|
2026-06-15 13:51:20 | INFO | Fetched 5984 rows
|
|
2026-06-15 13:51:20 | INFO | Truncated table Store_Master
|
|
2026-06-15 13:51:20 | INFO | Store_Master: inserted 5,984 rows into ClickHouse
|
|
2026-06-15 13:51:20 | INFO | Store_Master loaded successfully (5984 rows)
|
|
2026-06-15 13:51:20 | INFO | ================================================================================
|
|
2026-06-15 13:51:20 | INFO | Processing Table: SKU_Master
|
|
2026-06-15 13:51:20 | INFO | Fetching SKU Master data
|
|
2026-06-15 13:51:20 | INFO | Fetched 160 SKU Master rows
|
|
2026-06-15 13:51:20 | INFO | Fetched 160 rows
|
|
2026-06-15 13:51:21 | INFO | Truncated table SKU_Master
|
|
2026-06-15 13:51:21 | INFO | SKU_Master: inserted 160 rows into ClickHouse
|
|
2026-06-15 13:51:21 | INFO | SKU_Master loaded successfully (160 rows)
|
|
2026-06-15 13:51:21 | INFO | ================================================================================
|
|
2026-06-15 13:51:21 | INFO | Processing Table: display_master
|
|
2026-06-15 13:51:21 | INFO | Fetching Display Master data
|
|
2026-06-15 13:51:21 | INFO | Fetched 135 Display Master records
|
|
2026-06-15 13:51:21 | INFO | Fetched 135 rows
|
|
2026-06-15 13:51:21 | INFO | Truncated table display_master
|
|
2026-06-15 13:51:21 | INFO | display_master: inserted 135 rows into ClickHouse
|
|
2026-06-15 13:51:21 | INFO | display_master loaded successfully (135 rows)
|
|
2026-06-15 13:51:21 | INFO | ================================================================================
|
|
2026-06-15 13:51:21 | INFO | Processing Table: Employee_Master
|
|
2026-06-15 13:51:21 | INFO | Fetching Employee Master data
|
|
2026-06-15 13:51:21 | INFO | Fetched 2,273 Employee Master records
|
|
2026-06-15 13:51:21 | INFO | Fetched 2273 rows
|
|
2026-06-15 13:51:22 | INFO | Truncated table Employee_Master
|
|
2026-06-15 13:51:22 | INFO | Employee_Master: inserted 2,273 rows into ClickHouse
|
|
2026-06-15 13:51:22 | INFO | Employee_Master loaded successfully (2273 rows)
|
|
2026-06-15 13:51:22 | INFO | ================================================================================
|
|
2026-06-15 13:51:22 | INFO | Processing Table: Journey_Plan
|
|
2026-06-15 13:51:22 | INFO | Fetching Journey Plan for 2026-06
|
|
2026-06-15 13:51:22 | INFO | Fetched 21,142 Journey Plan records
|
|
2026-06-15 13:51:22 | INFO | Fetched 21142 rows
|
|
2026-06-15 13:51:22 | INFO | Deleting from Journey_Plan
|
|
2026-06-15 13:51:22 | INFO | Journey_Plan: inserted 21,142 rows into ClickHouse
|
|
2026-06-15 13:51:22 | INFO | Journey_Plan loaded successfully (21142 rows)
|
|
2026-06-15 13:51:22 | INFO | ================================================================================
|
|
2026-06-15 13:51:22 | INFO | Processing Table: coverage_remarks
|
|
2026-06-15 13:51:22 | INFO | Fetching Coverage Remarks
|
|
2026-06-15 13:51:22 | INFO | Fetched 29 Coverage Remark records
|
|
2026-06-15 13:51:22 | INFO | Fetched 29 rows
|
|
2026-06-15 13:51:23 | INFO | Truncated table coverage_remarks
|
|
2026-06-15 13:51:23 | INFO | coverage_remarks: inserted 29 rows into ClickHouse
|
|
2026-06-15 13:51:23 | INFO | coverage_remarks loaded successfully (29 rows)
|
|
2026-06-15 13:51:23 | INFO | ================================================================================
|
|
2026-06-15 13:51:23 | INFO | Processing Table: mapping_storevisibility
|
|
2026-06-15 13:51:23 | INFO | Fetching Mapping Store Visibility for 2026-06-03
|
|
2026-06-15 13:51:23 | INFO | Fetched 0 Mapping Store Visibility records
|
|
2026-06-15 13:51:23 | WARNING | mapping_storevisibility returned no rows
|
|
2026-06-15 13:51:23 | INFO | ================================================================================
|
|
2026-06-15 13:51:23 | INFO | Processing Table: Master_VisibilityReason
|
|
2026-06-15 13:51:23 | INFO | Fetching Master Visibility Reason data
|
|
2026-06-15 13:51:23 | INFO | Fetched 17 Master Visibility Reason records
|
|
2026-06-15 13:51:23 | INFO | Fetched 17 rows
|
|
2026-06-15 13:51:23 | INFO | Truncated table Master_VisibilityReason
|
|
2026-06-15 13:51:23 | INFO | Master_VisibilityReason: inserted 17 rows into ClickHouse
|
|
2026-06-15 13:51:23 | INFO | Master_VisibilityReason loaded successfully (17 rows)
|
|
2026-06-15 13:51:23 | INFO | ================================================================================
|
|
2026-06-15 13:51:23 | INFO | Processing Table: Master_VisibilityDefinition
|
|
2026-06-15 13:51:23 | INFO | Fetching Master Visibility Definition data
|
|
2026-06-15 13:51:23 | INFO | Fetched 861 Master Visibility Definition records
|
|
2026-06-15 13:51:23 | INFO | Fetched 861 rows
|
|
2026-06-15 13:51:23 | INFO | Truncated table Master_VisibilityDefinition
|
|
2026-06-15 13:51:24 | INFO | Master_VisibilityDefinition: inserted 861 rows into ClickHouse
|
|
2026-06-15 13:51:24 | INFO | Master_VisibilityDefinition loaded successfully (861 rows)
|
|
2026-06-15 13:51:24 | INFO | ================================================================================
|
|
2026-06-15 13:51:24 | INFO | Processing Table: Web_Logins
|
|
2026-06-15 13:51:24 | INFO | Fetching Web Login data for 2026-06-03
|
|
2026-06-15 13:51:24 | INFO | Fetched 200 Web Login records
|
|
2026-06-15 13:51:24 | INFO | Fetched 200 rows
|
|
2026-06-15 13:51:24 | INFO | Deleting from Web_Logins
|
|
2026-06-15 13:51:24 | INFO | Web_Logins: inserted 200 rows into ClickHouse
|
|
2026-06-15 13:51:24 | INFO | Web_Logins loaded successfully (200 rows)
|
|
2026-06-15 13:51:24 | INFO | ================================================================================
|
|
2026-06-15 13:51:24 | INFO | Processing Table: Promotion
|
|
2026-06-15 13:51:24 | INFO | Fetching Promotion data for 808 MIDs
|
|
2026-06-15 13:51:28 | INFO | Fetched 8,162 Promotion records
|
|
2026-06-15 13:51:28 | INFO | Fetched 8162 rows
|
|
2026-06-15 13:51:28 | INFO | Deleting from Promotion
|
|
2026-06-15 13:51:29 | INFO | Promotion: inserted 8,162 rows into ClickHouse
|
|
2026-06-15 13:51:29 | INFO | Promotion loaded successfully (8162 rows)
|
|
2026-06-15 13:51:29 | INFO | ================================================================================
|
|
2026-06-15 13:51:29 | INFO | Processing Table: PaidVisibility
|
|
2026-06-15 13:51:29 | INFO | Fetching Paid Visibility data for 808 MIDs
|
|
2026-06-15 13:51:30 | INFO | Fetched 1,064 Paid Visibility records
|
|
2026-06-15 13:51:30 | INFO | Fetched 1064 rows
|
|
2026-06-15 13:51:30 | INFO | Deleting from PaidVisibility
|
|
2026-06-15 13:51:30 | INFO | PaidVisibility: inserted 1,064 rows into ClickHouse
|
|
2026-06-15 13:51:30 | INFO | PaidVisibility loaded successfully (1064 rows)
|
|
2026-06-15 13:51:30 | INFO | ================================================================================
|
|
2026-06-15 13:51:30 | INFO | Processing Table: Master_Salesterritorylayer
|
|
2026-06-15 13:51:30 | INFO | Fetching Master Sales Territory Layer data
|
|
2026-06-15 13:51:30 | INFO | Fetched 33 Master Sales Territory Layer records
|
|
2026-06-15 13:51:30 | INFO | Fetched 33 rows
|
|
2026-06-15 13:51:30 | INFO | Truncated table Master_Salesterritorylayer
|
|
2026-06-15 13:51:30 | INFO | Master_Salesterritorylayer: inserted 33 rows into ClickHouse
|
|
2026-06-15 13:51:30 | INFO | Master_Salesterritorylayer loaded successfully (33 rows)
|
|
2026-06-15 13:51:30 | INFO | ================================================================================
|
|
2026-06-15 13:51:30 | INFO | Pipeline Completed Successfully
|
|
2026-06-15 13:51:30 | INFO | ================================================================================
|