=== Job Execution Log === Job: DataHub Daily Refresh Execution ID: 371 Start Time: 2026-03-02 12:30:00 Command: C:\PYTHON\DataHub\start_daily_refresh.bat Working Directory: C:\PYTHON\DataHub ================================================== ============================================================================ DataHub - Daily Incremental Refresh ============================================================================ Database temporaneo: C:\PYTHON\DataHub\db\temp.db Database produzione: \\Mediaset.it\share\Indirizzo_controllo_risorse\SOFTWARE\DataHub\DataHub.db ============================================================================ 2026-03-02 12:30:01 - data_ingestion_orchestrator - INFO - Avvio processo completo di DataIngestion orchestrato... 2026-03-02 12:30:01 - data_ingestion_orchestrator - INFO - --- Avvio importazione per: GEMMA --- 2026-03-02 12:30:01 - gemma_importer - INFO - Avvio importazione per 'stg_gemma' da \\mediaset.it\share\Indirizzo_controllo_risorse\SOFTWARE\SCHEDULATORE\INPUT\estr_gemma.csv 2026-03-02 12:30:01 - gemma_importer - INFO - Lettura file con: encoding='cp1252', delimitatore=';', skip righe=0 2026-03-02 12:30:03 - gemma_importer - INFO - File letto: 51,036 righe. 2026-03-02 12:30:03 - gemma_importer - INFO - Pulizia tabella di staging 'stg_gemma'... 2026-03-02 12:30:04 - gemma_importer - INFO - Pulizia completata. 2026-03-02 12:30:04 - gemma_importer - INFO - Inserimento di 51,036 righe in 'stg_gemma'... 2026-03-02 12:30:05 - gemma_importer - INFO - Inserimento dati in staging completato con successo. 2026-03-02 12:30:05 - data_ingestion_orchestrator - INFO - --- Fine importazione per: GEMMA --- 2026-03-02 12:30:05 - data_ingestion_orchestrator - INFO - Processo completo di DataIngestion terminato in 3.75 secondi. ============================================================ : DataHub - Gestione CoreDB = : Data: 02/03/2026 12:30:01 = ============================================================ IMPORTAZIONE DATI (Staging) ------------------------------\n Utilizzando percorso da server per gemma: \\mediaset.it\share\Indirizzo_controllo_risorse\SOFTWARE\SCHEDULATORE\INPUT\estr_gemma.csv \nAvvio processo di ingestione (Staging)... Processo di ingestione (Staging) terminato. \n============================== RISULTATI IMPORTAZIONE (Staging) ============================== \n\u2705 Gemma Staging: - Righe lette da CSV: 51,036 - Righe inserite staging: 51,036 \n------------------------------ Percorso database utilizzato: C:\PYTHON\DataHub\db\temp.db Controlla i file di log per maggiori dettagli. NOTA: I dati sono stati caricati nelle tabelle STAGING (stg_*). Eseguire il comando 'transform' per popolare le tabelle Core. ==============================\n 2026-03-02 12:30:06 - data_ingestion_orchestrator - INFO - Avvio processo completo di DataIngestion orchestrato... 2026-03-02 12:30:06 - data_ingestion_orchestrator - INFO - --- Avvio importazione per: OSSERVATORIO --- 2026-03-02 12:30:06 - osservatorio_importer - INFO - Avvio importazione per 'stg_osservatorio' da \\mediaset.it\share\Indirizzo_controllo_risorse\SOFTWARE\SCHEDULATORE\INPUT\osservatorio.txt 2026-03-02 12:30:06 - osservatorio_importer - INFO - Lettura file senza header con: encoding='cp1252', delimitatore=';' C:\PYTHON\DataHub\data_ingestion\_osservatorio_importer.py:32: ParserWarning: Skipping line 4118: expected 26 fields, saw 27 df = pd.read_csv( 2026-03-02 12:30:08 - osservatorio_importer - INFO - File letto: 35,203 righe. 2026-03-02 12:30:08 - osservatorio_importer - INFO - Pulizia tabella di staging 'stg_osservatorio'... 2026-03-02 12:30:09 - osservatorio_importer - INFO - Pulizia completata. 2026-03-02 12:30:09 - osservatorio_importer - INFO - Inserimento di 35,203 righe in 'stg_osservatorio'... 2026-03-02 12:30:09 - osservatorio_importer - INFO - Inserimento dati in staging completato con successo. 2026-03-02 12:30:09 - data_ingestion_orchestrator - INFO - --- Fine importazione per: OSSERVATORIO --- 2026-03-02 12:30:09 - data_ingestion_orchestrator - INFO - Processo completo di DataIngestion terminato in 3.17 secondi. ============================================================ : DataHub - Gestione CoreDB = : Data: 02/03/2026 12:30:06 = ============================================================ IMPORTAZIONE DATI (Staging) ------------------------------\n Utilizzando percorso da server per osservatorio: \\mediaset.it\share\Indirizzo_controllo_risorse\SOFTWARE\SCHEDULATORE\INPUT\osservatorio.txt \nAvvio processo di ingestione (Staging)... Processo di ingestione (Staging) terminato. \n============================== RISULTATI IMPORTAZIONE (Staging) ============================== \n\u2705 Osservatorio Staging: - Righe lette da CSV: 35,203 - Righe inserite staging: 35,203 \n------------------------------ Percorso database utilizzato: C:\PYTHON\DataHub\db\temp.db Controlla i file di log per maggiori dettagli. NOTA: I dati sono stati caricati nelle tabelle STAGING (stg_*). Eseguire il comando 'transform' per popolare le tabelle Core. ==============================\n 2026-03-02 12:30:10 - transform_process - INFO - Avvio processo di trasformazione da Staging a Core... 2026-03-02 12:30:10 - transform_process - INFO - Database target: C:\PYTHON\DataHub\db\temp.db 2026-03-02 12:30:10 - transform_process - INFO - --- Inizio Trasformazione Gemma --- 2026-03-02 12:30:10 - transform_process - INFO - Inizio trasformazione Gemma (stg_gemma -> gemma) 2026-03-02 12:30:10 - transform_process - INFO - Lettura dati da stg_gemma... 2026-03-02 12:30:11 - transform_process - INFO - Lette 51,036 righe da stg_gemma. 2026-03-02 12:30:11 - transform_process - INFO - Applicazione logica di trasformazione per i campi 'V_*'... 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 9296 record nella colonna 'V_C5' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 10736 record nella colonna 'V_I1' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 9538 record nella colonna 'V_R4' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 11860 record nella colonna 'V_LA5' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 12462 record nella colonna 'V_I2' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 11980 record nella colonna 'V_IRIS' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 12313 record nella colonna 'V_TOP' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 8870 record nella colonna 'V_FOC' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 12185 record nella colonna 'V_C20' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 12430 record nella colonna 'V_CI34' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 12483 record nella colonna 'V_C27' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 12369 record nella colonna 'V_CIN' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 11950 record nella colonna 'V_INF' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 12411 record nella colonna 'V_EMO' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 12419 record nella colonna 'V_ENE' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 12459 record nella colonna 'V_COM' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 12462 record nella colonna 'V_STO' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 12434 record nella colonna 'V_CRI' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Aggiornati 12463 record nella colonna 'V_ACT' con '!'. 2026-03-02 12:30:11 - transform_process - INFO - Logica di trasformazione completata. Totale campi aggiornati: 223,120. 2026-03-02 12:30:11 - transform_process - INFO - Pulizia e inserimento dati nella tabella 'gemma' Core... 2026-03-02 12:30:12 - transform_process - INFO - Calcolo hash MD5 per righe gemma... 2026-03-02 12:30:17 - transform_process - INFO - Hash MD5 calcolato per 51036 righe gemma. 2026-03-02 12:30:17 - transform_process - INFO - [Gemma] record pre-delete in 'gemma': 51009 2026-03-02 12:30:18 - transform_process - INFO - Tabella 'gemma' Core pulita. Inizio inserimento di 51,036 righe. 2026-03-02 12:30:19 - transform_process - INFO - Inserite 51,036 righe nella tabella 'gemma' Core. 2026-03-02 12:30:19 - transform_process - INFO - Trasformazione Gemma completata in 9.12 secondi. Status: SUCCESS 2026-03-02 12:30:19 - transform_process - INFO - --- Fine Trasformazione Gemma (Status: SUCCESS) --- 2026-03-02 12:30:19 - transform_process - INFO - Processo di trasformazione terminato (blocco finally). Durata: 9.17 secondi. ============================================================ : DataHub - Gestione CoreDB = : Data: 02/03/2026 12:30:10 = ============================================================ TRASFORMAZIONE DATI (Staging -> Core) ------------------------------\n Database target: C:\PYTHON\DataHub\db\temp.db \n============================== RISULTATI TRASFORMAZIONE (Staging -> Core) ============================== - Prodotti: \u2139\ufe0f SKIPPED - Saltata a causa di fallimento precedente. - Emesso: \u2139\ufe0f SKIPPED - Saltata a causa di fallimento precedente. - Diritti: \u2139\ufe0f SKIPPED - Saltata a causa di fallimento precedente. - Cast: \u2139\ufe0f SKIPPED - Saltata a causa di fallimento precedente. - Gemma: \u2705 SUCCESS - Trasformazione Gemma completata. - Boxoffice: \u2139\ufe0f SKIPPED - Dati Boxoffice gestiti nella fase Consolidate. - Veg: \u2139\ufe0f SKIPPED - Logica transform_veg sospesa/non applicabile qui. ------------------------------ \n\u2705 Trasformazione completata con successo per tutte le tabelle eseguite/applicabili. Tempo totale trasformazione: 9.17 secondi. ==============================\n ============================================================ : DataHub - Gestione CoreDB = : Data: 02/03/2026 12:30:20 = ============================================================ EXPORT DATABASE PRODUZIONE ------------------------------\n ====================================================================== EXPORT DATABASE PRODUZIONE (Build Local + Copy) ====================================================================== Inizio: 2026-03-02 12:30:20 Strategia: 1. Copia DB produzione dal server in locale (se esiste) 2. Export tabelle gestite su disco locale (veloce) 3. Ottimizzazione DB locale 4. Copia DB finale sul server (singola operazione) ====================================================================== Connessione database di lavoro: C:\PYTHON\DataHub\db\temp.db OK Connesso a database di lavoro ====================================================================== PREPARAZIONE DATABASE LOCALE ====================================================================== Rimozione DB locale temporaneo esistente... OK Rimosso: C:\PYTHON\DataHub\db\DataHub_production_temp.db Copia DB produzione dal server in locale... Da: \\Mediaset.it\share\Indirizzo_controllo_risorse\SOFTWARE\DataHub\DataHub.db (2847.64 MB) A: C:\PYTHON\DataHub\db\DataHub_production_temp.db OK Copiato (2847.64 MB) Questo preserva eventuali tabelle extra non gestite Connessione database locale: C:\PYTHON\DataHub\db\DataHub_production_temp.db OK Connesso con ottimizzazioni per disco locale ====================================================================== ====================================================================== Tabelle da esportare: 9 Tabelle da saltare (staging): 11 ====================================================================== Tabelle ESCLUSE (staging): - stg_box_dettaglio - stg_boxoffice - stg_cast - stg_diritti - stg_emesso - stg_gemma - stg_imdb - stg_osservatorio - stg_prodotti - stg_scelte_rete - stg_veg ====================================================================== Inizio export tabelle... ====================================================================== [1/9] Exporting boxoffice... 50000/207282 (24%)... 100000/207282 (48%)... 150000/207282 (72%)... 200000/207282 (96%)... 207282/207282 (100%)...OK 207282 record [2/9] Exporting cast... 50000/1758692 (3%)... 100000/1758692 (6%)... 150000/1758692 (9%)... 200000/1758692 (11%)... 250000/1758692 (14%)... 300000/1758692 (17%)... 350000/1758692 (20%)... 400000/1758692 (23%)... 450000/1758692 (26%)... 500000/1758692 (28%)... 550000/1758692 (31%)... 600000/1758692 (34%)... 650000/1758692 (37%)... 700000/1758692 (40%)... 750000/1758692 (43%)... 800000/1758692 (45%)... 850000/1758692 (48%)... 900000/1758692 (51%)... 950000/1758692 (54%)... 1000000/1758692 (57%)... 1050000/1758692 (60%)... 1100000/1758692 (63%)... 1150000/1758692 (65%)... 1200000/1758692 (68%)... 1250000/1758692 (71%)... 1300000/1758692 (74%)... 1350000/1758692 (77%)... 1400000/1758692 (80%)... 1450000/1758692 (82%)... 1500000/1758692 (85%)... 1550000/1758692 (88%)... 1600000/1758692 (91%)... 1650000/1758692 (94%)... 1700000/1758692 (97%)... 1750000/1758692 (100%)... 1758692/1758692 (100%)...OK 1758692 record [3/9] Exporting diritti... OK 50993 record [4/9] Exporting diritti_windows... OK 3391 record [5/9] Exporting emesso... 50000/8657442 (1%)... 100000/8657442 (1%)... 150000/8657442 (2%)... 200000/8657442 (2%)... 250000/8657442 (3%)... 300000/8657442 (3%)... 350000/8657442 (4%)... 400000/8657442 (5%)... 450000/8657442 (5%)... 500000/8657442 (6%)... 550000/8657442 (6%)... 600000/8657442 (7%)... 650000/8657442 (8%)... 700000/8657442 (8%)... 750000/8657442 (9%)... 800000/8657442 (9%)... 850000/8657442 (10%)... 900000/8657442 (10%)... 950000/8657442 (11%)... 1000000/8657442 (12%)... 1050000/8657442 (12%)... 1100000/8657442 (13%)... 1150000/8657442 (13%)... 1200000/8657442 (14%)... 1250000/8657442 (14%)... 1300000/8657442 (15%)... 1350000/8657442 (16%)... 1400000/8657442 (16%)... 1450000/8657442 (17%)... 1500000/8657442 (17%)... 1550000/8657442 (18%)... 1600000/8657442 (18%)... 1650000/8657442 (19%)... 1700000/8657442 (20%)... 1750000/8657442 (20%)... 1800000/8657442 (21%)... 1850000/8657442 (21%)... 1900000/8657442 (22%)... 1950000/8657442 (23%)... 2000000/8657442 (23%)... 2050000/8657442 (24%)... 2100000/8657442 (24%)... 2150000/8657442 (25%)... 2200000/8657442 (25%)... 2250000/8657442 (26%)... 2300000/8657442 (27%)... 2350000/8657442 (27%)... 2400000/8657442 (28%)... 2450000/8657442 (28%)... 2500000/8657442 (29%)... 2550000/8657442 (29%)... 2600000/8657442 (30%)... 2650000/8657442 (31%)... 2700000/8657442 (31%)... 2750000/8657442 (32%)... 2800000/8657442 (32%)... 2850000/8657442 (33%)... 2900000/8657442 (33%)... 2950000/8657442 (34%)... 3000000/8657442 (35%)... 3050000/8657442 (35%)... 3100000/8657442 (36%)... 3150000/8657442 (36%)... 3200000/8657442 (37%)... 3250000/8657442 (38%)... 3300000/8657442 (38%)... 3350000/8657442 (39%)... 3400000/8657442 (39%)... 3450000/8657442 (40%)... 3500000/8657442 (40%)... 3550000/8657442 (41%)... 3600000/8657442 (42%)... 3650000/8657442 (42%)... 3700000/8657442 (43%)... 3750000/8657442 (43%)... 3800000/8657442 (44%)... 3850000/8657442 (44%)... 3900000/8657442 (45%)... 3950000/8657442 (46%)... 4000000/8657442 (46%)... 4050000/8657442 (47%)... 4100000/8657442 (47%)... 4150000/8657442 (48%)... 4200000/8657442 (49%)... 4250000/8657442 (49%)... 4300000/8657442 (50%)... 4350000/8657442 (50%)... 4400000/8657442 (51%)... 4450000/8657442 (51%)... 4500000/8657442 (52%)... 4550000/8657442 (53%)... 4600000/8657442 (53%)... 4650000/8657442 (54%)... 4700000/8657442 (54%)... 4750000/8657442 (55%)... 4800000/8657442 (55%)... 4850000/8657442 (56%)... 4900000/8657442 (57%)... 4950000/8657442 (57%)... 5000000/8657442 (58%)... 5050000/8657442 (58%)... 5100000/8657442 (59%)... 5150000/8657442 (59%)... 5200000/8657442 (60%)... 5250000/8657442 (61%)... 5300000/8657442 (61%)... 5350000/8657442 (62%)... 5400000/8657442 (62%)... 5450000/8657442 (63%)... 5500000/8657442 (64%)... 5550000/8657442 (64%)... 5600000/8657442 (65%)... 5650000/8657442 (65%)... 5700000/8657442 (66%)... 5750000/8657442 (66%)... 5800000/8657442 (67%)... 5850000/8657442 (68%)... 5900000/8657442 (68%)... 5950000/8657442 (69%)... 6000000/8657442 (69%)... 6050000/8657442 (70%)... 6100000/8657442 (70%)... 6150000/8657442 (71%)... 6200000/8657442 (72%)... 6250000/8657442 (72%)... 6300000/8657442 (73%)... 6350000/8657442 (73%)... 6400000/8657442 (74%)... 6450000/8657442 (75%)... 6500000/8657442 (75%)... 6550000/8657442 (76%)... 6600000/8657442 (76%)... 6650000/8657442 (77%)... 6700000/8657442 (77%)... 6750000/8657442 (78%)... 6800000/8657442 (79%)... 6850000/8657442 (79%)... 6900000/8657442 (80%)... 6950000/8657442 (80%)... 7000000/8657442 (81%)... 7050000/8657442 (81%)... 7100000/8657442 (82%)... 7150000/8657442 (83%)... 7200000/8657442 (83%)... 7250000/8657442 (84%)... 7300000/8657442 (84%)... 7350000/8657442 (85%)... 7400000/8657442 (85%)... 7450000/8657442 (86%)... 7500000/8657442 (87%)... 7550000/8657442 (87%)... 7600000/8657442 (88%)... 7650000/8657442 (88%)... 7700000/8657442 (89%)... 7750000/8657442 (90%)... 7800000/8657442 (90%)... 7850000/8657442 (91%)... 7900000/8657442 (91%)... 7950000/8657442 (92%)... 8000000/8657442 (92%)... 8050000/8657442 (93%)... 8100000/8657442 (94%)... 8150000/8657442 (94%)... 8200000/8657442 (95%)... 8250000/8657442 (95%)... 8300000/8657442 (96%)... 8350000/8657442 (96%)... 8400000/8657442 (97%)... 8450000/8657442 (98%)... 8500000/8657442 (98%)... 8550000/8657442 (99%)... 8600000/8657442 (99%)... 8650000/8657442 (100%)... 8657442/8657442 (100%)...OK 8657442 record [6/9] Exporting gemma... OK 51036 record [7/9] Exporting prodotti... 50000/224486 (22%)... 100000/224486 (45%)... 150000/224486 (67%)... 200000/224486 (89%)... 224486/224486 (100%)...