1i$dZddlZddlZddlmZddlmZddlZejde ee j ddl m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZddlm Z ddl!m"Z"ej#e$Z%d$d efd Z&d Z'd Z(d e)de*fdZ+dZ,dZ-dZ.dZ/dZ0d%de*de*fdZ1e$dkrddl2Z2e2j3dZ4e45ddde45d dd!e46Z7e1e7j8e7j9"Z:ej;e:rdnd#dSdS)&z ImdbUpdate v2 - Pipeline DuckDB Scarica dataset IMDb, trasforma via DuckDB in-memory, produce imdb.parquet e db_split_IMDB.accdb. N)Path)datetime)BASE_DIRTEMP_DIR EMPTY_DIRACCESS_DB_TEMPLATEACCESS_DB_FINAL IMDB_URLS EXPORT_PATHSGEMMA_LOADER_PATHGEMMA_LOADER_PASSWORDLOG_DIR MAX_DIRECTORS MAX_ACTORSDUCKDB_THREADSDUCKDB_MEMORY_LIMITPARQUET_OUTPUT_PATHPARQUET_PUBLISH_PATHS)ImdbDownloader)ImdbTransformer)AccessExporterlog_filecd}d}tjg}|rE|jdd|tj|dtjtj|||dS)Nz4%(asctime)s - %(name)s - %(levelname)s - %(message)sz%Y-%m-%d %H:%M:%STparentsexist_okzutf-8)encoding)levelformatdatefmthandlers)logging StreamHandlerparentmkdirappend FileHandler basicConfigINFO)r log_format date_formatr!s C:\PYTHON\ImdbUpdate\main.py setup_loggingr-sGJ%K%''(HIdT:::+HwGGGHHH l cDtdtjr;tdtt jttjdtdtdS)Nz!Preparazione directory temporaneaz Eliminazione T)rzDirectory creata: )loggerinforexistsshutilrmtreer%r.r,prepare_temp_directoryr6.s KK3444  .H../// h N4     KK/X//00000r.cTtdtdtdtt}|t }tdt |d|S)N<============================================================zFASE 2: Download dataset IMDbzDownload completato: z file)r0r1rrdownload_and_extract_allr len) downloaderfiless r,download_imdb_datasetsr=9s KK KK/000 KK))J  / / : :E KK9E 999::: Lr.r<returnctdtdtdtttt t tt5}| |}dddn #1swxYwYtd|ddtD]}t|tj z } t| dd tjt|td |{#t $r*}td |d |Yd}~d}~wwxYwdS) uM Step 3: DuckDB transform → imdb.parquet + copia in publish paths. r8u)FASE 3: Transform DuckDB → imdb.parquet)temp_dirparquet_output max_directors max_actorsduckdb_threadsduckdb_memory_limitNzTransform completato: ,z righe in parquetTrz Parquet copiato in: z Copia parquet fallita in z: )r0r1rrrrrrr transformrrnamer%r3copy2 Exceptionwarning)r< transformer row_countdest_dirdestes r,transform_and_export_parquetrQEs KK KK;<<< KK *#%/    1 ))%00 111111111111111 KKGGGGGHHH*JJH~~ 3 88 J NN  = = = L,d 3 3 3 KK777 8 8 8 8 J J J NNHHHQHH I I I I I I I I J 4s+B$$B(+B(4AE F E??Fcbtdtdtdt}ttz }|s9td|tddS||t| ttd| t| ttdS)u9Step 4: Parquet → Access (.accdb) + copia destinazioni.r8u$FASE 4: Export Parquet → MS AccesszTemplate non trovato: z:Crea cartella Empty/ e inserisci db_split_IMDB.accdb vuotoFIMDBT)r0r1rrrr2errorcreate_access_dbr export_parquet_to_accessrenrich_ott_in_accesscopy_to_destinationsr )exportertemplates r,export_to_accessr[fs KK KK6777 KKH--H ??   8h88999 QRRRu h888 %%&9?FSSS !!/222 !!/<@@@ 4r.cvtdtdtd t}|tt t dS#t$r(}td|Yd}~dSd}~wwxYw)u=Step 5: IMDB → GemmaLoader via tabella linkata (opzionale).r8z-FASE 5: Copia IMDB in GemmaLoader (opzionale)z'Copia GemmaLoader fallita (opzionale): N) r0r1rcopy_to_gemma_loaderrr r rJrK)rYrPs r,import_to_gemma_loaderr^|s KK KK?@@@ KKF!##%%&9;LNcddddd FFFDDDEEEEEEEEEFs4B B8B33B8cZtdtdtd tjddtjdD]}|dt jdd}t|z }| td |dS#t$r(}t d |Yd}~dSd}~wwxYw) Nr8zFASE 6: Scrittura log finaleTrzUPDATE_IMDB *.txtzUPDATE_IMDB al z%d-%m-%yz.txtz Log creato: zErrore scrittura log: ) r0r1rr%globunlinkrnowstrftimetouchrJrK)old_log log_filenamerrPs r,write_log_filergs+ KK KK./// KK 5 dT2222|$788  G NN    R)@)@)L)LRRR \) -8--..... 5553334444444445sB&C88 D*D%%D*c 8tdtdtdtjstddSd}dD]u}tj|D]^}|j}|||z }td|jd|d z d d _vtd |d z d d dS)z6Pulizia completa file temporanei (tsv.gz, txt, accdb).r8zFASE 7: Pulizia file temporaneiz(Directory Temp/ non esiste, skip puliziaNr)*.tsv.gz*.txt*.accdb*.duckdb *.duckdb.wal Eliminato: z (@.2f GB)zSpazio liberato totale: z GB) r0r1rr2r`statst_sizerarH) total_freedpatternfsizes r,cleanup_temp_filesrxs  KK KK1222 KK ?   >???KONNw'' N NA6688#D HHJJJ 4 K KKLLL$'2BLLLL M M M M  N  KKK;'+BKKKKLLLLLr.c2tjsdStdt tjdt tjdzt tjdzt tjdzt tjdz}|stddSt d |D}td t|d |d z d d|D]8}|td|j 9tddS)z9Pulizia file residui da esecuzioni precedenti interrotte.Nz!Pulizia file residui all'avvio...rirjrkrlrmzNessun file residuo trovatoc3HK|]}|jVdSN)rrrs).0rvs r, z%cleanup_on_startup..s->>!QVVXX%>>>>>>r.zTrovati z file residui (rorprqrnzPulizia completata) rr2r0r1listr`sumr:rarH)files_to_clean total_sizervs r,cleanup_on_startuprs{ ?   KK3444 X]: & &'' X]7 # #$$ % X]9 % %&& ' X]: & &'' ( X]> * *++  ,  1222>>~>>>>>J KK_3~..__zW?U____``` ..   ,AF,,---- KK$%%%%%r.F skip_download keep_tempctj}tdz d|ddz }t |t dt dt dt dtt dtt d t |st|st|rKt d td z td z td z tdz tdz d}nt}t|st ddStst ddSt!t#|st%nt dtj|z }t dt d|t ddS#t&$rt dYdSt($r(}t d|Yd}~dSd}~wwxYw)u Pipeline v2: download → DuckDB transform → parquet → Access → GemmaLoader Args: skip_download: Se True, salta download (usa file esistenti in Temp/) keep_temp: Se True, mantiene file temporanei logs imdb_update_z %Y%m%d_%H%M%Sz.logr8z'ImdbUpdate v2 - Avvio (DuckDB pipeline)zBase directory: zTemp directory: zParquet output: z.Download saltato - uso file esistenti in Temp/zname_basics.txtztitle_akas.txtztitle_basics.txtztitle_ratings.txtztitle_principals.txt) name_basics title_akas title_basics title_ratingstitle_principalszTransform fallitoFzExport Access fallitoz+File temporanei preservati (keep_temp=True)zCompletato con successo in Tz Interrotto dall'utentezErrore fatale: N)rrbrrcr-r0r1rrrr6rKr=rQrTr[r^rgrxKeyboardInterruptrJ exception)rr start_timerr<elapsedrPs r,mainrsJ& #\*2E2Eo2V2V#\#\#\\H( KK KK9::: KK KK-8--... KK-8--... KK8#6889999 !   % " $ $ $  - NNK L L L$,/@$@$,/?$?$,/A$A$,/B$B$,/E$E EE+,,E,E22  LL, - - -5 !!  LL0 1 1 15      G  KKE F F F,..:- F ;';;<<< Ft 1222uu .1..///uuuuus,rs3   33ttH~~,--...*)))))++++++......  8 $ $D$111   B, F F F555(MMM,&&&:MMMMMMM` zOOO $X $1R S S SF  :   0     Dd(.G  CH' QQq!!!!!-r.