aidZddlZddlZddlmZddlmZddlZddlmZddl Z e j e Z GddZ dZe d kr edSdS) zq Modulo per download e decompressione file IMDb Sostituisce ScaricaFile() e la decompressione PowerShell del VBA N)Path)Dict)tqdmceZdZdZdefdZdededefdZdd ed edefd Zd edefd Z de eefde eeffdZ dS)ImdbDownloaderz3Gestisce download e decompressione dei dataset IMDb download_dircft||_|jdddS)zO Args: download_dir: Directory dove salvare i file T)parentsexist_okN)rrmkdir)selfrs &C:\PYTHON\ImdbUpdate\src\downloader.py__init__zImdbDownloader.__init__s6 !.. t<<<<<urlfilenamereturnc|j|z }td|d| tj|dd}|t |jdd}t|d5}t|d dd | 5}| d D];}|r7| || t|< dddn #1swxYwYdddn #1swxYwYtd||S#tj$r&} td|d| d} ~ wwxYw)a Scarica un file da URL con progress bar Sostituisce: VBA ScaricaFile() con MSXML2.XMLHTTP Args: url: URL del file da scaricare filename: Nome del file di destinazione Returns: Path del file scaricato z Download z da Ti,)streamtimeoutzcontent-lengthrwbBi)totalunit unit_scale unit_divisordesci ) chunk_sizeNzDownload completato: zErrore download : )rloggerinforequestsgetraise_for_statusintheadersopenr iter_contentwriteupdatelenRequestExceptionerror) r rrfilepathresponse total_sizefpbarchunkes r download_filezImdbDownloader.download_files$x/ 333c33444 |CcBBBH  % % ' ' 'X-112BAFFGGJh%% 4$#!%!  4 !)!6!6$!6!G!G44 4GGENNN KKE 3334 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 KK::: ; ; ;O(    LL;H;;;; < < <  saA#ED$AD8 DD D D D EDE"D#!EE:!E55E:Ngz_file output_filec||d}td|jd|j t j|d5}t |d5}t j||dddn #1swxYwYdddn #1swxYwYtd||S#t$r+}t d|jd |d}~wwxYw) a Decomprime un file .gz Sostituisce: PowerShell GZipStream del VBA Args: gz_file: File .gz da decomprimere output_file: File di output (opzionale, default rimuove .gz) Returns: Path del file decompresso NzDecompressione  -> rbrzDecompressione completata: zErrore decompressione r) with_suffixr r!namegzipr'shutil copyfileobj Exceptionr-)r r6r7f_inf_outr4s rdecompress_gzipzImdbDownloader.decompress_gzipEs  !--b11K JglJJ 8HJJKKK 7D)) 4T+t,,4&tU333444444444444444 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 KKCkCC D D D     LLE',EE!EE F F F  s`C B")B ? B" B B"B B" C "B&&C )B&*!C D&C<<Dtsv_filec|d}|rA||td|jd|j|Std||S)z Rinomina .tsv in .txt (come nel VBA originale) Args: tsv_file: File .tsv da rinominare Returns: Path del file rinominato z.txtz Rinominato r:zFile non trovato: )r<existsrenamer r!r=warning)r rEtxt_files r rename_to_txtzImdbDownloader.rename_to_txtcs''// ??    OOH % % % KKHhmHHHH I I IO NN::: ; ; ;Orurlsci}|D]\}} |d}|||}||}||}|||<Q#t$r*} t d|d| Yd} ~ d} ~ wwxYwt dt|dt|d|S)z Download e estrazione di tutti i file IMDb Sostituisce: Loop VBA per download + decompressione Args: urls: Dictionary {nome_dataset: url} Returns: Dictionary {nome_dataset: path_file_txt} z.tsv.gzzErrore elaborazione rNz Processati /z file) itemsr5rDrKrAr r-r!r+) r rLextracted_filesr=r gz_filenamer6rErJr4s rdownload_and_extract_allz'ImdbDownloader.download_and_extract_allws  ID# !%... ,,S+>> //88 --h77(0%%     ?D??A??@@@   I#o"6"6IITIIIJJJsA A(( B2 BB)N) __name__ __module__ __qualname____doc__rrstrr5rDrKrrRrrrrs===T====(((((((Tt$$<dt($T#s(^$S$Y$$$$$$rrcddl}|jdtt t jjddlm}m }tj tj dt|}d|di}||}td|D]\}}td|d |dS) zTest del modulorN) IMDB_URLSTEMP_DIRz4%(asctime)s - %(name)s - %(levelname)s - %(message)s)levelformat title_ratingsz File processati:z r)syspathinsertrWr__file__parentconfigrZr[logging basicConfigINFOrrRprintrO)r_rZr[ downloader test_urlsfilesr=r`s rmainrlsJJJHOOAs4>>0788999******** lE  ))J!)O"<=I  / / : :E kkmm## d !4!!4!!""""##r__main__)rVr>r?pathlibrtypingrr"rre getLoggerrSr rrlrXrrrqs   8 $ $KKKKKKKK\###, zDFFFFFr