a-i`dZddlmZmZddlmZmZmZmZddlZ ddl Z ddl m Z ddlmZddlmZmZmZGdd Zy) z_ Yahoo Finance Data Collector Sprint 1: Foundation - Market data collection from Yahoo Finance )datetime timedelta)DictListOptionalTupleN)logger)DatabaseManager)FundamentalData PriceDataStockc $eZdZdZddededefdZd dededed e d e f d Z d!ded ed ede e fdZ d"dedede de fdZdede fdZde defdZdeedeee ffdZdede efdZde ede efdZde efdZdede eeeffdZy)#YahooFinanceCollectorz:Collects market data from Yahoo Finance for Italian stocks db_manager max_retries retry_delayc.||_||_||_y)z Initialize Yahoo Finance collector Args: db_manager: Database manager instance max_retries: Maximum number of retry attempts retry_delay: Delay between retries in seconds N)rrr)selfrrrs F/mnt/ssd/data/python-lab/Trading/src/data_collector/yahoo_collector.py__init__zYahooFinanceCollector.__init__s%&&tickerstartendprogress auto_adjustcJddl}t|jD] } tj|||||}|cStjS#t $r} ||jdz kr_|j d|zz} tjd|d|dzd|jd t| d | d |j| nLtjd|d |jd t| tjcYd} ~ cSYd} ~ d} ~ wwxYw)a_ Download data with retry logic for network failures. Args: ticker: Stock ticker symbol start: Start date end: End date progress: Show progress bar auto_adjust: Auto adjust prices Returns: DataFrame with price data or empty DataFrame on failure rNrrrrzDownload failed for (attempt /): . Retrying in s... after attempts: )timerangeryfdownload Exceptionrr warningstrsleeperrorpd DataFrame) rrrrrrr(attemptdfe wait_times r_download_with_retryz*YahooFinanceCollector._download_with_retry!s5 T--. *G *[[u#^ij  *||~ *T--11 $ 0 0AL AINN%9&GVWK=XYZ^ZjZjYkknorstounvwEFOEPPT$UVJJy)LL#7xwtGWGWFXXcdghidjck!lm<<>)*  *sA D"B5DD"D"prioritymarketreturnc F |j|}|stjd|y|jj 5}|j t j|j}|r5tjd|d|j||cdddSt ||jd||jd|jd||d }|j||j|j|tjd |d |j|j||cdddS#1swYyxYw#t $r%}tj"d |d|Yd}~yd}~wwxYw)a^ Add a new stock to the database Args: ticker: Stock ticker symbol (e.g., 'ENEL.MI', 'MC.PA', 'SAP.DE') priority: Stock priority level ('primary' or 'secondary') market: Stock market name (default: 'Borsa Italiana') Returns: Stock object if successful, None otherwise z Could not fetch info for ticker NrStock z already exists in databaselongNamesectorindustryT)rnamer?r@r9r8 is_activez Added stock: z - zError adding stock : )_get_stock_infor r-r get_sessionqueryr filter_byfirstinfoexpungegetaddcommitrefreshrAr,r0) rrr8r9 stock_infosessionexisting_stockstockr5s r add_stockzYahooFinanceCollector.add_stock@sm) --f5J!A&JK,,. '!(u!5!?!?v!?!N!T!T!V!KK&0K LMOON3)  !# F;%>>(3'^^J7!%" E" & mF83uzzlCD&;   >  LL.vhb< = sH+E2E2A!E&) E23B)E& E2&E/+E2/E22 F ;FF days force_updatecx |jj5}|jtj |j }|s#t jd|d dddy|sh|jtj |jj}|dkDrt jd|d |d  dddy |jtj |jj}|j|dkDrt jd |d |dnt jd|dtj}|t!|z } t jd|d| j#d|j#|j%|| |dd} | j&r"t j(d| dddyt jdt+| d||dz dz} t+| | dzkr5t j(dt+| d|dt-| d|d n%t jd t+| d!|d"t/| j0t2j4r | j0j7d| _d} d} | j9D]+\}}d#}|jtj |j|$j }|r|s||d%t;||d%n |j<|_||d&t;||d&n |j>|_||d't;||d'n |j@|_ t;||d(|_!||d)t-||d)n |jD|_"||d*t;||d*n |jF|_#tj|_$| d+z } at|j|||d%t;||d%nd||d&t;||d&nd||d't;||d'ndt;||d(||d)t-||d)nd||d*t;||d*ndd,- }|jK|| d+z } .|j| dkDrt jd.| d/| d |nt jd.| d0| dddy #1swYyxYw#tL$r%}t jd1|d2|Yd}~yd}~wwxYw)3a. Collect historical price data for a stock Args: ticker: Stock ticker symbol days: Number of days of historical data to collect force_update: Force update even if data exists Returns: True if successful, False otherwise r<r= not found in databaseNFstock_idrzPrice data for z already exists (z records)Tu🗑️ Deleted z existing price records for z (force_update mode)u🆕 No existing data for z, starting fresh downloadrTzDownloading price data for z from z to rzNo price data available for z Downloaded z rows from Yahoo Finance for gv@?u!⚠️ WARNING: Downloaded only z rows for z , expected ~z; days. Yahoo Finance may have limited data for this ticker.u✓ Data validation passed: z rows is adequate for z days periodc ||}t|tjrt|dkDr|jdnd}tj |r|SdS#t tf$rYywxYwNr isinstancer1SerieslenilocnotnaKeyError IndexErrorrowcolvals r get_valuezBYahooFinanceCollector.collect_historical_prices..get_valuesc("%c(C)#ryy958X\chhqkt*,((3-3ATA (*5(#'(AAAA+*A+rYdateOpenHighLowCloseVolume Adj Closeryfinance rYrnopenhighlowclosevolume adj_close data_sourceAdded z new, updated z price records for z'Error collecting historical prices for rC)'rrErFr rGrHr r0r idcountrIdeleterMrnowrrnr7emptyr-rbintr`columnsr1 MultiIndexget_level_valuesiterrowsfloatrwrxryrzr{r| created_atrLr,)rrrTrUrPrRexisting_count deleted_countend_date start_dater4 expected_rows records_addedrecords_updatedrnrhrkexisting price_datar5s rcollect_historical_pricesz/YahooFinanceCollector.collect_historical_priceswsu ,,.p ' e,66f6EKKMLL6&1G!HI p p $ i0"EHH5# &) ofX=N~N^^g$hi##p p * i0"EHH5" NN$$q( &7 Fbcibjj~$A &@Ha$bc$<<>% t(<<  9& HYGZZ^_g_l_l_n^opq..vZX`esx.y88NN%A&#JK Sp p V k#b'2OPVxXY!%u s 2 r7]S00NN;CG9JvhW%%(%7$8 4&IOP KK">s2wiG]^b]cco pqbjj"--8!#!jnBKCQWBXBdyf'=!>jn@I#u@U@aiU&; |jd?d@|jdAdB|jdCr|jdCdDz nddE|jdFdG|dH| dI|dJ|dK|dL|jdMdN|jdOdP|jdQdR|jdSdT|jdUdV|jdVdW|dX|dY| dZ|jd[d\| d]| d^| d_|jd`da|jdbdc|jdcdd|jdedf|jdfdg|jdhdi|jdjdk|jdldm|jdndo|j7|jdpdq|j7|jdrdsdtdudv} ddwlm}|}|j=||_|jA||jBxsdx|_"|jG||_$t j&|dy|j>dz|jDd{|jH|jK||jM|jO|||t jPd}| dddy~#t($r%}t j|d|Yd}~[d}~wwxYw#t($r$}t j|d||Yd}~d}~wwxYw#1swYyxYw#t($r%}t jd|d|Yd}~yd}~wwxYw)z Collect fundamental data for a stock Args: ticker: Stock ticker symbol Returns: True if successful, False otherwise r<r=rWNFz"No fundamental data available for EURUSDcurrencyu ⛔ Skipping z: Currency is 'z ', expected . totalDebt totalCashoperatingCashflowcapitalExpenditures freeCashflow debtToEquityrzStockholders Equityz$Total Equity Gross Minority InterestzStockholder Equityz%: D/E calculated from balance sheet: z.4fz (debt=z,.0fz , equity=)z5: Could not fetch balance sheet for D/E calculation: balance_sheet gY@r\z,: D/E discrepancy detected - Balance Sheet: z , Yahoo: z (raw: z.2fz ), Diff: yahoo_normalizedz+: D/E from Yahoo appears to be percentage (z), normalizing to yahoo_rawmissingz': No D/E data available from any sourcerYrnpe_ratio trailingPEpb_ratio priceToBookps_ratiopriceToSalesTrailing12Months peg_ratiopegRatio ev_to_ebitdaenterpriseToEbitda profit_margin profitMarginsoperating_marginoperatingMarginsroereturnOnEquityroareturnOnAssetsdividend_yield dividendYieldd payout_ratio payoutRatiodebt_to_equitydebt_to_equity_rawdebt_to_equity_calculateddebt_to_equity_sourceshareholders_equity current_ratio currentRatio quick_ratio quickRatiorevenue_growth revenueGrowthearnings_growthearningsGrowth market_cap marketCapbeta total_debt total_cashnet_debtenterprise_valueenterpriseValueoperating_cashflowcapital_expenditures free_cashflow gross_margin grossMargins ebitda_margin ebitdaMarginsebitda total_revenue totalRevenueebit net_incomenetIncomeToCommon trailing_eps trailingEpsshares_outstandingsharesOutstanding dividend_rate dividendRateex_dividend_dateexDividendDatedividend_payment_datelastDividendDateschema_versionr}ru)WarrenAnalyzerz: Schema v4 metrics - ROIC: z, Interest Coverage: z , F-Score: z): Could not calculate Schema v4 metrics: zAdded fundamental data for Tz&Error collecting fundamental data for rC))rrErFr rGrHr r0rDr-rKr*Tickerrrrbrindexlocdebugr,absr rrr_safe_extract_ratio _parse_datesrc.analysis.warren_analyzerrcalculate_roicroiccalculate_interest_coverager?interest_coveragecalculate_piotroski_fscorepiotroski_fscorerLrM_backfill_historical_fcfrI)rrrPrRrOACCEPTED_CURRENCIESrrr operating_cfcapexrrrrr ticker_objrlatestr5debt_to_equity_finalrr discrepancy fundamentalranalyzers rcollect_fundamental_dataz.YahooFinanceCollector.collect_fundamental_datasO ,,.J ' e,66f6EKKMLL6&1G!HI J J "11&9 !NN%Gx#PQ J J  (-en#%>>*5#66NN]6(/(S_`s_ttu#vw )J J .(^^K8 '^^K8 )~~.AB "'<= )j.D)J6H!+~ >  (\-E%J[$05$8M&0^^N%C"'+#,0)h!#6!2J$.$<$>/#BS#Hqu: ", !>!:$$8%:&(:':(/H):*+@+:,)<-:.#-.."@/:0!+| <1:4$.>>/#B5:6%/NN3C$D7:: *~~k:;:<$/=:B *C:D *E:F&G:H&0^^4E%FI:L(4M:N*/O:P#0Q:T",!?U:V#-.."AW:X&>>(3Y:\#-.."@]:^$/_:` *~~.ABa:b", !>c:f(2~~6I'Jg:h#-.."@i:j&*%5%5jnnEU6V%Wk:l+/*:*::>>J\;]*^m:p$%q:r!+s: z \K-/H'/'>'>v'FK$4<4X4XY_afamamasqs4tK13;3V3VW]3^K0LLF8+G HXHXGYZ66A6S6S5TU,,7,H,H+I"KL K( --gujI 9&BCUJ J R!hNNfX-bcdbe#fgghl!\NNfX-VWXVY#Z[[\J J X  LLA&A3O P s[1A[%,[15,[%![1*9[%#[1,B[%-C%ZL?[%BZ5.A [%;[1 Z2 Z-'[%-Z22[%5 [">[[%[""[%%[.*[1.[11 \:\\rRrOc  tj|j}|j}| |jry|j Dcgc] }t |jdvs|"}}|sy|j|d}t|jddD]} |j| } | tj| r, tj| j} |j#t$j'|j(| j+} | rt%|j(| t-| |jd|jddd } |j/| |j1ycc}w#t $rY wxYw#t $r/}t3j4d |jd |Yd}~yd}~wwxYw) z Store historical free cash flow entries (up to 3 periods) so that the analyzer can compute a 3y average FCF yield for cyclical sectors. Uses yfinance cashflow statement; falls back silently if unavailable. N) freecashflowzfree cash flowrrmrrr yfinance_cashflow)rYrnrrrrr}z&Could not backfill historical FCF for rC)r*rrcashflowrrr.lowerrlistrrKr1isna to_datetime to_pydatetimer,rFr rGrrHrrLrMr r-)rrPrRrOr cashflow_dfidxrow_keysfcf_rowrivalue period_daterfundamental_histr5s rrz.YahooFinanceCollector._backfill_historical_fcfs 0 Y5<<0J$--K"k&7&7(3'8'8sCHNN NN Mt!0 Y NNCELL>QSTUSVW X X Ys`9F+F+ F+F/F+4AF+ #F.B'F+F+ F($F+'F((F++ G#4%GG#tickersci}|D]} tjd||jj5}|j t j |j}|s(tjd|dd||< ddd|j tj |jjtjjj}|r|jtd z}n!tj td z }tj }|j|jkDr|td z }|j#|||dd } | j$r|rtj j|jjz j&} | d krDtjd |d|jjdd||< ddd2tjd |dd||< ddd[t)| j*t,j.r | j*j1d| _d} | j3D]\} } |j tj |j| j}|rHd}t|j| || dt5|| dnd|| dt5|| dnd|| dt5|| dndt5|| d|| dt7|| dnd|| dt5|| dndd }|j9|| dz } |j;tjd| d|d||<ddd|S#1swYxYw#t<$r+}tj>d|d |d||<Yd}~Bd}~wwxYw)!z Update daily prices for multiple stocks Args: tickers: List of stock ticker symbols Returns: Dictionary with ticker as key and success status as value zUpdating daily prices for r<r=z not found, skippingFNrXrrZrr zNo new data for z# (likely market closed, last data: rTz# (possible data issue or delisting)rrmc ||}t|tjrt|dkDr|jdnd}tj |r|SdS#t tf$rYywxYwr^r_rgs rrkz.get_value^sc,&)#h#-c299#=9vhFijujzjzjjkBjCCD-E!F26 (OY+Y+T)9&Ad'ef*/ YY+Y+^""**bmm<%'ZZ%@%@%C %&M%'[[]!+ c#MM)4&YtYD"UW! $$,&/%*XX!%BKCQWBXBdyf'=!>jnBKCQWBXBdyf'=!>jn@I#u@U@aiU&; 55s55cklqcr5pBGCC!$C:>C CJYsYtYv6Yu6Y$6Ypo49oc4iob(c(htn(TXc]x7I*HUO.sxho8N/Orr)r4rrtypingrrrrpandasr1rur*logurur src.database.db_managerr src.database.modelsr r r rrrrr;s3 )..3AAd d r