ݫGi_dZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZmZdd lmZ dd lmZmZdZdZdZdZdZdZdZdZ dZ!ddZ"dZ#y#e$rGd d ZGddZY>wxYw)z3 Content fetching methods for Link Checker package N)urlparse) webdriver)Service)sync_playwright)requests)ContentFetchErrorSocialMediaError) get_proxy)ForeStyleceZdZdZdZdZy)r N)__name__ __module__ __qualname__YELLOWGREENRED@/var/www/html/utilities/link_checker_package/content_fetchers.pyr r srr ceZdZdZy)r rN)rrr RESET_ALLrrrr r s rr c tj}|jd|jd|jd|jd|r|dk7r|jd|z|jd|z|jd|jd d g|jd d |jd t |dd}tj ||}|j d|jd|j|tjd|j}|j} |j} |j|rd|vsd|vsd|vrd|vrtt j"dzt$j&zt)|} | r@t+| dkDr2tt j,dzt$j&zd| | gStt j.dzt$j&zgdSgdSd|vsd|vsd|vrgd Sd!|vrgd Sd"|vrgd#St+| dkrgd$St1| } | j2d%k(st+| j2d&krd'| | gSd| | gS#t4$r} t7d(| d)} ~ wwxYw)*a> Fetch content using Selenium Args: useragent (str): User agent string url (str): URL to fetch proxy (str): Proxy string cloudflare (bool): Whether to handle Cloudflare config (dict): Configuration Returns: list: [status, content, current_url] z --no-sandboxz --disable-gpuheadlesszwindow-size=1280,800unknownz--proxy=http://z --user-agent=z:--no-first-run --no-service-autorun --password-store=basicexcludeSwitcheszenable-automationuseAutomationExtensionFz---disable-blink-features=AutomationControlledpathschromedriver_path)serviceoptionszEObject.defineProperty(navigator, 'webdriver', {get: () => undefined})< Access denied Cloudflare Just a moment)Just a moment... detected, trying Aparser2&Aparser successfully extracted content200Aparser failed, returning 403403rrz not foundErreurError)404rrr2429)r3rr)600rr/300zSelenium content fetch failed: N)r ChromeOptions add_argumentadd_experimental_optionrChromeexecute_scriptset_page_load_timeoutgettimesleeptitle page_source current_urlquitprintr rr rAparserLinkExtractorlenrrrpath Exceptionr ) useragenturlproxy cloudflareconfigr#sdriverrAcontentrCaparser_contentoes rselenium_contentrUs;G))+^,_-Z(34 Ui'  !7%!? @_y89YZ''(9" " E>" " w<" " " [ ! 66S=CK!O7K0 0w ,, G"A! EFFGsHG=J#0J#1J#5J#J# J#J#'6J#J## J?,J::J?c r dd|zi}t|||}|d} |d} |d} |d} t5} |dk(s|s| jj}n| jj|}| | |j |d d }n0|j || d t | t | d dgd d| i}|j }|jd |j|d|j}|j}|j}|j|j|rd|vs d|vsd|vsd|vrd|vrt!t"j$dzt&j(zt+|}|rIt-|dkDr;t!t"j.d zt&j(zd!||gcd d d St!t"j0d"zt&j(zgdcd d d Sgdcd d d Sd#|vrd!d|gcd d d St3|}|j4d$k(st-|j4d%krd&||gcd d d St-|d'krt|||||cd d d Sd!||gcd d d S#t$r}dt|vr2|j|jddgcYd }~cd d d Sdt|vrgdcYd }~cd d d Sdt|vr=|j|jt|||||cYd }~cd d d S|j|jgdcYd }~cd d d Sd }~wwxYw#1swYy xYw#t$r}t7d(|d }~wwxYw))a~ Fetch content using Playwright Args: useragent (str): User agent string url (str): URL to fetch cur: Database cursor con: Database connection proxy (str): Proxy string cloudflare (bool): Whether to handle Cloudflare config (dict): Configuration Returns: list: [status, content, current_url] serverhttp://latitude longitudetimezonelocaler)rLNen-UST) user_agentr\ignore_https_errors)rZrY geolocationAccept-Language)r^ timezone_idr\r` permissionsr_extra_http_headersi`load) wait_untilSSL_ERROR_UNKNOWNrNS_ERROR_PROXY_FORBIDDENr.Timeout)r,rrr&r'r/r(r)r*r+r,r-z /search?q=r5r6r7z!Playwright content fetch failed: ) get_locationrfirefoxlaunch new_contextfloatnew_pageset_default_timeoutgotorIstrcloserUrArKrQrEr rr rrFrGrrrrHr )rJrKcurconrLrMrN proxy_to_uselocationlatlngr[r\pbrowsercontextpagerTrArCrQrRrSs rplaywright_contentris `I i%'  S%0z"{#J'(#  Q 1!iu))**,))***>{ck!--("(,."--( (".3CjeCj Q!.(,)6(. ##%D  $ $U + + #& 1$JJLE((KllnG JJL MMO"e+|u/DQVZimrZr&%/dkk,WWZ_ZiZiij*>s*C*s?/Cb/H!$**/W"WZ_ZiZi"ij$)?K#HAQ 1Q 1D"$((-L"Lu"^_#2GQ 1Q 1J /KQ 1Q 1N{*r;/QQ 1Q 1T%Avv}AFF aw 4YQ 1Q 1\7|c!' 3z6R_Q 1Q 1b7K0cQ 1Q 1D +&#a&0JJLMMO8OMQ 1Q 1N03q69**QQ 1Q 1R#a&(JJLMMO+IsE:vVVYQ 1Q 1\JJLMMO**aQ 1Q 1D +EQ 1Q 1f I"CA3 GHHIs2NB)NJ#1CN N0N NN NN% N/6N% N/N NN N# N ,0NN N! N+N;N <N N ;NN N N#N7N 8N< NN  NNNN N6#N11N6c t|}dd|zi}|d}d|zdz}d}t|d5} | j|dddt5} | jj d | } | j ||d d } | jjd d d | j} | jd | j|tjd| j#}| j$}| j'}| j!| j!|cdddS#1swYxYw#t$rD}dt|vr-| j!| j!Yd}~dddyYd}~d}~wwxYw#1swYyxYw#t$r}t)d|d}~wwxYw)z Process Instagram URLs Args: useragent (str): User agent string url (str): URL to fetch insta_user (dict): Instagram user data config (dict): Configuration Returns: str: Content string rWrXcookies {"cookies": }zinsta_cookies.jsonwNTrrLr]Europe/Tallinnr^ storage_stater\rb screenshots snapshotssources rirzInstagram processing failed: )r openwriterchromiumrmrntracingstartrprqrrr?r@rIrsrtrArKrQr )rJrK insta_userrNrLrwr new_cookies cookies_filefhr{r|r}r~rTrArCrQs rprocess_instagramrs+D&! i%' Y'$w.4 + , $ " HH[ ! "  !jj''\'JG))$*, *G OO ! !ddD ! Q##%D  $ $V ,  # 2JJLE((KllnG JJL MMO7   " "$ A&JJLMMO'   '   : D!>qcBCCDs-F,EF,A2F &E,A F 9 F,E F, F-FF  F,F FF  F)%F,)F,, G5GGc t|}dd|zi}|d}d|zdz}d} t| d5} | j|dddt5} | jj d | } | j || d d } | jjd d d | j}|jd |j|tjd|j!}|j"}|j%}d|vsd|vr dddy|j| j|cdddS#1swYxYw#t$r2}|j| jYd}~dddyd}~wwxYw#1swYyxYw#t$r}t'd|d}~wwxYw)a Process Facebook URLs Args: useragent (str): User agent string cur: Database cursor url (str): URL to fetch fb_user (dict): Facebook user data config (dict): Configuration Returns: str: Content string rWrXrrrzfacebook_cookies.jsonrNTrr]rrrrr%rzLog into Facebookz logout.phpzFacebook processing failed: r rrrrrmrnrrrprqrrr?r@rIrtrArKrQr )rJrurKfb_userrNrLrwrrrrr{r|r}r~rTrArCrQs rprocess_facebookrs.C&! i%' )$$w.4 . , $ " HH[ ! "  !jj''\'JG))$*, *G OO ! !ddD ! Q##%D  $ $V ,  # 2 JJLE((KllnG"g-W1L3  6 JJL MMO=   " "$   %     @ C!=aSABBCs-F,EF,A2F &E",5F !F,*!F  F,EF," F+ F F F,FF  F)%F,)F,, G5GGcr t|}dd|zi}|d}d|zdz}d}t|d5} | j|dddt5} | jj d | } | j ||d d } | jjd d d | j} | jd | j|tjd| j!}| j"}| j%}| j| j|cdddS#1swYxYw#t$r2}| j| jYd}~dddyd}~wwxYw#1swYyxYw#t$r}t'd|d}~wwxYw)z Process TikTok URLs Args: useragent (str): User agent string url (str): URL to fetch tiktok_user (dict): TikTok user data config (dict): Configuration Returns: str: Content string rWrXrrrztiktok_cookies.jsonrNTrr]rrrrrrzTikTok processing failed: r)rJrK tiktok_userrNrLrwrrrrr{r|r}r~rTrArCrQs rprocess_tiktokrS*A&! i%' i($w.4 , , $ " HH[ ! "  !jj''\'JG))$*, *G OO ! !ddD ! Q##%D  $ $V ,  # 2 JJLE((KllnG JJL MMO5   " "$   %     8 A!;A3?@@A~-FEFA2F&E,A F9 FE F F  F9F=FF  FFFF F6#F11F6cr t|}dd|zi}|d}d|zdz}d}t|d5} | j|dddt5} | jj d | } | j ||d d } | jjd d d | j} | jd | j|tjd| j!}| j"}| j%}| j| j|cdddS#1swYxYw#t$r2}| j| jYd}~dddyd}~wwxYw#1swYyxYw#t$r}t'd|d}~wwxYw)z Process Reddit URLs Args: useragent (str): User agent string url (str): URL to fetch reddit_user (dict): Reddit user data config (dict): Configuration Returns: str: Content string rWrXrrrzreddit_cookies.jsonrNTrr]rrrrrrzReddit processing failed: r)rJrK reddit_userrNrLrwrrrrr{r|r}r~rTrArCrQs rprocess_redditrrrc B d}|j||g|jrY|j}|d}|djd}|d|d|d|d|d |d |dj |d }|Sd d d d d d d d }|S#t $r d d d d d d d d cYSwxYw)z Get location data for proxy Args: cur: Database cursor con: Database connection proxy (str): Proxy string Returns: dict: Location data z.SELECT * FROM proxy_locations WHERE proxy = %srr\,cityregioncountryryrzr[)rrrrYrZr[r\rLN)rrrrYrZr[r\)executerowcountfetchallsplitstriprI)rurvrLsql proxy_dataresponseloca location_datas rrkrks% > C%! <<J!!}HH%++C0D ("8,#I.$UO%e_$Z0q'--/ M( ! M       sA9B< BBBc tdjj}tj|}|j S#t $r}td| tdjj}tj|}|j cYd}~S#t $r}td|Yd}~Yd}~yd}~wwxYwd}~wwxYw)zd Fetch a random proxy from proxy file Returns: str: Proxy string (ip:port) z/root/flask/proxies/proxy.txtz Error fetching proxy from file: z"/opt/aparser/files/proxy/proxy.txtNz)Error fetching proxy from fallback file: z127.0.0.1:8080)rread splitlinesrandomchoicerrIrE)linesrLrTe2s rfetch_proxy_from_filers $45::<GGI e${{}  $ 045 $=>CCEPPREMM%(E;;=  $ =bTB C## $ $sCA A C$C&A B71C$7 CCCCCC$cd} tj|d}|jdj}|S#t$r}t d|Yd}~yd}~wwxYw)z] Get user agent from Node.js script Returns: str: User agent string znode /root/user_agents.jsT)shellzutf-8zError getting user agent: NzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36) subprocess check_outputdecoder BaseExceptionrE)commandrJrTs rget_user_agent_from_scriptrsh*GA++G4@ $$W-335  A *1#./AAs7< AAAc  ttjd|ztjz|s t }|s t }d|}||d}td|td|ddd|d d d d d dddddd }tj}t|D]F} tj} td|dzd|d|j|||ddddd} tj| z } td| ddtd | jtd!| jd} t|} tt| j}t!| j"d"kD}|j"dk(xst!|j"d"k}|rT|rRd} | j$|j$k7rtd#|d$| jntd%|d$| j| jd&k(rT||dz krLtd'|dzd|d(tj&d" d)t)vs |j+t!| j,}| j.jd*}td+|d,|r0t1|}td-|d,||krtd.|d/|| jd0vrEd1| j2t| j| gd)t)vr |j+cScS| jd2vr\| rd3nt| j| j2t| j| gd)t)vr |j+cScSt| j| j2t| j| gd)t)vr |j+cScSd6d7|dgS#YUxYw#YcSxYw#YcSxYw#YcSxYw#tj4$rq}td4|dzd5|||dz k(r6d6d7|dgcYd}~d)t)vr |j+cS#YcSxYwcStj&d"Yd}~nd}~wt6$r}td8|dzd5|||dz k(rJd9dl}|j;d6d7|dgcYd}~d)t)vr |j+cS#YcSxYwcStj&d"Yd}~nd}~wwxYwd)t)vs |j+#YxYw#d)t)vr |j+w#YwxYwwxYw):au Fetch content using curl_cffi with proxy support and retry logic Args: url (str): URL to fetch useragent (str): User agent string (optional) proxy (str): Proxy string (optional) max_retries (int): Maximum number of retries for 500 errors Returns: list: [status_code, content, final_url, redirect_detected] zUsing curl_cffi for rX)httphttpsz Using proxy: zUsing User-Agent: Nr*z...z`text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8zen-US,en;q=0.9zgzip, deflate, brzhttps://www.google.com/z keep-alive1documentnavigatenonez max-age=0) z User-AgentAcceptrazAccept-EncodingReferer ConnectionzUpgrade-Insecure-RequestszSec-Fetch-DestzSec-Fetch-ModezSec-Fetch-Sitez Cache-ControlzFetching URL (attempt rr5z)...F)r$xT chrome110)proxiesheadersverifytimeoutallow_redirectsstream impersonatezRequest completed in z.2fz secondszStatus: z Final URL: r6z%Cross-domain home redirect detected: z -> zHome redirect detected: iz*Got 500 status code, retrying... (attempt )sessionzContent-LengthzContent length: z byteszExpected length: z(WARNING: Content may be incomplete! Got z bytes, expected )rjr,)i,i-i.i/i3i4r7zRequest error (attempt z): 500rzError (attempt r)rEr rr rrr curl_requestsSessionranger?r> status_coderKrrsrGrHnetlocr@localsrtrQrinttext RequestsErrorrI traceback print_exc)rKrJrL max_retries proxy_urlrrrattempt start_timer elapsed_timeredirect_detectedoriginal_parsed final_parsedoriginal_has_pathfinal_is_homepagecontent_lengthcontent_length_headerexpected_lengthrTrs rcurl_cffi_contentrs $++.se4 4u FG %' .0 %!I 95G M% !" y"~.c 23 t+.,"%($$ $ G##%G%RQ J *7Q;-q TJ K{{! $'# H 99;3L ),s);8D E HX1123 4 K ~. /!& &smO#C $56L!$O$8$8 9A = !-!2!2c!9!WSARAR=SVW=W  %6$(!"))\-@-@@A#d8<<.YZ4SEhll^LM##s*wq/HB7Q;-qQ\P]]^_` 1 HFH$MMOG!!1!12N$,$4$4$8$89I$J ! $^$4F; <$"%&;"<)/):&AB!O3D^DTTefuevwx##6x}}c(,,.?ARS*FH$MMO%)%%)GG!2HP PT#&!RT#UQ,,Q28R U T#5TT#U%S88S>TUT##U5UU  U6U.-U6.U2 0U6cDttjdz|ztjzd}dddd|ddd }d d d } t j |tj|| }|j}|jddk(r5ttjdztjz|ddSttjdztjzt|y#t$r}t|Yd}~yd}~wwxYw)z Extract content using Aparser when Cloudflare protection is detected Args: url (str): URL to extract content from Returns: str: Extracted content or empty string if failed zUSING APARSER zhttp://pma6.remov.ee:9096/API oneRequestz Net::HTTP secondpackdefault)parserpreset configPresetquerytWCj3yvB4v4US2wjfrSC)actiondatapasswordzapplication/textzUTF-8)z content-typezAccept-Charset)rrsuccessrzAParser response receivedr resultStringzAParser failed to fetch contentrN) rEr rr rrpostjsondumpsr>rrrI)rK parser_urlpayloadrresrrTs rrFrFs $++( (3 . @A0J!"%  + G 2WMG mmJTZZ-@'R88: << "a ' $**::U__L MF#N3 3 $((>>P Q (O  asBD8D D DD)NN)$__doc__r?rrrr urllib.parserseleniumr!selenium.webdriver.chrome.servicerplaywright.sync_apir curl_cffir exceptionsr r utilsr coloramar r ImportErrorrUrrrrrrkrrrrFrrrrs  !5//; $IGVpId8Dt<C|7Ar7Ar1 f$,A @#D&W sA//B  B