Ngaphakathi i-ConveyThis Tech: Ukwakha i-Website Crawler yethu
Ukuphucula amava oMsebenzisi: I-ConveyThis Yazisa uLawulo lwe-URL
Uninzi lweConveyLe abaxhasi bakhetha ukuba zonke ii-URL zewebhusayithi zabo ziguqulelwe ngokufanelekileyo, nto leyo enokuba ngumsebenzi onzima, ngakumbi kwiindawo ezibanzi eziguqulelwe kwiilwimi ezininzi.
Ingxelo yabasebenzisi ibonise ukuba abanye abathengi baye bafumanisa ukuba ukuqalisa kweeprojekthi zabo zoguqulelo lwewebhusayithi kuyamangalisa. Bahlala bebuza ukuba kutheni bekwazi ukujonga kuphela i-URL yephepha lasekhaya kuluhlu lwenguqulelo, kunye nendlela yokuyila iinguqulelo zesiqulatho sabo.
Oku kwabonisa indawo enokubakho yophuculo. Sibone ithuba lokuququzelela inkqubo yokungena etyibilikayo kunye nolawulo lweprojekthi olusebenzayo. Noko ke, sasingenaso isicombululo esibambekayo ngelo xesha.
Isiphumo, njengoko usenokuba uye waqikelela, yaba kukwaziswa kwenqaku loLawulo lwe-URL. Ivumela abasebenzisi ukuba baskene ii-URL zewebhusayithi yabo kwaye bavelise umxholo wabo oguqulelweyo nge- ConveyLe Dashboard, ngokukhawuleza nangempumelelo.
Kutshanje, eli nqaku liye lasuswa kuLuhlu loGuqulelo lasiwa kwiphepha elitsha lolawulo loguqulelo olusekwe kwi-URL entsha. Ngoku, sikholelwa ukuba lixesha lokutyhila ibali elingemva kokusungulwa kweli nqaku.
Ukwanga iGolang: Hambisa Olu' Uhambo olusingise kwiiNkonzo zoGuqulelo eziPhuculweyo
Ukuqala kokuvalwa komsebenzi ngo-2020 ngenxa yobhubhane kwandinika ithuba lokuba ndifunde ulwimi lwenkqubo yesiGolang olwalubekwe ecaleni ngenxa yeengxaki zexesha.
Iphuhliswe nguGoogle, iGolang okanye iGo iye yafumana ukuthandwa kwiminyaka yakutshanje. Ulwimi lwenkqubo oluqokelelwe ngokweenkcukacha, iGolang yenzelwe ukuba abaphuhlisi benze ubugcisa obusebenzayo, obuthembekileyo, kunye nekhowudi efanayo. Ubulula bayo buxhasa ukubhalwa nokugcinwa kweenkqubo ezibanzi nezinzima ngaphandle kokuncama isantya.
Xa ndicinga ngeprojekthi esecaleni enokubakho ukuze ndiziqhelanise noGolang, umkhandi wewebhu wathi qatha engqondweni yam. Ifikelele kwiikhrayitheriya ezikhankanyiweyo kwaye inokunikwa isisombululo kubasebenzisi be-ConveyThis. I-web crawler okanye i-'bot' yinkqubo etyelela iwebhusayithi ukukhupha idatha.
Kwi-ConveyThis, injongo yethu yayikukuphuhlisa isixhobo sabasebenzisi ukuskena indawo yabo kwaye babuyise zonke ii-URL. Ukongeza, besifuna ukulungelelanisa inkqubo yokuvelisa iinguqulelo. Okwangoku, abasebenzisi kufuneka bandwendwele iwebhusayithi yabo ngolwimi oluguqulelweyo ukuze bavelise, umsebenzi oye ube nzima kwiindawo ezinkulu, ezineelwimi ezininzi.
Nangona iprototype yokuqala yayithe ngqo - inkqubo ethatha i-URL njengegalelo kwaye iqala ukukhasa kwisiza - yayikhawuleza kwaye isebenza. U-Alex, u-ConveyThis' CTO, wabona ukubakho kwesi sisombululo kwaye wanika ithuba lokuya phambili kuphando kunye nophuhliso lokucokisa ingqikelelo kunye nokucamngca ngendlela yokusingatha inkonzo yemveliso yexesha elizayo.
Ukujonga i-Serverless Trend nge-Go kunye ne-ConveyThis
Kwinkqubo yokugqibezela i-bot ye-web crawler, sizifumene sijijisana nee-nuances ezahlukeneyo ze-CMS kunye nokudityaniswa. Umbuzo ke waphakama - singabonisa njani ngcono abasebenzisi bethu nge-bot?
Ekuqaleni, siye saqwalasela indlela ezanyiweyo kunye nevavanyiweyo yokusebenzisa i-AWS ngojongano lweseva yewebhu. Nangona kunjalo, iingxaki ezininzi ezinokuthi zavela. Besinokungaqiniseki malunga nomthwalo weseva, ukusetyenziswa ngaxeshanye ngabasebenzisi abaninzi, kunye nokungabi namava kwethu ngokusingathwa kwenkqubo yeGo.
Oku kwasikhokelela ekubeni sithathele ingqalelo imeko yokubamba engenaseva. Oku kubonelele ngezibonelelo ezinjengolawulo lweziseko zophuhliso ngumboneleli kunye nokukaleka okungokwemvelo, oko kuyenza ibe sisisombululo esifanelekileyo se-ConveyThis. Kwathetha ukuba akufuneki sikhathazeke malunga nomthamo weseva kuba isicelo ngasinye siza kusebenza kwisikhongozeli saso sodwa.
Nangona kunjalo, umva ngo-2020, ikhompyuter engenaseva yeza nomda wemizuzu emi-5. Oku kungqineke kuyingxaki kwi-bot yethu enokuthi ifuneke ukukhasa iisayithi ezinkulu ze-e-commerce ezinamaphepha amaninzi. Ngethamsanqa, ekuqaleni kuka-2020, i-AWS yandisa umda ukuya kwimizuzu eli-15, nangona ukwenza eli nqaku kubonakale kungumsebenzi onzima. Ekugqibeleni, sifumene isisombululo ngokuqalisa ikhowudi engenamncedisi kunye ne-SQS - inkonzo ye-AWS yokufola umyalezo.
Uhambo oluya kuNxibelelwano lweBot yeXesha elililo ngeConveyThis
Njengoko sasicombulula ingxaki yokusingatha imicimbi, saba nomnye umqobo ekwakufuneka siwoyise. Ngoku besine-bot esebenzayo, ebanjwe ngendlela esebenzayo, enokunyuka. Umsebenzi oseleyo ibikukubuyisela idatha eyenziwe ngebhot kubasebenzisi bethu.
Ndijonge ukusebenzelana okuphezulu, ndigqibe kunxibelelwano lwexesha lokwenyani phakathi kwe-bot kunye nedeshibhodi yeConvey. Ngelixa ixesha lokwenyani lingeyomfuneko kwinqaku elinjalo, bendifuna abasebenzisi bethu bafumane impendulo kwangoko yakuba i-bot iqalile ukusebenza.
Ukufezekisa oku, siphuhlise iseva yewebhu ye-Node.js elula, ebanjwe kumzekelo we-AWS EC2. Oku kufuna ezinye ii-tweaks kwi-bot yonxibelelwano kunye nomncedisi we-websocket kunye nokuthunyelwa ngokuzenzekelayo. Emva kovavanyo olucokisekileyo, sasikulungele ukutshintshela kwimveliso.
Into eyaqala njengeprojekthi esecaleni ekugqibeleni yafumana indawo yayo kwideshbhodi. Ngokusebenzisa imiceli mngeni, ndiye ndafumana ulwazi kwi-Go kwaye ndaphucula izakhono zam kwindawo ye-AWS. Ndifumene i-Go iluncedo kakhulu kwimisebenzi yenethiwekhi, inkqubo yentsebenziswano, kunye nekhompyuter engenaseva, inikwe inkumbulo ephantsi.
Sinezicwangciso zexesha elizayo njengoko i-bot izisa amathuba amatsha. Sijonge ukubhala kwakhona isixhobo sethu sokubala amagama ukuze sisebenze kakuhle, kwaye sinokubanako ukusisebenzisa ukufudumala kwe-cache. Ndiyathemba ukuba ukonwabele oku kukroba kwihlabathi letekhnoloji ye-ConveyThis kangangoba ndikonwabele ukwabelana ngayo.
Ukulungele ukuqalisa?
Ukuguqulela, okungaphezulu lee kunokwazi nje iilwimi, yinkqubo entsonkothileyo.
Ngokulandela iingcebiso zethu kunye nokusebenzisa i-ConveyThis , amaphepha akho aguqulelweyo aya kuhambelana nabaphulaphuli bakho, bazive bevela kulwimi ekujoliswe kulo.
Ngoxa kufuna umgudu, umphumo uba nomvuzo. Ukuba uguqulela iwebhusayithi, i-ConveyThis ingakongela iiyure ngomatshini woguqulelo oluzenzekelayo.
Zama i-ConveyThis simahla kangangeentsuku ezisi-7!
