
內容簡介
Booter – 機器人及爬蟲管理是預防措施(事先處理)及處理機器人及爬蟲造成的損害的外掛程式。
此外掛程式使用多種現有技術,進一步智能化並幾乎完全自動化。
為使此外掛程式運作正確,需要遵從說明並手動輸入一些資料(必須由人類操作以避免錯誤)。
預防層面
Booter 允許您管理和建立進階的動態 robots.txt 檔案。
查看 404 錯誤記錄以查看最常見的錯誤連結。
阻止造成高負載的有害機器人,因為它們經常爬取網頁或用於搜尋安全漏洞。
處理層面
Booter 允許您限制爬蟲及機器人的請求數量,如果或當它們超過每分鐘指定的請求數量,它們將被拒絕一段指定期間。
駁回我們不想要的連結,不只是封鎖,而是通過發送適當的 HTTP 狀態碼來使搜尋引擎忘記它們。
治療損害的使用說明
啟動此插件程式。
啟用 404 錯誤記錄選項。
設定存取速率限制。
監視 404 記錄,嘗試找到在 URL 中最常出現的共同部分。
將共同部分輸入到「駁回連結」頁面,並確保駁回程式碼為 410。
清除 404 錯誤記錄。
重複此程序直到每幾小時的 404 錯誤記錄保持空白。
每隔幾天檢查您的網站索引涵蓋範圍的狀態。
外掛標籤
開發者團隊
② 後台搜尋「Booter – Bots & Crawlers Manager」→ 直接安裝(推薦)
原文外掛簡介
Booter – Bots & Crawlers Manager is a preventative measure (treatment in advance) and treatment of damages caused by crawlers and bots.
The plugin uses a number of existing technologies which are known by crawlers and bots and takes them one step forward – smartly and almost completely automatically.
To allow the plugin to function correctly, you must follow the instructions and manually enter some data (which must be done by a human being to avoid errors).
At the prevention level
Booter allows you to manage and create an advanced dynamic robots.txt file.
View a 404 error log to see the most common bad links.
Blocking bad bots that cause high server loads due to very frequent page crawls, or are used to search for security vulnerabilities.
At the treatment level
Booter allows you to limit the amount of requests from crawlers and bots, if or when they exceed the specified amount of requests per minute, it will be rejected for a specified period of time.
Rejecting links that we do not want in the fastest way, not by just blocking but by sending the appropriate HTTP status code to make search engines forget them.
Instructions for use in case of damage treatment
Activate the plugin.
Enable the 404 error log option.
Set the access rate limit.
Watch the 404 log, try to find common parts in the URLs that repeats most often.
Enter the common parts to the “reject links” page, and ensure the rejection code is 410.
Clear the 404 error log.
Repeat the process once every few hours until the 404 error log remains blank.
Check the status of your website’s index coverage every few days.
