OpenAI has unveiled its personal net crawler bot, GPTBot, and has offered net admins with the means to dam it in the event that they need to.
AI coaching strategies have develop into a scorching matter, with the trade nonetheless making an attempt to determine the legality and ethics of coaching AI fashions utilizing information on the web. OpenAI is addressing these considerations head-on, by giving net admins the flexibility to dam GPTBot.
Utilization
Internet pages crawled with the GPTBot consumer agent could doubtlessly be used to enhance future fashions and are filtered to take away sources that require paywall entry, are identified to assemble personally identifiable info (PII), or have textual content that violates our insurance policies. Permitting GPTBot to entry your website may help AI fashions develop into extra correct and enhance their normal capabilities and security. Beneath, we additionally share the best way to disallow GPTBot from accessing your website.
Disallowing GPTBot
To disallow GPTBot to entry your website you possibly can add the GPTBot to your website’s robots.txt:
Person-agent: GPTBot Disallow: /
