Packages and Binaries:

golang-github-temoto-robotstxt-dev

This package contains a robots.txt exclusion protocol implementation for Go language (golang).

This package contains the dev files.

Installed size: 54 KB
How to install: sudo apt install golang-github-temoto-robotstxt-dev

robotstxt

This package contains a robots.txt exclusion protocol implementation for Go language (golang).

Installed size: 5.17 MB
How to install: sudo apt install robotstxt

robots.txt-check

root@kali:~# robots.txt-check -h
Usage of robots.txt-check:
  -bot string
    	 (default "GoogleBot")
  -robots-url string