Packages and Binaries:

golang-github-temoto-robotstxt-dev

This package contains a robots.txt exclusion protocol implementation for Go language (golang).

This package contains the dev files.

Installed size: 54 KB
How to install: sudo apt install golang-github-temoto-robotstxt-dev


robotstxt

This package contains a robots.txt exclusion protocol implementation for Go language (golang).

Installed size: 5.17 MB
How to install: sudo apt install robotstxt

  • libc6
robots.txt-check
root@kali:~# robots.txt-check -h
Usage of robots.txt-check:
  -bot string
    	 (default "GoogleBot")
  -robots-url string
    	

Updated on: 2021-Nov-26