Bots and parsers

Often it happens that your site is visited by unwanted people / bots aim to either harm your site, or copy it. Such activities have a negative impact on your balance sheet and resources consumed by your Web application. For many of our customers this risk is the argument in the choice of host. We are well aware of these concerns and offer our solution to this problem.
 
We practice a refund for the activities of criminals. In other words, if your site is attacked by bots, you can block them. You can do it yourself (on page Nginx editor or using support services).
 
To independently monitor the activity on your site you need:

1. Enable access logs for the domain

You can do it in your domain settings. Click "Edit" button next to the desired domain, and enable "Access Logs".

2. Set the SSH connection to the server

You can find this information in our instructions.

3. Connect to the server via SSH and run the following command

grep `date +%d/%b/%Y`: ~/domains/logs/site.ru.access.log| awk '{ print $1}' | sort | uniq -c | sort -
Where,
  • ~/domains/logs/site.ru.access.log is the path to the log file, site, and 5/Oct/2015 - date.
The result of the command will be a list of IP addresses with the number of connections carried out during 5/Oct/2015. If you notice that the number of the request is too high, and it is not Google or Yandex search bot, lock it.
 
On the edit page Nginx config domain, add the desired IP address to the blocking list of IP addresses (pointing each IP with a new line).