Block robots from accessing webpages using .htaccess

Use htaccess file.

this is run by the server and ensured by the server that the rules written in it are enforced.

On the other hand, the robots.txt file is just a guideline file. Robots are not bound to follow it.

To ensure denial of all requests for the restricted directory, put the following in the .htaccess file (keep it in the same directory),

deny from all

Allow access from one IP only and deny others,

order deny, allow
deny from all
allow from 255.255.12.34
ErrorDocument 403 errordoc.html

Allow access from a range of IPs (may be a LAN or a country),

order deny, allow
deny from all
allow from 255.255.0

Deny access from certain IP (e.g. a bot),

order allow, deny
deny from 255.255.123.456
allow from all

Block a bad bot

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^dirtybot
RewriteRule ^(.*)$ http://emptypage/

or use [F] to send a Forbidden signal

RewriteRule .* – [F]

Block more than one bots

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^dirtybot [OR]
RewriteCond %{HTTP_USER_AGENT} ^badCrawler [OR]
RewriteCond %{HTTP_USER_AGENT} ^fakeOne
RewriteRule ^(.*)$ http://emptypage/

Deny more than one IPs or bot IPs

order allow,deny
deny from 255.255.123.456
deny from 123.255.123.456
deny from 456.255.123.456
allow from all

Deny more than one IPs or bot IPs ranges (maybe countries)

order allow,deny
deny from 255.255.
deny from 123.456.
allow from all

Block particular ISPs,

order allow,deny
deny from this-bad-isp.com
deny from subdomain.bad-isp.com
allow from all

Block some referrers or websites from accessing your content like images or CSS or js files

RewriteEngine on
RewriteCond %{HTTP_REFERER} ^http://.*somebadforum\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*example\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*lastexample\.com [NC]
RewriteRule .* - [F]

Stop indexing files in a directory using .htaccess

How to stop indexing of files in a directory using .htaccess?

Sometimes we do not want to show all the files in a directory (which by default is a behavior of web servers).

That can be done using htaccess file.

Though there’s a simple solution to this, put an index.html file showing a decent message “Not authorized to access this”. But that will have little control over management.

What if we want to have more control and do it at the initial stage only?

Use .htaccess file.

Put this in your .htaccess file

IndexIgnore *

here * (asterisk) matches all the files so none would be displayed.

If you do not want to show .css or .conf or .php or .sh files only and let other files being shown, then,

IndexIgnore *.css
IndexIgnore *.php
IndexIgnore *.sh
IndexIgnore *.conf

or in a single line

IndexIgnore *.css *.php *.conf *.sh

This can be made to work for relative paths also if you want to put all your htaccess statements in a single file at the root level (recommended)

IndexIgnore /common-files/scripts/*

To make some file like “myfile.html” your default file (instead of index.html) which opens when a user enters a URL to access some directory, then use,

DirectoryIndex myfile.html

You can specify more than one file name, in case the first is not found then the immediate next one would be opened.

DirectoryIndex myfile_1.html myfile_2.html

On the contrary, if you want to show file index under a directory (opposite of stopping index) then use this,

Options +Indexes

Secure your files from accidental display to the users,

<Files ".htaccess">
    Order Deny,Allow
    Deny from all
    Allow from 122.255.43.3
</Files>

Use regular expressions (using ~) to specify file names (to secure files other than .htaccess)

<Files ~ "\.(gif|jpe?g|png)$">
   Order Deny,Allow
   Deny from all
  Allow from 122.255.43.3
</Files>

or

<FilesMatch "\.(gif|jpe?g|png)$">

How to increase file upload limit using .htaccess

Sometimes server settings do not allow you to upload files more than a fixed limit (usually 2 MB). But, some websites might require allowing files more than this limit. In such cases, we need to set some options in httpd.conf configuaretion file. This sometimes is not allowed on shared servers. “.htaccess” file comes to the rescue.

Put the below lines in a plain text file. Upload that to your home directory on your web server and rename it as “.htaccess”. This would allow an upload of up to 20 MB file sizes.

php_value upload_max_filesize 20M
php_value post_max_size 20M
php_value max_execution_time 200
php_value max_input_time 200

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *