Use htaccess file.
this is run by the server and ensured by the server that the rules written in it are enforced.
On the other hand, the robots.txt file is just a guideline file. Robots are not bound to follow it.
To ensure denial of all requests for the restricted directory, put the following in the .htaccess file (keep it in the same directory),
deny from all
Allow access from one IP only and deny others,
order deny, allow deny from all allow from 255.255.12.34 ErrorDocument 403 errordoc.html
Allow access from a range of IPs (may be a LAN or a country),
order deny, allow deny from all allow from 255.255.0
Deny access from certain IP (e.g. a bot),
order allow, deny deny from 255.255.123.456 allow from all
Block a bad bot
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^dirtybot
RewriteRule ^(.*)$ http://emptypage/
or use [F] to send a Forbidden signal
RewriteRule .* – [F]
Block more than one bots
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^dirtybot [OR]
RewriteCond %{HTTP_USER_AGENT} ^badCrawler [OR]
RewriteCond %{HTTP_USER_AGENT} ^fakeOne
RewriteRule ^(.*)$ http://emptypage/
Deny more than one IPs or bot IPs
order allow,deny deny from 255.255.123.456 deny from 123.255.123.456 deny from 456.255.123.456 allow from all
Deny more than one IPs or bot IPs ranges (maybe countries)
order allow,deny deny from 255.255. deny from 123.456. allow from all
Block particular ISPs,
order allow,deny deny from this-bad-isp.com deny from subdomain.bad-isp.com allow from all
Block some referrers or websites from accessing your content like images or CSS or js files
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^http://.*somebadforum\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*example\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*lastexample\.com [NC]
RewriteRule .* - [F]
Stop indexing files in a directory using .htaccess
How to stop indexing of files in a directory using .htaccess?
Sometimes we do not want to show all the files in a directory (which by default is a behavior of web servers).
That can be done using htaccess file.
Though there’s a simple solution to this, put an index.html file showing a decent message “Not authorized to access this”. But that will have little control over management.
What if we want to have more control and do it at the initial stage only?
Use .htaccess file.
Put this in your .htaccess file
IndexIgnore *
here * (asterisk) matches all the files so none would be displayed.
If you do not want to show .css or .conf or .php or .sh files only and let other files being shown, then,
IndexIgnore *.css IndexIgnore *.php IndexIgnore *.sh IndexIgnore *.conf
or in a single line
IndexIgnore *.css *.php *.conf *.sh
This can be made to work for relative paths also if you want to put all your htaccess statements in a single file at the root level (recommended)
IndexIgnore /common-files/scripts/*
To make some file like “myfile.html” your default file (instead of index.html) which opens when a user enters a URL to access some directory, then use,
DirectoryIndex myfile.html
You can specify more than one file name, in case the first is not found then the immediate next one would be opened.
DirectoryIndex myfile_1.html myfile_2.html
On the contrary, if you want to show file index under a directory (opposite of stopping index) then use this,
Options +Indexes
Secure your files from accidental display to the users,
<Files ".htaccess">
Order Deny,Allow
Deny from all
Allow from 122.255.43.3
</Files>
Use regular expressions (using ~) to specify file names (to secure files other than .htaccess)
<Files ~ "\.(gif|jpe?g|png)$"> Order Deny,Allow Deny from all Allow from 122.255.43.3 </Files>
or
<FilesMatch "\.(gif|jpe?g|png)$">
How to increase file upload limit using .htaccess
Sometimes server settings do not allow you to upload files more than a fixed limit (usually 2 MB). But, some websites might require allowing files more than this limit. In such cases, we need to set some options in httpd.conf configuaretion file. This sometimes is not allowed on shared servers. “.htaccess” file comes to the rescue.
Put the below lines in a plain text file. Upload that to your home directory on your web server and rename it as “.htaccess”. This would allow an upload of up to 20 MB file sizes.
php_value upload_max_filesize 20M php_value post_max_size 20M php_value max_execution_time 200 php_value max_input_time 200
