Linux, Programming, Telephony, Asterisk, Apache, Tips & Tricks » Apache

URL Redirecting (301,302) in Varnish

Shahid — Wed, 22 Oct 2008 17:37:04 +0000

There is no straight forward method for 301/302 URL redirection as we did with rewrite rule. But there are some situations where we need to redirect a URL on a host to some URL on different host. In this situation we need to use 301/302 redirection. There are two ways we can implement this.

First method is to use set req.http.host and set req.url in the vcl.conf file. Using these two methods we perform normal redirection. Using these method we will redirect the url to some other URL on the same host. For example say the URL http://example.com/error500.php will be redirect to the URL http://example.com/redirect.php. We specify 301/302 redirection to the URL http://foo.com/index.php in the redirect.php file using header method.

In the sub vcl_recv method use the following code.

sub vcl_recv {
   if (req.http.host ~ "example.com") {
      if (req.url ~ "error500.php") {
         set req.http.host = "example.com";
         set req.url = "/redirect.php";
      }
   }
}

In the redirect.php file use the following method.

Second method is to use vcl_error error method. In this error method we can specify HTML tags, so using meta tags we can implement redirection. This option is available with the latest version of varnish.

In the sub vcl_recv method use the following code.

sub vcl_recv {
   if (req.http.host ~ "example.com") {
      if (req.url ~ "error500.php")
         error;
      }
   }
}
sub vcl_error {
        set obj.http.Content-Type = "text/html; charset=utf-8";
        if (req.http.host ~ "example.com") {
                if (req.url ~ "error500.php") {
                        synthetic {"
                                
                                
                                
                                        
                                                
                                        
                                
                        "};
                }
        }
        deliver;
}

Logrotate in linux

Shahid — Tue, 01 Jul 2008 06:17:18 +0000

Logrotate

Log files are important for Linux system security and trouble shooting. Log files tend to grow, and if a log grows too much you compress it and takes the backup. Logrotate enables you to do this in a much better way. The process that is in charge of compressing and rotating these logfiles is called logrotate and it is executed once per day upon Debian installations.

Here you will find the logrotate driver script. Every day this script runs and examines two things:

The configuration file /etc/logrotate.conf
The configuration directory /etc/logrotate.d/

A typical logrotate configuration file looks like this:

/var/log/apache/*.log {
        weekly
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                if [ -f /var/run/apache.pid ]; then
                        /etc/init.d/apache restart > /dev/null
                fi
        endscript
}

If you want to add new log rotate for any other file you can new logrotate configuration file to /etc/logrotate.d

Assuming we have a new service “mon” which produces its output in /var/log/mon/mon.log we can cause this to be rotated every day with a script like this:

/var/log/mon/*.log {
  daily
  missingok
  rotate 7
  compress
  delaycompress
  create 640 mon mon
  sharedscripts
     /etc/init.d/mon restart
  endscript
}

For more help man logrotate

Apache Load Balance Using Haproxy

Shahid — Sun, 29 Jun 2008 17:25:32 +0000

Haproxy

HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for web sites crawling under very high loads while needing persistence or Layer7 processing. Supporting tens of thousands of connections is clearly realistic with todays hardware. Its mode of operation makes its integration into existing architectures very easy and riskless, while still offering the possibility not to expose fragile web servers to the Net.

1. Preperation

For configuring haproxy you need the following setups. Here I used ACL.

1.1. Using ACL

The use of Access Control Lists (ACL) provides a flexible solution to perform content switching and generally to take decisions based on content extracted from the request, the response or any environmental status. The principle is simple

1.2. Load Balancer

Hostname: lb.example.com

IP: 192.168.0.110

1.3 Web Server 1

Hostname: http1.example.com

IP: 192.168.0.111

1.4. Web Server 2

Hostname: http2.example.com

IP: 192.168.0.112

1.5. First download and install haproxy

To get latest version click here

$ wget  http://haproxy.1wt.eu/download/1.2/src/haproxy-1.2.18.tar.gz
$ tar -xzvf haproxy-1.2.18.tar.gz
$ cd  haproxy-1.2.18
$ sudo make  TARGET=linux24

1.6. Configuring Load Balancer System IP : 192.168.0.110

set ENABLED to 1 in /etc/default/haproxy

 $ sudo vim /etc/default/haproxy

# Set ENABLED to 1 if you want the init script to start haproxy.

ENABLED=1

# Add extra flags here.
#EXTRAOPTS="-de -m 16"

We back up the original /etc/haproxy.cfg and create a new one like this

cp /etc/haproxy.cfg /etc/haproxy.cfg_orig
cat /dev/null > /etc/haproxy.cfg
vi /etc/haproxy.cfg

Sample haproxy configuration file /etc/haproxy.cfg which uses ACL. In the below example if you use the domain example.com haproxy always uses web server 1,if you use loadbalancer.com

global
global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
        #log loghost    local0 info
        pidfile /var/run/haproxy.pid
        daemon
#       debug           # If you want to start haproxy in debugging mode uncomment this line and comment the above line.
        nbproc      4   # Number of processing cores. Dual Dual-core Opteron i    s 4 cores for example.
defaults
        log     global
        mode    http
        option  httplog
        option  httpchk
        option  dontlognull
        retries 3
        option          redispatch
        maxconn 2000
        contimeout      50000
        clitimeout      500000
        srvtimeout      500000

backend back_std
        balance roundrobin
        option redispatch
        cookie JSESSIONID prefix
        option httpclose
        option forwardfor
        option httpchk HEAD /check.txt HTTP/1.0
        server ws4 192.168.0.111:82 weight 1 check
#      stats uri /status   # Custom status URI
        stats auth username:password
        stats enable
        stats refresh 10

backend back_lb
        balance roundrobin
        option redispatch
        cookie JSESSIONID prefix nocache indirect
        option httpchk HEAD /check.txt HTTP/1.0
        server ws1 192.168.0.111:81 cookie ws1 weight 1 check
        server ws2 192.168.0.112:81 cookie ws2 weight 1 check
        option httpclose
        option forwardfor
        stats enable
        stats refresh 10

frontend http-in
        bind :80
        acl hosts_std hdr_end(host) -i example1..com          # Which is not load balanced
        acl hosts_std hdr_end(host) -i example1.com:80

        acl hosts_lb hdr_end(host) -i loadbalancer.com      # Which is load balanced
        acl hosts_lb hdr_end(host) -i loadbalancer.com:80

        use_backend back_std if hosts_std       # If requests comes to the domain which is in host_std then that URL will use back_std
        use_backend back_lb if hosts_lb          # If requests comes to the domain which is in host_lb then that URL will use back_lb

To allow HAProxy to bind to the shared IP address, we add the following line to /etc/sysctl.conf:

$ sudo vim /etc/sysctl.conf

[...]
net.ipv4.ip_nonlocal_bind=1
[...]

and run

$ sudo sysctl -p

1.7 Start haproxy

$ sudo /etc/init.d/haproxy start

1.8. Configuring Web Server 1 & Web Server2

We will configure HAProxy as a transparent proxy, i.e., it will pass on the original user’s IP address in a field called X-Forwarded-For to the backend web servers. Of course, the backend web servers should log the original user’s IP address in their access logs instead of the IP addresses of our load balancers. Therefore we must modify the LogFormat line in /etc/apache2/apache2.conf and replace %h with %{X-Forwarded-For}i:

 $ sudo vim /etc/apache2/apache2.conf

#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
#LogFormat "%h %l %u %t \"%r\" %>s %b" common
#LogFormat "%{Referer}i -> %U" referer
#LogFormat "%{User-agent}i" agentLogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

Also, we will configure HAProxy to check the backend servers health by continuously requesting the file check.txt (translates to /var/www/check.txt if /var/www is your document root) from the backend servers. Of course, these requests would totally bloat the access logs and mess up your page view statistics (if you use a tool like Webalizer or AWstats that generates statistics based on the access logs).

Therefore we open our virtualhost configuration (in this example it’s in /etc/apache2/sites-available/default) and put these two lines into it (comment out all other CustomLog directives in your vhost configuration):

$ sudo touch /var/www/check.txt
$ sudo vi /etc/apache2/sites-available/default

[...]
SetEnvIf Request_URI "^/check\.txt$" dontlog
CustomLog /var/log/apache2/access.log combined env=!dontlog
[...]

Adds the ports to /etc/apache2/ports.conf file.

Listen 81
Listen 82

    Listen 443

Afterwards we restart Apache:

$ sudo /etc/init.d/apache2 restart

Now to check the status of haproxy use the custom URI or use http://example1.com/haproxy?stats

I hope it helped, a comment with your questions are welcome

Monitoring services using Mon

Shahid — Sun, 29 Jun 2008 12:09:27 +0000

Monitoring services using Mon

mon is a general-purpose scheduler and alert management tool used for monitoring service availability and
triggering alerts upon failure detection.

Mon uses some Linux command and services such as ping command to check host connectivity and mail servers for sending mail, so you must install postfix and other packages so that mon should work smoothly.

Installing and configuring Mon in Ubuntu

Now we will see how we can install and configure mon in Ubuntu.

Install process

$ sudo apt-get install mon

Start mon

$ sudo /etc/init.d/mon start

Configuring Mon

For configuring mon edit the configuration file /etc/mon/mon.cf

hostgroup servers 192.168.1.6  #There must be a one line gape after hostgroup line.

watch servers
    service ping
        interval 1m
        monitor fping.monitor
        period wd {Mon-Fri} hr {7am-10pm}
            alert mail.alert user@example.com
            alertevery 3m
        period wd {Sat-Sun}
            alert mail.alert user@example.com

First we define the hosts that we want to monitor. Then we create the a “watch” section for this hosts with only one service to monitor, the “ping” service. The service will be monitored using the monitor “fping.monitor” that you can find in “/usr/lib/mon/mon.d/fping.monitor”. Then we define the periods in which we want to monitor the hosts and the alert we will use if the monitor detects a fail. In this case we use the “mail.alert” agent that you can find in “/usr/lib/mon/alert.d/mail.alert”. An email to “user@example.com” will be sent if a fail is detected in the monitor.

Sample example for configuring mon for monitoring a host connectivity, mysql, apache and a web page.

hostgroup ping_services 192.168.1.6

hostgroup webservers itswork.com

hostgroup mysql_server localhost

hostgroup apache_service 192.168.1.6

watch ping_services
service ping
    interval 1m
    monitor fping.monitor
    period wday {Mon-Sun} hour {0-23}
        alert mail.alert user@example.com
        alertevery 1m

watch webservers example.com
service site
        interval 1m
        monitor http.monitor -p 80 -u /index.php
        period wday{Mon-Sun} hr {0am-24pm}
            alert mail.alert user@example.com

watch apache_service 192.168.1.6
service apache
        interval 1m
        monitor http.monitor
        period wday{Mon-Sun} hr {0am-24pm}
            alert mail.alert user@example.com

watch mysql_server
    service mysql
        interval 1m
        monitor mysql.monitor --username=mon --password=mon --database=mon localhost
        period wday {Mon-Sun}
            alert mail.alert -S "Host 1 Mysql Server is down!!" user@example.com
            upalert mail.alert -S "Host1 MYSQL server is back up" user@example.com
            alertevery 60s
            alertafter 3

For more help click here

Monit a service monitoring tool in linux

Shahid — Sun, 29 Jun 2008 10:30:58 +0000

Monit is a utility for managing and monitoring processes, files, directories and devices on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. E.g. monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You may use monit to monitor files, directories and devices for changes, such as timestamps changes, checksum changes or size changes. Monit will monitor a service and in case of any problems monit will will raise an email alert.

Install the package monit using apt on Ubuntu.

$ sudo apt-get install monit

First edit the file /etc/default/monit and change startup=0 to startup=1. The file should look like below text.

# Defaults for monit initscript
# sourced by /etc/init.d/monit
# installed at /etc/default/monit by maintainer scripts
# Fredrik Steen 
# You must set this variable to for monit to start

startup=1

# To change the intervals which monit should run uncomment# and change this variable.
# CHECK_INTERVALS=180

Edit the monit configuration file /etc/monit/monitrc and uncomment the following lines

$ sudo vim /etc/monit/monitrc

Start monit in background (run as daemon) and check the services at 2-minute intervals.

set daemon  120

You can set the alert recipients here, which will receive the alert for each service. The event alerts may be restricted using the list.

set alert abc@xyz.com                                   # receive all alerts
set alert hij@xyz.com only on { timeout }       # receive just service-

Monit has an embedded webserver, which can be used to view the configuration, actual services parameters or manage the services using the web interface.

set httpd port 2812 and
     use address localhost  # only accept connection from localhost
     allow localhost        # allow localhost to connect to the server and
     allow admin:monit      # require user 'admin' with password 'monit'

Next session in the configuration file is where the monit monitors the services, for that we need to specify the service which needs to be monitored. A sample example for monitoring apache is as follows.

check process apache with pidfile /var/run/apache2.pid
    start program = "/etc/init.d/apache2 start"
    stop program  = "/etc/init.d/apache2 stop"
    if cpu > 60% for 2 cycles then alert
    if cpu > 80% for 5 cycles then restart
    if totalmem > 200.0 MB for 5 cycles then restart
    if children > 250 then restart
    if loadavg(5min) greater than 10 for 8 cycles then stop
    if 3 restarts within 5 cycles then timeout
    group server

For more examples click here
Next we will see some monit commonds.

To satrt monit

$ sudo /etc/init.d/monit start

To start the services which the monit is monitorin, sudo monit start all will start all the services in the monit configuration file, if specify a name insted of start all monit will start only the specified service.

$ sudo monit start all
$ sudo monit start apache

To get the status of monit services

$ sudo monit status

To get the summary of mont services

$ sudo monit summary

To stop all the services monitored my monit use stop all, or specify the name of the servive you need to stop

$ sudo monit  stop all
$ sudo monit stop apache

For more help click here

Installing Symfony Framework in Ubuntu

Shahid — Mon, 23 Jun 2008 16:24:27 +0000

Install these packages with apt:

php5
php5-cli
php5-dev
php5-sqlite
php-pear

$ sudo apt-get install php5 php5-cli php5-dev php5-sqlite php-pear

Test if Apache and PHP are up and running typing in the browser:

http://localhost/

You should see something like:

Apache/2.0.55 (Ubuntu) PHP/5.1.2-1ubuntu1 Server at localhost Port 80

From the command line, as root:

$ sudo pear upgrade PEAR
$ sudo pear channel-discover pear.symfony-project.com

Edit /etc/php5/cli/php.ini and change the line:

memory_limit = 8M To memory_limit = 16M

Then

$ sudo pear install symfony/symfony
$ sudo pear install –alldeps http://phing.info/pear/phing-current.tgz

Using the command:

$ sudo pear config-show

we find the symfony libraries have been installed in:

$ php_dir/symfony/ /usr/share/php/symfony/ main libraries
$ data_dir/symfony/ /usr/share/php/data/symfony/ skeleton of symfony applications, default modules and configuration
$ doc_dir/symfony/ /usr/share/php/docs/symfony/ documentation

From the command line enter bellow command:

$ symfony -Vsymfony version 0.6.2

From the command line, as root:

$ mkdir /var/www/myproject
$cd /var/www/myproject
$ symfony init-project myproject
$ symfony init-app myapp

Edit /Virtual Host configuration file , and append the following at the end:


ServerAdmin webmaster@localhost
ServerName servername.com
ServerAlias www.servername.com
DocumentRoot /var/www/project/web
Alias /sf /usr/share/php/data/symfony/web/sf

     Options FollowSymLinks MultiViews
     AllowOverride all
     Order allow,deny
     allow from all