For my Oracle 11gR2 RAC Cluster I wanted to test the Grid Naming Service (GNS). Since I used Openfiler 2.3 for my shared storage devices (ASM), I also wanted to use this system as the DHCP and DNS server. A very good document about using Openfiler for ASM devices is written by Jeffrey Hunter and can be found here. This document will describe howto install and configure a DHCP server and DNS server on the Openfiler server.

In this setup my I use three networks:

  1. This network (192.168.123.0) is attached to my home network and has access to the internet. To install packages on Openfiler you need a access to the internet.
  2. This network (192.168.0.0) is uses for the public network. The listeners, GNS VIP, SCAN VIP, Node VIP and nodes public interface are using this network. DHCP can be used in this network segment for the Nodes VIP and SCAN VIP. Names for this VIPs are resolved by the GNS.
  3. This network (10.0.0.0) is used for the Oracle private interconnect and for the shared storage (ISCSI). The IP addresses of the private interfaces of the RAC nodes can be assigned dynamically.

A Grid Naming Service Configuration Example can be found in the Oracle documentation.

Installing packages on an Openfiler server

Openfiler uses a packages system, called conary, developed by rPath. Take a look at the QuickReference to learn more about the parameters that can be used with the conary command. It’s important that your Openfiler server has access to the internet to download the extra packages.

Installing a new package is as easy as using the following command:

# conary update <packagename>

For the DHCP server and DNS server we need the packages dhcp and bind:runtime. Install both packages:

[root@openfiler ~]# conary update dhcp

Applying update job:

Install dhcp(:devel :devellib :doc :lib :runtime)=3.0.7-0.4-1

[root@openfiler ~]# conary update bind:runtime

Including extra troves to resolve dependencies:

bind:lib=9.4.3_P5-1.1-1 info-named:user=1-1-0.1

Applying update job 1 of 2:

Install info-named(:user)=1-1-0.1

Applying update job 2 of 2:

Update  bind(:lib) (9.3.4_P1-0.5-1[ipv6,~!pie,ssl] -> 9.4.3_P5-1.1-1)

Update  bind-utils(:doc :runtime) (9.3.4_P1-0.5-1[ipv6,~!pie,ssl] -> 9.4.3_P5-1.1-1)

Install bind:runtime=9.4.3_P5-1.1-1

To query which files are included in the packages you can use the command:

# conary q <packagename> -lsl

Take a look at the dhcp package:

[root@openfiler ~]# conary q dhcp –ls

/usr/include/dhcpctl.h

/usr/include/isc-dhcp/boolean.h

/usr/include/isc-dhcp/dst.h

/usr/include/isc-dhcp/int.h

/usr/include/isc-dhcp/lang.h

/usr/include/isc-dhcp/list.h

/usr/include/isc-dhcp/result.h

/usr/include/isc-dhcp/types.h

/usr/include/omapip/alloc.h

/usr/include/omapip/buffer.h

/usr/include/omapip/omapip.h

/usr/share/man/man3/dhcpctl.3.gz

/usr/share/man/man3/omapi.3.gz

/usr/share/man/man3/omshell.3.gz

/usr/lib64/libdhcpctl.a

/usr/lib64/libdst.a

/usr/lib64/libomapi.a

/usr/share/doc/dhcp-3.0.7/LICENSE

/usr/share/doc/dhcp-3.0.7/README

/usr/share/doc/dhcp-3.0.7/RELNOTES

/usr/share/doc/dhcp-3.0.7/dhcpd.conf.sample

/usr/share/man/man1/omshell.1.gz

/usr/share/man/man5/dhcp-eval.5.gz

/usr/share/man/man5/dhcpd.conf.5.gz

/usr/share/man/man5/dhcpd.leases.5.gz

/usr/share/man/man8/dhcpd.8.gz

/usr/share/man/man8/dhcrelay.8.gz

/var/lib/dhcp/dhcpd.leases

/etc/init.d/dhcpd

/etc/init.d/dhcrelay

/etc/sysconfig/dhcpd

/etc/sysconfig/dhcrelay

/usr/bin/omshell

/usr/sbin/dhcpd

/usr/sbin/dhcrelay

The installation of the packages is done. Next we need to configure both the servers.


Configuration of the DHCP server

We’re going to configure the DHCP server. Configuration of the DHCP server is done in the file /etc/dhcpd.conf. Since this file is not included in the DHCP package, you need to create it yourself.

Configure DHCP for both the public and the private network. At least the file should contain the following entries:

ddns-update-style none;

allow booting;

allow bootp;

# subnet for the public network

subnet 192.168.0.0 netmask 255.255.255.0 {

option routers                  192.168.0.1;

option subnet-mask              255.255.255.0;

option domain-name              “example.com”;

# domain-name-server is this Openfiler server

option domain-name-servers      192.168.0.250;

range dynamic-bootp 192.168.0.100 192.168.0.150;

default-lease-time 21600;

max-lease-time 43200;

}

# subnet for the private network

subnet 10.0.0.0 netmask 255.255.255.0 {

range dynamic-bootp 10.0.0.100 10.0.0.150;

default-lease-time 21600;

max-lease-time 43200;

}

Check the configuration file with the command service dhcpd configtest. This only checks for syntax errors.

[root@openfiler ~]# service dhcpd configtest

Internet Systems Consortium DHCP Server V3.0.7

Copyright 2004-2008 Internet Systems Consortium.

All rights reserved.

For info, please visit http://www.isc.org/sw/dhcp/

To start the DHCP server use the following command:

[root@openfiler ~]# service dhcpd start

Starting dhcpd: Internet Systems Consortium DHCP Server V3.0.7

Copyright 2004-2008 Internet Systems Consortium.

All rights reserved.

For info, please visit http://www.isc.org/sw/dhcp/

Wrote 0 leases to leases file.

Listening on LPF/eth2/00:0c:29:94:29:56/10.0.0/24

Sending on   LPF/eth2/00:0c:29:94:29:56/10.0.0/24

Listening on LPF/eth1/00:0c:29:94:29:4c/192.168.0/24

Sending on   LPF/eth1/00:0c:29:94:29:4c/192.168.0/24

No subnet declaration for eth0 (192.168.123.240).

** Ignoring requests on eth0.  If this is not what

you want, please write a subnet declaration

in your dhcpd.conf file for the network segment

to which interface eth0 is attached. **

Sending on   Socket/fallback/fallback-net

[  OK  ]

You can ignore the warning about the subnet declaration for eth0, since we did not configure DHCP for this network segment.

To make sure the DHCP server is started at boottime use the chkconfig command:

[root@openfiler ~]# chkconfig dhcpd on

[root@openfiler ~]# chkconfig dhcpd –list

dhcpd           0:off   1:off   2:on    3:on    4:on    5:on    6:off

The configuration of the DHCP server is done.

Configuration of the DNS server

Configuration of the DNS server is a bit more complicated than the DHCP server. The following files must be created:

/etc/named.conf

/srv/named/data/example.com.zone

/srv/named/data/0.168.192.in-addr.arpa.zone

The first one contains the main configuration file. The last two contain the information about the zones.

Create the file /etc/named.conf with at least the following content:

options {

directory “/srv/named/data”;

};

# forward zone

zone “example.com” IN {

type master;

file “example.com.zone”;

};

# reverse zone

zone “0.168.192.in-addr.arpa” IN {

type master;

file “0.168.192.in-addr.arpa.zone”;

};


The file /srv/named/data/example.com.zone should contain the following:

$ORIGIN .

$TTL 86400      ; 1 day

example.com     IN SOA  openfiler.example.com. foo.bar.tld. (

200612069  ; serial

7200       ; refresh (2 hours)

300        ; retry (5 minutes)

604800     ; expire (1 week)

60         ; minimum (1 minute)

)

NS      openfiler.example.com.

$ORIGIN example.com.

openfiler                       A       192.168.0.250

rac01                           A       192.168.0.101

rac02                           A       192.168.0.102

; delegation records for the GNS of the rac cluster

$ORIGIN cluster.example.com.

@                               IN              NS      gns.cluster.example.com.

gns.cluster.example.com.        IN              A       192.168.0.200

Important in the above configuration file are the delegation entries for the subdomain cluster.example.com.

The file /srv/named/data/0.168.192.in-addr.arpa.zone should contain:

$ORIGIN .

$TTL 86400      ; 1 day

0.168.192.in-addr.arpa IN SOA openfiler.example.com. foo.bar.tld. (

200612066  ; serial

7200       ; refresh (2 hours)

300        ; retry (5 minutes)

604800     ; expire (1 week)

60         ; minimum (1 minute)

)

NS      openfiler.example.com.

$ORIGIN 0.168.192.in-addr.arpa.

250                     PTR     openfiler.example.com.

101                     PTR     rac01.example.com.

102                     PTR     rac02.example.com.

Start the nameserver with the service command:

[root@openfiler ~]# service named start

Starting named:                                            [  OK  ]

Make sure that the nameserver is started at boottime:

[root@openfiler ~]# chkconfig named on

[root@openfiler ~]# chkconfig named –list

named           0:off   1:off   2:on    3:on    4:on    5:on    6:off


Configuration of RAC nodes

To make sure that all the hostnames can be resolved the file /etc/resolv.conf must be edited. Create this file with the following entries:

search example.com cluster.example.com

nameserver 192.168.0.250

The first line makes sure that you don’t have to specify the fully qualified host name, if you want to resolve a name in your local network.

To make sure that the file /etc/resolv.conf doesn’t get overwritten by the dhcp configuration of the private interface you have to create the file /etc/dhclient-enter-hooks with the following content:

make_resolv_conf() {

echo “doing nothing to resolv.conf”

}

More about this can be found here.


Installation of the Grid Infrastructure

When installing the Oracle 11g Release 2 Grid Infrastructure, make sure you choose the Advanced Installation as your installation type. In the screen Grid Plug and Play Information, you need to supply the GNS subdomain name and the GNS VIP Address as specified in the DNS configuration.

Just follow the normal installation procedure.

When the installation is finished you can test your new setup with the dig command:

[root@rac01 logs]# dig rac-scan.cluster.example.com

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 <<>> rac-scan.cluster.example.com

;; global options:  printcmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63099

;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:

;rac-scan.cluster.example.com.  IN      A

;; ANSWER SECTION:

rac-scan.cluster.example.com. 120 IN    A       192.168.0.147

rac-scan.cluster.example.com. 120 IN    A       192.168.0.146

rac-scan.cluster.example.com. 120 IN    A       192.168.0.148

;; AUTHORITY SECTION:

cluster.example.com.    86400   IN      NS      gns.cluster.example.com.

;; Query time: 11 msec

;; SERVER: 192.168.0.250#53(192.168.0.250)

;; WHEN: Mon Apr 26 18:11:20 2010

;; MSG SIZE  rcvd: 160

As shown in the output above, the scan address is resolved to 3 different ip-addresses.

There is a bug (9018141) about GNS not removing old SCAN ip-addresses after a DHCP lease expires. On My Oracle Support there’s also a workaround for this bug.