学步园

http://levipereira.wordpress.com/2011/07/19/monitoring-the-cluster-in-real-time-with-chm-cluster-health-monitor/

Why Cluster Health Monitor ?

Oracle Clusterware & Oracle database performance/node reboot due to lack of CPU/Memory resources cause Customers to ask how to monitor their OS. Some customers have rudimentary scripts that utilize vmstat, mpstat but they are often node collected
at regular intervals. In some cases, we have seen customers collect this once per hour which does not make it very useful when the node is hung/evited via reboot in the middle of the hour. OSwatcher did a wonderful job of making the data collection uniform
with uniform collection intervals. Cluster Health Monitor extends OSwatcher by ensuring it is always scheduled and collects data points while providing a client GUI to view current load.

With this new tool we need to buy a new display to monitor the activities of the cluster in real time.

Why do this? Because it’s cool you have full control of your environment in real time.

In this post I’ll show you how to install and configure the IPD Cluster Monitor.

• 2 Servers Host (linux OEL 5) already with Oracle Clusterware/RAC installed.
• 1 Desktop Client  (my laptop) to monitor the Cluster (using GUI Mode).

Let’s start.

What platforms can I run the Cluster Health Monitor?   Updated 19/07/2011

The
Cluster Health Monitor is NOT available for Itanium platform (Linux, Windows, and HP Itanium) on all version.

11.2.0.1 and earlier: Linux and Windows only (download from OTN)
11.2.0.2: Solaris (Sparc and x86-64) and Linux
11.2.0.3 (to be released): AIX, Solaris (Sparc and x86-64), Linux , and Windows

The Cluster Health Monitor is integrated part of 11.2.0.2 Oracle Grid Infrastructure for Linux (not on Linux Itanium) and Solaris (Sparc 64 and x86-64 only), so installing 11.2.0.2 Oracle Grid Infrastructure on those platforms will automatically
install the Cluster Health Monitor. AIX will have the Cluster Health Monitor starting from 11.2.0.3. The Cluster Health Monitor is also enabled for Windows (except Windows Itanium) in 11.2.0.3.

Installation

For OTN version of Cluster Health Monitor, the complete steps to install the tool is explained in the readme file shipped with the product

For 11.2.0.2 or later version, the cluster health monitor is installed automatically when Grid Infrastructure (aka CRS) is installed.  The resource name for Cluster Health Monitor is ora.crf that is managed by ohasd.

Where can I get latest copy of Cluster Health Monitor?

The Cluster Health Monitor is integrated part of 11.2.0.2 Oracle Grid Infrastructure for Linux (not on Linux Itanium) and Solaris (Sparc 64 and x86-64 only), so installing 11.2.0.2 Oracle Grid Infrastructure on those platforms will automatically
install the Cluster Health Monitor. AIX will have the Cluster Health Monitor starting from 11.2.0.3. The Cluster Health Monitor is also enabled for Windows (except Windows Itanium) in 11.2.0.3.

Prior to 11.2.0.2 on Linux and prior to 11.2.0.3 Windows excluding Itanium platform, the Cluster Health Monitor can be downloaded from OTN.

http://www.oracle.com/technetwork/database/clustering/downloads/index.html

Important:  GUI
Mode (Available only with OTN version) to version 11.2.0.1. GUI Mode is  not avaliable to 11.2.0.2

Online mode can be used to detect problems live on the problem environment. The data can be viewed using Cluster Health Monitor utility. The GUI is not installed on the nodes of the server but can be installed on any other client.

If you are using Oracle Clusterware 11.2.0.2 until today (09/08/2011) the  GUI Mode is not avaliable.

If you are using 11.2.0.1 or previous you must install CHM on Servers and Client.

I’ll show you how to perform a full installation of CHM on servers and client.

Installing CHM on Servers Linux

On Linux, the tool requires Linux kernel version greater than or equal to 2.6.9 and architecture is x86. The install will work on x86_64 as well if the kernel is configured to run 32-bit binaries.

1. In Linux, create user ‘:’ (e.g. crfuser:oinstall) on all the nodes where tool is being installed. Make sure username’s home is the same on all nodes. Typically, on most systems, you will issue:

On all nodes:

while logged in as root.

2. In Linux, setup passwordless ssh for the user created in step 1. Test that the ” can ssh to all nodes (including the local node) using hostname (without domain) without password and without any user intervention like acknowledging prompts.

You can use this post:
http://levipereira.wordpress.com/2010/12/07/configure-ssh-for-user-equivalence/
P.S When prompt : “Enter passphrase (empty for no passphrase):” type [enter] don’t create passphrase.

The CHM have your own database. So, you must specify the location of database of CHM to store OS metrics.
This location MUST be outside of the location where you unzipped the ZIP file because all the directories under that location which were created by unzip will be removed.
BDB files can be kept as it is for later usage. The location should be a path on a volume with at least 2GB per node space available and writable by privileged user only.
It cannot be on root filesystem in Linux. This location is required to be same on all hosts.
The path MUST not be on shared disk. If a shared BDB path is provided to multiple hosts, BDB corruption will happen.

I create a new disk of 5GB on each server and create an new VG on Linux to Store DB of CHM.

4. Login as ‘crfuser’ on Linux.
Unzip the crfpack.zip file.

5. Run crfinst.pl (see below for usage details) script on a node with desired node list, specified as comma separated list, for cluster-wide install. You will find this script in the install subdirectory.

6. Once the step 5 finishes, it will instruct you to run crfinst.pl script with -f and optionally -b on each node while logged in as root/admin to finalize the install on that node.

7. Enable the tool on all nodes. Once the finalize operation is complete, run
the following while logged in as privileged user:

to enable the tool.

DO NOT bypass any of above steps or try other ways to install because the daemons will not work correctly, and you will not be supported.

Finished install CHM on Servers.

Installing GUI Mode on Windows 7 x64

Standalone UI installation. Oracle recommends to not install the UI on the servers. You can use this option to install the UI-only client on a separate machine outside of cluster.

We must have perl installed on the Desktop Client to install the GUI Mode in Windows.

We not need to install perl in Windows. We can use the perl installed on Oracle Client. (if you installed Oracle Client on your destkop.. of course!!!).

As I have installed Oracle Client on my Destop, I will use the perl of installation of Oracle Client.

1. Download CHM from OTN for Windows.

2. Unzip the crfpack.zip file
3. Install CHM GUI Mode on C:\oracle\product\crf

4. Set the environment variable PERL to Oracle Client on file “C:\oracle\product\crf\bin\crfgui.bat”
Change this:

To:

Now you can use your CHM Gui Mode, connect on your cluster.

You can monitor your cluster.

Inside the GUI, you can use ‘node ‘ command to open a view which gives more detailed information about a node in a Node View. Alternatively, you can double click a node to get the Node View.
A Node View presents the detailed statistics on interesting processes, disks and NICs based on heuristics.

Click on Host Holanda…

Click on Host Alemanha..

One can drill down the details on partitions for the disks listed in Node View by double clicking the disk. The information is presented in the Disk View.
The Disk View provides a detailed list of partitions and corresponding stats for each one of them. It also clearly marks partitions which are found to belong to certain categories like Voting/OCR/SWAP/ASM disks.

Both Cluster View and Node View show text alerts at the bottom. These alerts are generated when the sampled value of a resource metric either goes above or falls below a threshold that could lead to potential problems on the node and
hence on the cluster.

You can check problem of  network private (interconnect) on S.O level for example:

LATENCY(ms) and other things are monitored.

The default refresh rate of the GUI is 1 second. To change the refresh rate, use -r with number of seconds (i.e. -r 5 for a 5 second refresh rate)

C:\oracle\product\crf\bin>crfgui.bat -r 5 -m 192.168.217.10

Historical Mode

Invoking the GUI with ‘-d’ option starts it in historical mode.

C:\oracle\product\crf\bin>crfgui.bat -d “hh:mm:ss” -m 192.168.217.10

where -d is used to specify hours (), minutes () and seconds () in the past from the current time to start the GUI from e.g. crfgui -d “05:10:00″ starts the GUI and displays information from the database which is 5 hours and
10 minutes in the past from the current time.

Invoking the GUI with ‘-i’ option provides the same shell at the command prompt as is seen in the GUI windows with a prompt of ‘toprac>’. You can use ‘?’ at this prompt to get detailed information about available commands and options.

Enjoy