天道酬勤

今天我的虚拟测试环境中的11.2的库自上次关机后今天启动突然报错，记录以下过程备查。

1
2
3
4
5
6
7
8
9

SQL> SELECT * FROM v$version;
 
BANNER
--------------------------------------------------------------------------------
Oracle DATABASE 11g Enterprise Edition RELEASE 11.2.0.2.0 - 64bit Production
PL/SQL RELEASE 11.2.0.2.0 - Production
CORE    11.2.0.2.0      Production
TNS FOR Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production

启动数据库时报错：

1
2
3
4
5
6
7
8
9

SQL> startup
ORACLE instance started.
 
Total SYSTEM Global Area  734892032 bytes
Fixed SIZE                  2230120 bytes
Variable SIZE             448792728 bytes
DATABASE Buffers          276824064 bytes
Redo Buffers                7045120 bytes
ORA-00205: error IN identifying control FILE, CHECK alert LOG FOR more info

根据alert log找到trace文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

[oracle@ORCL112 ~]$ more /u01/app/oracle/diag/rdbms/orcl112/orcl112/trace/orcl112_asmb_6745.trc
Trace FILE /u01/app/oracle/diag/rdbms/orcl112/orcl112/trace/orcl112_asmb_6745.trc
Oracle DATABASE 11g Enterprise Edition RELEASE 11.2.0.2.0 - 64bit Production
WITH the Partitioning, OLAP, Data Mining AND REAL Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_1
SYSTEM name:    Linux
Node name:      ORCL112
RELEASE:        2.6.18-92.el5
Version:        #1 SMP Fri May 23 23:40:43 EDT 2008
Machine:        x86_64
VM name:        VMWare Version: 6
Instance name: orcl112
Redo thread mounted BY this instance: 0 <none>
Oracle process NUMBER: 16
Unix process pid: 6745, image: oracle@ORCL112 (ASMB)
 
*** 2011-11-10 12:42:57.949
*** SESSION ID:(8.1) 2011-11-10 12:42:57.950
*** CLIENT ID:() 2011-11-10 12:42:57.950
*** SERVICE NAME:() 2011-11-10 12:42:57.950
*** MODULE NAME:() 2011-11-10 12:42:57.950
*** ACTION NAME:() 2011-11-10 12:42:57.950
 
WARNING: failed TO START ASMB (connection failed) state=0x1 sid='+ASM' home='/u01/app/grid/product/11.2.0/grid'
ORA-15055: unable TO CONNECT TO ASM instance
ORA-12547: TNS:lost contact

1
2
3
4
5

[grid@ORCL112 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER     STATE_DETAILS   Local Resources
ora.DG01.dg    ONLINE  ONLINE       orcl112                         
ora.asm        ONLINE  ONLINE       orcl112     Started

MOS上有相关文档：[ID 1307075.1]
是由于$GRID_HOME/bin/oracle和$ORACLE_HOME/bin/oracle文件权限不当导致连接ASM实例失败的。
ls -al $GRID_HOME/bin/oracle
-rwxr-x--x 1 grid oinstall 200678464 Feb 28 14:54 oracle
 
ls -al $ORACLE_HOME/bin/oracle
-rwxr-x--x 1 oracle asmadmin 228886191 Feb 28 15:41 oracle 
而在我的环境中只有$GRID_HOME/bin/oracle文件权限发生改变
[grid@ORCL112 bin]$ cd /u01/app/grid/product/11.2.0/grid/bin
[grid@ORCL112 bin]$ ls -l oracle
-rwxrwxr-x 1 grid oinstall 200678464 Jul 15 23:18 oracle

解决方法很简单：
oracle binary的正确权限是：6751
cd $GRID_HOME/bin
chmod 6751 oracle
 
cd $ORACLE_HOME/bin
chmod 6751 oracle
修改之后再检查一下
ls -l oracle
-rwsr-s--x 1 grid oinstall 152400480 Nov 24 15:49 oracle
如果数据库启动仍然失败，重新启动GI

修改之后数据库正常启动
[root@ORCL112 bin]# chmod 6751 oracle
[root@ORCL112 bin]# ls -l oracle
-rwsr-s--x 1 grid oinstall 200678464 Jul 15 23:18 oracle
SQL> startup
ORACLE instance started.
 
Total SYSTEM Global Area  734892032 bytes
Fixed SIZE                  2230120 bytes
Variable SIZE             448792728 bytes
DATABASE Buffers          276824064 bytes
Redo Buffers                7045120 bytes
DATABASE mounted.
DATABASE opened.

-The End-

Multipath options with DS3000/DS4000/DS5000：

AIX MPIO is supported on AIX starting from AIX 5.2 TL10, AIX 5.3 TL6 and AIX 6.1 SP2 though various PTFs and fixes may be required (see interoperability matrix and AIX MPIO documentation below). It is supported with DS4000 firmware 6.60.xx.xx and higher on DS4800, DS4700, DS4200, DS4500 and DS4300. MPIO is the strategic driver for all current DS4000 platforms in AIX 6.1 and up – except for DS4400 which is RDAC only. RDAC is the default driver on all levels of AIX 5.2 and 5.3.

In AIX Version 6.1, the DS4K products are configured as Multipath I/O (MPIO) devices by default. A DS4K product using the FCPARRAY (RDAC) driver may be migrated to the MPIO driver by using the instructions found in the documents below, or all DS4K devices may be migrated by uninstalling the devices.fcp.disk.array.rte package and then running the cfgmgr command or rebooting.

Default driver on AIX6.1：

# oslevel -r
6100-03
# lsdev -Cc disk
hdisk0 Available 08-08-00-5,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 08-08-00-8,0 16 Bit LVD SCSI Disk Drive
hdisk2 Available 05-08-01     MPIO Other DS4K Array Disk
hdisk3 Available 05-08-01     MPIO Other DS4K Array Disk
# lspath
Enabled hdisk0 scsi0
Enabled hdisk1 scsi0
Enabled hdisk2 fscsi0
Enabled hdisk3 fscsi0
Enabled hdisk2 fscsi1
Enabled hdisk3 fscsi1
# mpio_get_config -Av
Frame id 0:
    Storage Subsystem worldwide name: 60ab80011f0e000004d6f113e
    Controller count: 2
    Partition count: 1
    Partition 0:
    Storage Subsystem Name = 'DS4800'
        hdisk      LUN #   Ownership          User Label
        hdisk2         0   B (preferred)      1
        hdisk3         1   A (preferred)      EBS

Default driver on AIX5.3：
BTW:当前主机连接一台DS4800存储

# oslevel -r
5300-10
# manage_disk_drivers  
1: DS4100: currently RDAC; supported: RDAC/fcparray, MPIO
2: DS4300: currently RDAC; supported: RDAC/fcparray, MPIO
3: DS4500: currently RDAC; supported: RDAC/fcparray, MPIO
4: DS4700/DS4200: currently RDAC; supported: RDAC/fcparray, MPIO
5: DS4800: currently RDAC; supported: RDAC/fcparray, MPIO

Change default drive to MPIO from RDAC:

# manage_disk_drivers -c 5
DS4800 currently RDAC/fcparray
Change to alternate driver? [Y/N] Y
DS4800 now managed by MPIO
 
It is necessary to perform a bosboot before rebooting the system in
order to incorporate this change into the boot image.
In order to change to the new driver, either a reboot or a full
unconfigure and reconfigure of all devices of the type changed
must be performed.
# bosboot -a
bosboot: Boot image is 40130 512 byte blocks.
# shutdown -Fr
 
# manage_disk_drivers
1: DS4100: currently RDAC; supported: RDAC/fcparray, MPIO
2: DS4300: currently RDAC; supported: RDAC/fcparray, MPIO
3: DS4500: currently RDAC; supported: RDAC/fcparray, MPIO
4: DS4700/DS4200: currently RDAC; supported: RDAC/fcparray, MPIO
5: DS4800: currently MPIO; supported: RDAC/fcparray, MPIO
# lsdev -Cc disk
hdisk0 Available 08-08-00-5,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 08-08-00-8,0 16 Bit LVD SCSI Disk Drive
hdisk2 Available 05-08-01     MPIO Other DS4K Array Disk
# lspath
Enabled hdisk0 scsi0
Enabled hdisk1 scsi0
Enabled hdisk2 fscsi0
Enabled hdisk2 fscsi1
#  mpio_get_config -Av
Frame id 0:
    Storage Subsystem worldwide name: 60ab80011f0e000004d6f113e
    Controller count: 2
    Partition count: 1
    Partition 0:
    Storage Subsystem Name = 'DS4800'
        hdisk      LUN #   Ownership          User Label
        hdisk2         0   A (preferred)      1

AIX 6.1 TL4的命令与之前的版本有所不同：

#manage_disk_drivers
    Usage :
    manage_disk_drivers [-l]
    manage_disk_drivers -d device -o driver_option
    manage_disk_drivers -h
    Meaning of the Flags:
    -l: Display the Devices, Present Drivers, and Driver Options
    -d: The Device to change the Driver
    -o: Selects the new Driver
    -h: should be help but is interpreted as an illegal option
#manage_disk_drivers -l 查看当前使用的是哪种多路径模式
Device  Present Driver Driver Options
DS4100  AIX_APPCM  AIX_APPCM,AIX_fcparray
DS4300  AIX_APPCM  AIX_APPCM,AIX_fcparray
DS4500  AIX_APPCM  AIX_APPCM,AIX_fcparray
DS4700  AIX_APPCM  AIX_APPCM,AIX_fcparray
DS4800  AIX_APPCM  AIX_APPCM,AIX_fcparray
DS5020  AIX_APPCM  AIX_APPCM
DS5100/DS5300  AIX_APPCM  AIX_APPCM AIX_APPCM
#manage_disk_drivers -d DS4800 -o AIX_fcparray
将DS4800由AIX_APPCM（MPIO）改成AIX_fcparray（RDAC）
#bosboot -a

-The End-

为了能给客户提供更好的服务，一如继往尽可能的在自己的本本上搭建一个环境，做的越多看的越多，胆量却越来越小了，有了环境就多了一个选择，不至于在客户生产系统上去验证自己的推断。

本次测试环境

OEL 5.2 ：VTL（Vistor）2.1.1 + TSM Server 5.5.3 + Client 5.5.3 + Administrator Center（AC）6.2

OEL 4.8：Tsm Client 5.5.3 + Storage Agent 5.5.3+ TDPO 5.5.2 + Oracle 9208

这里下载。

对于TSM Server、Client、Storage Agent、TDPO的安装、配置就不说了，但有一个地方得注意一下，下载的TDPO里不包含license文件（agent.lic），有tdpoerror.log日志文件里会有提示：

05/13/2011 11:26:30 TID<8556> ==> ANU2512E Could not open license file: /opt/tivoli/tsm/client/oracle/bin/agent.lic

可从其它地方拷贝过来测试使用。

lan-free测试：

[oracle@db92 ~]$ cat test.sh

rman target / log=test.log <<EOF

run {

allocate channel ‘dev_0′  type ‘sbt_tape’ parms ‘ENV=(TDPO_OPTFILE=/opt/tivoli/tsm/client/oracle/bin/tdpo.opt)’;

backup format ‘db_%t’ database;

}

EOF

using target database controlfile instead of recovery catalog

allocated channel: dev_0

channel dev_0: sid=17 devtype=SBT_TAPE

channel dev_0: Data Protection for Oracle: version 5.5.2.0

Starting backup at 13-MAY-11

channel dev_0: starting full datafile backupset

channel dev_0: specifying datafile(s) in backupset

including current SPFILE in backupset

including current controlfile in backupset

input datafile fno=00001 name=/u01/app/oracle/oradata/db92/system01.dbf

input datafile fno=00002 name=/u01/app/oracle/oradata/db92/undotbs01.dbf

input datafile fno=00004 name=/u01/app/oracle/oradata/db92/indx01.dbf

input datafile fno=00006 name=/u01/app/oracle/oradata/db92/users01.dbf

input datafile fno=00003 name=/u01/app/oracle/oradata/db92/cwmlite01.dbf

input datafile fno=00005 name=/u01/app/oracle/oradata/db92/tools01.dbf

channel dev_0: starting piece 1 at 13-MAY-11

channel dev_0: finished piece 1 at 13-MAY-11

piece handle=db_751048255 comment=API Version 2.0,MMS Version 5.5.2.0

channel dev_0: backup set complete, elapsed time: 00:01:36

Finished backup at 13-MAY-11

released channel: dev_0

RMAN>

如果备份是走LAN‐FREE ，备份结束后LanFree Data bytes: 应该大于0 。

[oracle@db92 ~]$ dsmc selective “/home/oracle/test.sh”

IBM Tivoli Storage Manager

Command Line Backup/Archive Client Interface

Client Version 5, Release 5, Level 3.0

Client date/time: 05/13/2011 16:43:35

(c) Copyright by IBM Corporation and other(s) 1990, 2010. All Rights Reserved.

Node Name: DB92

Please enter your user id <DB92>:

Please enter password for user id “DB92″:

Session established with server TSM_SVR1: Linux/i386

Server Version 5, Release 5, Level 3.0

Server date/time: 05/13/2011 10:47:01  Last access: 05/13/2011 10:46:42

Selective Backup function invoked.

Normal File–>               191 /home/oracle/test.sh  ** Unsuccessful **

ANS1114I Waiting for mount of offline media.

Retry # 1  Normal File–>               191 /home/oracle/test.sh [Sent]

Selective Backup processing of ‘/home/oracle/test.sh’ finished without failure.

Total number of objects inspected:        1

Total number of objects backed up:        1

Total number of objects updated:          0

Total number of objects rebound:          0

Total number of objects deleted:          0

Total number of objects expired:          0

Total number of objects failed:           0

Total number of bytes transferred:     446  B

LanFree data bytes:                     211  B

Data transfer time:                    0.00 sec

Network data transfer rate:        33,503.60 KB/sec

Aggregate data transfer rate:          0.38 KB/sec

Objects compressed by:                    0%

Elapsed processing time:           00:00:01

如果你有很多TSM Server的可以安装：Integrated Solutions Console (ISC) and Administrative Center，类似IBM的HMC管理小型机，不太适合像我这样的字符控。AC6.2已经集成了ISC，不用再单独安装了，如下命令可以启动AC：

[root@TSMSRV bin]# cd /opt/ibm/ac/profiles/TIPProfile/bin

[root@TSMSRV bin]# ./startServer.sh server1 -username tipadmin -password tipadmin

http://192.168.248.134:16310

 
-The End-

搭建个TSM的学习环境，没想碰到VMware的一个bug了（与CPU设置有关），而这个bug只针对RHEL5.2 X86版本。bug症状是系统启动时一直停在Starting udev，解决方法超级简单，移步这里即可。

-The End-