Clicky Web Analytics


Posted By Ritesh Chhajer on Friday, July 30, 2010
14713 Views 15 Comments

For last few months, I've been busy like hell dealing with umpteen number of Oracle 11gR2 RAC installations and upgrades and it's been a roller coaster ride discovering a new feature or bug every day. So I reckon it's high time to consolidate and put together a summarized checklist for fellow DBAs. Obviously, I won't be able to cover all the 11gR2 features but nevertheless would like to share my learnings and discoveries.

First things first.
The Basics:
  • The clusterware is now known as Grid Infrastructure.
  • CRS and ASM binaries reside under a single Oracle Home known as GRID Home.
  • GRID Home and the Oracle Home must be installed in different locations.
  • Oracle Grid Infrastructure cannot be placed under Oracle Base since permissions will be changed to root.
  • OCR and Voting File can no longer be placed on RAW or block devices. They need to be placed either in ASM or on Shared NFS Storage.
  • Voting file does not need to be backed up using dd command and is not supported anymore.The contents of Voting file are backed into the OCR.
For more details on the architecture and understanding of background processes, refer Metalink Note: 1053147.1

Single Client Access Name(SCAN) as the name suggests is a single name for clients to access RAC database. The obvious benefit is as nodes get added to the cluster, client TNS remains unchanged. It's like a virtual load balancer. You can find more details on Metalink Note:887522.1

Always remember the following about SCAN during your RAC installation:
  • SCAN name can be maximum up to 15 characters excluding the domain name. Keep this in mind while requesting for scan name and IPs otherwise you won't be able to proceed with your installation since SCAN is a mandatory requirement.
  • Ask your clients to use 11gR2 version of client coz with older version(11gR1/10gR2), if one of the SCAN listener goes down, the connection will fail. Only 11gR2 client will give full functionality of SCAN. You can have workarounds though by explicitly specifying all the 3 SCAN IPs in the client TNS.
  • Set REMOTE_LISTENER to SCAN. DO NOT set to TNSNAMES alias. Example: remote_listener='scan-clu:1521'. Default installation will not update remote_listener and it's imperative to set it correctly if you wish your clients to use single name in the TNS.
  • LOCAL_LISTENER should be set to node VIP as FQDN(Fully Qualified Domain Name). Default install would put the VIP but not as fully qualified. FQDN is important otherwise you may notice intermittent connection failures.
  • Best way to troubleshoot SCAN is to set ORACLE_HOME to GRID_HOME since SCAN listener runs from Grid Home and check for 'lsnrctl servics SCAN_LISTENER' to verify if all instances are registered correctly.

Now, let us look at some of the cool features that 11gR2 brings on the table.
  • ASMCMD now has the 'cp' command to copy files from non ASM file system to ASM file system.
  • ADRCI - Command line tool to view alert log and other diagnostic files.DIAGNOSTIC_DEST replaces all old udump,bdump,cdump parameters. All trace information is now in $ORACLE_BASE/diag
  • Multiple AU sized diskgroups can now be created.
  • ASMCA provides a GUI tool to manage diskgroups. I found this one pretty neat though I'm not a great fan of GUI and prefer to do most of the stuff on the prompt.
  • The OUI now provides option to update minimum required kernel parameters.
  • For ASM, there is now 'md_backup' and 'md_restore' to backup and restore metadata in case of disk corruption.
  • For ASM, remember to set COMPATIBLE.ASM and COMPATIBLE.RDBMS diskgroup attributes to to utilize bunch of new features.
  • DEFERRED_SEGMENT_CREATION is set to TRUE which essentially saves space and means that initial extent won't be allocated till 1st row is inserted.
  • Partitioning enhancements include interval, system and reference partitioning along with virtual column based partitioning.
  • Lots of new parameters introduced with reference to parallelism. For example, PARALLEL_FORCE_LOCAL when set to true would restrict the slaves to run on the same node where QC is connected.PARALLEL_DEGREE_POLICYwhen set to AUTO would determine automatic degree of parallelism.PARALLEL_IO_CAP_ENABLED is deprecated and remapped to PARALLEL_DEGREE_LIMIT set to IO. PARALLEL_THREADS_PER_CPU is now underscore parameter.With 11gR2, parallel hints work at the statement level.
  • For upgrading 10g databases to 11g, RAT(real application testing) provides database replay and SPA(SQL Performance Analyzer) which are quite useful to predict and compare post upgrade execution plans and database performance. Shall post more on that in my next blog.
Now, the other side of the coin where you need to be aware that these are features and not bugs:)
Features Contd.:
  • Schema passwords are case sensitive by default. This is purportedly advanced security but poses risks for client connections since everyone is used to the idea of having Oracle password being case insensitive and it's just not feasible to ask developers to change their code. Best is to turn it off immediately post installation before creating the schemas. 'sec_case_senstive_logon=false'
  • GSD is disabled by deafult as it's not required. So immediately post installation, when you check for 'crsctl status resource -t', don't get baffled on why it's disabled.
  • When an instance is manually shutdown, service won't fail over to the available node as it used to do in 10g. If the instance crashes, service fail over happens as expected. This I reckon is a major fundamental change to the CRS stack and it'll take a while to get used to this change. If you really want to the service to fail over when an instance is manually shutdown then only way for you is to automate using FAN callouts.
  • When an attempt is made to start service on a stopped instance, CRS (Oops! GRID) will first start the instance and then the service. This is because the entire stack is tightly coupled.
  • SMTP ports are blocked by default. ORA-24247: network access denied by access control list (ACL) is raised to signal this. All UTL_TCP/UTL_HTTP/UTL_SMTP need XML DB to be installed. Refer Metalink Note:742014.1. So if mail functionality is being used in PL/SQL, beware of this. Also you need to explicitly grant ACL(AccessControl List) privileges to the schema using DBMS_NETWORK_ACL_ADMIN.
As always, there are bugs with any Oracle release. Here are the ones which I came across:
  • Hugepages not being used. First check if /etc/security/limits.conf is set correctly for 'memlock'. You can also check 'ulimit -l'. However there is a bug with where if database is started by srvctl and not sqlplus then hugepages are not used. The workaround is to edit either $GRID_HOME/bin/ohasd or /etc/init.d/ohasd and put 'ulimit -l unlimited' explicitly. This bug is supposedly fixed in
  • If loopback database link is used, the SQL would hang forever waiting on 'DFS Lock Wait'. Till date the issue persists and there doesn't seem to be fix from Oracle. Unofficially though I know one of the underscore parameter fixes the issue but would essentially disable majority of cluster features.
Whatever I've discussed aforementioned is just tip of the iceberg. Only time will tell what's in store. More bugs or more features!

Rants & Raves Minimize

  • Gravatar
    Aroop Friday, July 30, 2010 at 11:59 PM

    Most of the basic stuff is available on Metalink but services thing is quite a revelation to me. I got to test it out soon. Nice post though. Keep it up dude.

    • Gravatar
      Ritzy Monday, August 02, 2010 at 11:23 PM
      Yeah - Service behavior was very surprising and it seemed to be major fundamental change with 11gR2 to me. I initially played around with all the 'n' number of options which now comes with 11gR2 services and it took a while to realize it's a feature and not a bughappy

  • Gravatar
    Pete Tuesday, August 10, 2010 at 7:21 PM

    I'm using database with ASM but not able to use IDP. Any idea?

    • Gravatar
      Ritzy Thursday, August 12, 2010 at 5:33 AM

      Thanks for your response.
      With IDP, you can place frequently accessed data(HOT) on outermost tracks and reduce the latency
      For intelligent data placement to work, COMPATIBLE.ASM and COMPATIBLE.RDBMS diskgroup attributes must be set to
      You can verify using v$asm_diskgroup and change using asmca(GUI) or through SQL(Login as sysasm):

      SQL> alter diskgroup data set attribute 'compatible.asm'='';
      SQL> alter diskgroup data set attribute 'compatible.rdbms'='';

      SQL> select name,compatibility,database_compatibility from v$asm_diskgroup;

      Query following dictionary views to check if IDP is being used effectively:

      Hope it helps

  • Gravatar
    Jeremy Thursday, August 12, 2010 at 8:03 AM

    I'm facing issues with hugepages. It was not being used and I followed your steps but now when I try to start the database, it's giving me errors.

    ORA-27102: out of memory
    Linux-x86_64 Error: 28: No space left on device

    If I disable hugepages, DB is starting though.
    I verified SGA is well within hugepages.

    Please respond

    • Gravatar
      Ritzy Thursday, August 12, 2010 at 8:19 AM
      Check your shmmax. sysctl -a|grep shmmax. The error what you got suggests that hugepages cannot fit into shmmax. Increase shmmax and make sure SGA can fit in that otherwise DB may still start but you would see fragmented shared memory segments(ipcs -m).

      In short, your SGA should fit in hugepages which should fit in shmmax

      • Gravatar
        Jeremy Thursday, August 12, 2010 at 8:36 AM

        THANK YOU Sir so much!
        My problem is fixed nowhappy

      • Gravatar
        Mary Tuesday, November 09, 2010 at 5:17 PM
        I'm also not able to use hugepages. /proc/meminfo shows that hugepages are not in use. However I'm able to start my database. I verified all parameters in sysctl.conf and limits.conf are good. My database version is and OS is Linux 5

        • Gravatar
          Ritesh Chhajer Tuesday, November 09, 2010 at 5:24 PM
          AMM and hugepages
          By any chance, are you using automatic memory management? Please note that AMM is not compatible with hugepages on Linux. Please check if you have set memory_target to a value > 0. If AMM is enabled, you would see in memory files under /dev/shm from where SGA would get allocated. If you want to use hugepages, then you need to disable AMM.

          If the issue is not related to AMM, double check the memlock value in /etc/security/limits.conf.

          • Gravatar
            Mary Tuesday, November 09, 2010 at 5:37 PM

            Thank you so much for your quick response. The issue was indeed related to AMM. For last few hours, I have been spending all the time troubleshooting sysctl.conf and limits.conf parameters. You saved my day.

  • Gravatar
    Ritzy Thursday, September 02, 2010 at 7:40 PM
    Case sensitive password file
    While setting up physical standby database on 11gR2, primary was not able to ship the logs to standby
    Querying v$archive_dest for dest_id=2 returns following error:
    ORA-16191: Primary log shipping client not logged on standby

    The error persists despite the following facts:
    1. Both Primary and Standby are able to connect to each other using "sys/passw@connectstring as sysdba"
    2. remote_login_passwordfile is set to EXCLUSIVE
    3. log_archive_dest_2, fal_server, fal_client are all set correctly and no issues with tnsnames.ora
    4. Password file recreated on both primary and standby and bounced the databases.
    5. remote_listener,local_listener,service_names are all set correctly.

    Or further troubleshooting, the trace file shows:
    .. Detailed OCI error val is 1031 and errmsg is 'ORA-01031: insufficient privileges

    This suggests there is some issue with password file
    SQLPLUS connections are working fine but primary is not able to ship logs.

    Finally, figured out resolution is to re-create password file using orapwd with "ignorecase=y" option.

    • Gravatar
      john Wednesday, November 16, 2011 at 1:05 AM


      This was my exam question last week and still not sure about the answer.
      Any help is much appreciated.

      You have a four node cluster with four node VIPS and three scan VIPS already in use after Grid infrastructure installation
      You plan to manage a customer-developed, web based application with the Oracle Grid Infrastructure to provide high availability.
      Which two statements are true about the vip application resource that must be created ?

      a-)The vip application must be active on the same subnet and ethernet adaptor as the interconnect
      b-)The vip application must be active on the same subnet and ethernet adaptor as the public ip address
      c-)The vip application must not be online on the same node as any of the scan vips.
      d-)The vip application must be active on a different node than the application depending on Che
      by using the dispersion start dependecy in the resource definition
      e-) The vip application must be enabled to run as the root user

      • Gravatar
        Ritzy Monday, November 21, 2011 at 1:15 PM

        I would say b and e

  • Gravatar
    Gerardo Thursday, November 01, 2012 at 3:11 PM

    Hi Ritzy,

    Thanks a lot for your time to write this blog, is very helpful for people like me that is on the way of Oracle DBA.

    I have a problem with the SCAN, I am testing an application that reports errors to login in the DB, I mean, if I point the configuration direct to the IP SCAN the application works very very slow, in other hand if a point direct to the nodes, the application works very fast, Do you have any idea about this problem?.
    Thanks in advance for your help.


    • Gravatar
      Ritzy Friday, November 02, 2012 at 11:05 AM

      That looks weird. Please clarify what error is reported. Either SCAN has to work on not work. It cannot be slow. What's the client version and how are listener parameters set?

  • Wednesday, November 17, 2010 at 2:14 PM
    via pingback
    Getting started with - New features|Oracle Talk

  • Tuesday, November 09, 2010 at 4:46 PM
    via pingback
    Getting started with - The Upgrade|Oracle Talk

  • Recommended Oracle DBA Books Minimize


    Tag Cloud Minimize

    Archive Posts Minimize

      This posting is provided "AS IS" with no warranties, and confers no rights. You assume all risk for your use.

      This posting has nothing to do with my present or past employer.