1
Introducing Oracle Parallel Server

This chapter provides a conceptual and component overview of Oracle Parallel Server. This information helps you prepare and plan your Oracle Parallel Server installation and configuration in an Oracle8i environment.

Specific topics discussed are:

• Introduction

• Oracle Parallel Server and the Optimal Flexible Architecture

• Oracle Parallel Server Software Components

• Oracle Parallel Server Instances

• Oracle Parallel Server Features

Introduction

Oracle Parallel Server is an architecture that allows multiple instances to access a shared database. Oracle Parallel Server offers the following (terms will be described later in this chapter):

• Superior scalability through distribution of workload across nodes

• High-availability through multiple nodes accessing the database; if one node fails, the database is still accessible through surviving nodes

An Oracle Parallel Server can handle node or disk failure in a clustered environment with no or minimal downtime. The Oracle Parallel Server architecture provides the following features:

• Separate Oracle instances running on different nodes operate against a set of common data files that reside on shared disks physically accessible by all nodes that make up the cluster.

• All instances can execute transactions concurrently against the same database, and each instance can have multiple users executing transactions concurrently.

Coordination of each node accessing the shared database provides the following:

• Data consistency and integrity

• Data and lock recovery support

Oracle Parallel Server and the Optimal Flexible Architecture

This guide describes the features of the Oracle Parallel Server in relation to the Optimal Flexible Architecture (OFA), whereby Oracle software and database directories and file names following a particular format. This guide recommends using tools that adhere to the OFA structure, such as the Oracle Database Configuration Assistant, to configure and administer an Oracle Parallel Server databases. Therefore, all examples and features are described from a OFA point-of-view. You are free to use other naming conventions.

Oracle Parallel Server Software Components

The following components comprise Oracle Parallel Server:

Component	Description
Oracle8i Enterprise Edition	Provides the applications and files to manage a database. All other Oracle Parallel Server components are layered on top of Oracle8i Enterprise Edition.
Oracle Parallel Server Option	Provides the necessary Oracle Parallel Server scripts, initialization files, and data files.
Oracle Parallel Server Management	Provides a single point for starting, stopping, and monitoring the activity of parallel servers and parallel server instances from within Oracle Enterprise Manager Console. See Chapter 5, "Installing and Configuring Oracle Parallel Server Management" and Chapter 6, "Administering Oracle Parallel Server".
Operating System Dependent layer	Consists of several software components developed by vendors. The Operating System Dependent layer maps the key OS/cluster-ware services required for proper operation of Oracle Parallel Server. See "Operating System Dependent Layer".

Operating System Dependent Layer

A vendor-supplied Operating System Dependent layer that passed certification must be installed after Oracle Parallel Server Option is installed. The Operating System Dependent layer consists of several software components developed by vendors. The Operating System Dependent layer maps the key OS/cluster-ware services required for proper operation of Oracle Parallel Server.

Operating System Dependent Components

The Operating System Dependent layer consists of:

Component	Description	Required?
Cluster Manager (CM)	Discovers the state of the cluster and instance membership.	Yes
Inter-Process Communication (IPC)	Provides reliable transfer of messages between instances on different nodes.	Yes
Input/Output (IO)	Provides I/O to access shared disks.	No
Startup (START)	Provides one-time configuration to startup functionality.	No

These components provide key services required for proper operation of the Oracle Parallel Server Option and are used by various clients, such as Integrated Distributed Lock Manager.

Figure 1-1 illustrates the Operating System Dependent components in a cluster with two nodes:

Figure 1-1 Operating System Dependent Components

Cluster Manager

The Cluster Manager (CM) monitors process health, specifically of the database instance. LMON, a background process that monitors the health of the Integrated Distributed Lock Manager, registers and de-registers from CM.

It is critical that all Oracle Parallel Server instances receive the same membership information when events occur. Notification changes cause relevant Oracle Parallel Server recovery operations to be initiated. If any node or instance is determined to be dead or otherwise not a properly functioning part of the system, CM terminates all processes on that node or instance. Thus, any process or thread running Oracle code can safely assume its node or instance is an active member of the system.

If there is a failure, recovery is transparent to user applications. CM automatically reconfigures the system to isolate the failed node and instance and notify Integrated Distributed Lock Manager of the status. Integrated Distributed Lock Manager subsequently recovers any of the locks from the failed node or instance. Oracle Parallel Server can then recover the database to a valid state.

The Integrated Distributed Lock Manager relies on the Cluster Manager for timely and correct information. If the Integrated Distributed Lock Manager cannot get the information it needs, it will shut down the instance.

CM performs the following two types of operations:

• Node Monitoring

• Cluster Management

The description of these operations is based on Oracle's implementation of the Operating System Dependent layer. Your vendor's implementation may differ.

Node Monitoring

CM manages access to shared disks and monitors the status of various cluster resources, including nodes, networks and instances.

Node monitoring:

• Provides the basic node management interface modules needed by Oracle Parallel Server in a cluster environment.

• Allows the Oracle Parallel Server environment to discover and track the membership state of nodes by providing a common view of cluster membership across the cluster.

• Runs on all nodes and monitors the topology of the cluster by querying all nodes for their current membership.

• Detects changes in the state of active nodes signaling those events, diagnosing the changes, and coordinating a new common and consistent state among all nodes.

• Notifies Oracle Parallel Server of the actual changes to membership.

Cluster Management

CM determines what groups are up and manages instance members. Each instance registers with its database specific group. This CM function was performed by the Group Membership Service (GMS) prior to this release.

Inter-Process Communication (Node-to-Node)

Oracle Parallel Server derives most of its functional benefits from its ability to run on multiple interconnected machines. Oracle Parallel Server relies heavily on the underlying Inter-Process Communication (IPC) component to facilitate this.

IPC defines the protocols and interfaces required for the Oracle Parallel Server environment to transfer reliable messages between instances. Messages are the fundamental logical units of communication in this interface. The core IPC functionality is built around an asynchronous, queued messaging model. IPC is designed to send/receive discrete messages as fast as the hardware allows. With an optimized communication layer, various services can be implemented above it. This is how the Integrated Distributed Lock Manager carries out all of its communication.

Input/Output

Note: Input/Output is available on Windows NT only.

The Input/Output (IO) component provides interprocess capabilities that a cluster implementation must support to enable proper operation of the Oracle Parallel Server environment.

The Oracle Parallel Server environment is extremely dependent on the ability of the underlying OS/cluster implementation to support simultaneous disk sharing across all nodes that run coordinated Oracle Parallel Server instances. Unlike switch-over based technologies, all Oracle Parallel Server instances are active and can operate on any database entity in the shared physical database simultaneously. It is this capability that gives Oracle Parallel Server a large portion of its parallel scalability. It is the role of the Integrated Distributed Lock Manager to coordinate the simultaneous access to shared databases in a way that maintains consistency and data integrity.

At a high level, the Oracle Parallel Server shared I/O model can be described as a distributed disk cache implemented across all nodes that define the Oracle Parallel Server cluster. The core of Oracle Parallel Server can be viewed as a major client of the cache. Disk blocks from the shared devices are read into a particular node instance cache only after mediation by the Integrated Distributed Lock Manager. The other node instance may read the same blocks into its cache and operate on them simultaneously. Updates to those blocks are carefully coordinated. In general, all shared disk based I/O operations are mediated by the Integrated Distributed Lock Manager. The set of distributed Integrated Distributed Lock Managers on each node can be thought of as managing the distributed aspects of the cache.

Disk update operations must be carefully coordinated so that all nodes see the same data in a consistent way. Any Oracle Parallel Server instance intending to update a cached data block must enter into a dialog with the Integrated Distributed Lock Manager to ensure it has exclusive right to update the block. Once it does this, the instance is free to update the block until its rights have been revoked by the Integrated Distributed Lock Manager. When the exclusive update right is revoked, the instance with block updates must write the block to disk so that the other node can see the changes. Given this rather high-level view of the Integrated Distributed Lock Manager I/O consistency model, it is clear that disk blocks can migrate around to each instance's block cache and all updates are flushed to disk when an instance other than the owner desires access to the block. It is this property that directly determines the reliance of Oracle Parallel Server on shared disk implementations.

Startup

Note: Startup is available on Windows NT only.

The Startup (START) component initiates the Oracle Parallel Server components in a specific order during instance startup. It is up to the vendor to determine this startup sequence.

Oracle Parallel Server Instances

Each Oracle Parallel Server instance, a running instance of Oracle8i software, is comprised of a System Global Area (SGA) and Oracle background processes. The SGA is a shared memory region that contains data and control information for an instance. Users can connect to any instance to access the information that resides within the shared database.

Oracle Parallel Server instances coordinate with the following components:

Component	Description
node	A server where an instance resides.
cluster	A set of physically interconnected nodes, and a shared disk storage subsystem.
database	The set of all instances with access to the same data files. A database is limited to a set of Oracle Parallel Server instances that run on only the nodes defined within a cluster. The database name is defined by global database name, a name comprised of the database name and domain, entered during installation.

All instances share:

• the same set of data files

• the same control files

An instance contains:

• at least the following background processes:
  • PMON

  • SMON

  • DBWR

  • LGWR

• additional background processes:
  • LCKN to manage the locks used by an instance and to coordinate requests for those locks by other instances. Additional lock processes, LCK1 through LCK9, are available for systems that require exceptionally high throughput of instance lock requests. The single lock process per instance, LCK0, is usually sufficient for most systems.

  • LMON to monitor the health of the Integrated Distributed Lock Manager.

  • LMD0 to process lock requests

  • BSP0 to manage out-going messages to requesting nodes for the Consistent Read (CR) Server. (The LCK process manages in-coming lock requests for instances holding locks needed by other instances.)

• the Integrated Distributed Lock Manager (IDLM) component in the SGA, an area for global locks and resources. This area was formerly part of the external Distributed Lock Manager (DLM).

See Oracle8i Concepts for information about Oracle8i database processes and memory structures.

.

The data files are located on disk drives that are shared between the multiple nodes. If one node fails, client applications (written to do so) can re-route users to another node. One of the surviving nodes automatically performs recovery by rolling back any incomplete transactions that the other node was attempting. This ensures the logical consistency of the database.

Note: Because an instance does not include data files, you can start up an instance without mounting the data files.

Unique Identification of Instances

Following the OFA naming convention, each node's instance has its own INITSID.ORA file that uniquely defines the instance with instance-related parameters. This file calls INITDB_NAME.ORA, which lists common parameters shared from node-to-node. Both files are created by the Oracle Database Configuration Assistant after installation.

The INITSID.ORA file is impacted by a number of elements, including:

Element	Description
Oracle System Identifier (SID)	Uniquely identifies a node's instance. The database name, specified by the DB_NAME parameter in the INITDB_NAME.ORA file, and unique thread ID make up each node's SID. The thread ID starts at 1 for the first instance in the cluster, and is incremented by 1 for the next instance, and so on. The SID is defined as an ORACLE_SID environment variable on UNIX platforms and in the ORACLE_SID registry entry on Windows NT.
Instance Name	Represents the name of the instance and is used to uniquely identify a specific instance when multiple instances share common service names. The instance name is the same as the SID. It is depicted by the INSTANCE_NAME parameter in the INITSID.ORA file.
Rollback Segments	Specifies at least two rollback segments to store "undo" information. If the ROLLBACK_SEGMENTS parameter is set in the INITSID.ORA file, an instance acquires all of the rollback segments named in this parameter, even if the number of rollback segments exceeds the minimum number required by the instance. By default, the Oracle Database Configuration Assistant creates rollback segment identified by RBSthread_id_rollback_number. For example, on node with a sequence number of 1, the rollback segments are RBS1-1 and RBS1-2.
Thread ID	Specifies the redo thread that is to be used by the instance. Any available redo thread number can be used, but an instance cannot use the same thread number as another instance. Also, an instance cannot start when its redo thread is disabled. An instance cannot mount a database if the thread is used by another instance or if the thread is disabled. The thread starts at 1 node for the first instance in the cluster, and is incremented by 1 for the next instance, and so on.Redo thread numbers are used in the naming of redo log files, which record changes made to data. When redo log files are generated, they include the thread, allowing you to easily identify a particular node's redo log files. Threads are depicted by the THREAD parameter in the INITSID.ORA file.

Examples of Relationships Between Elements

If the DB_NAME is OP and the thread IDs are 1, 2, and 3, then the SID for each node is:

Thread ID	SID
1	OP1
2	OP2
3	OP3

If the DB_NAME is OP and the node numbers are 0, 1, 4, and 6, then the instance elements are effected in the following manner:

Computer	Thread ID	SID	INSTANCE_NAME	ROLLBACK_SEGMENT name
Computer A	1	OP1	OP1	RBS1-1, RBS1-2
Computer B	2	OP2	OP2	RBS2-1, RBS2-2
Computer C	3	OP3	OP3	RBS3-1, RBS3-2
Computer D	4	OP4	OP4	RBS4-1, RBS4-2

Data Files

A database is logically divided into tablespaces that contain all data stored in the database. Tablespaces, in turn, are made up of one or more data files.

With Oracle Parallel Server, all participating instances access the same data files.

Figure 1-2 shows the relationship between two Oracle instances and the shared disks on which the data files are stored:

Figure 1-2 Oracle Parallel Server Instance Architecture

Oracle Parallel Server Features

Some of Oracle Parallel Server features to take note of include:

Feature	Description
Integrated Distributed Lock Manager (IDLM)	Maintains a list of system resources and provides locking mechanisms to control allocation and modification of Oracle resources. Every process interested in the database resource protected by the IDLM must open a lock on the resource. See "Integrated Distributed Lock Manager".
Parallel Cache Management (PCM)	Provides instance locks (with minimal use of the IDLM) that cover one or more data blocks of any class: data block, index blocks, undo blocks, segment headers, and so on. Oracle Parallel Server uses these instance locks to coordinate access to shared resources. The IDLM maintains the status of the instance locks See "Parallel Cache Management".
Oracle Parallel Query	Ships queries between nodes so that multiple nodes can execute on behalf of a single query. See "Oracle Parallel Query".

See Oracle8i Parallel Server Concepts and Administration for further information about these and other features.

Integrated Distributed Lock Manager

The Integrated Distributed Lock Manager (IDLM) maintains a list of system resources and provides locking mechanisms to control allocation and modification of Oracle resources. Resources are structures of data. The IDLM does not control access to tables or anything in the database itself. Every process interested in the database resource protected by the IDLM must open a lock on the resource.

Oracle Parallel Server uses the IDLM facility to coordinate concurrent access to resources, such as data blocks and rollback segments, across multiple instances. The Integrated Distributed Lock Manager facility has replaced the external Distributed Lock Manager which was used in earlier releases of Oracle Server.

The IDLM uses the LMON and LMDN processes. LMON manages instance and processes deaths and associated recovery for the IDLM. In particular, LMON handles the part of recovery associated with global locks. The LMDN process handles remote lock requests (those which originate from another instance).

The IDLM:

• Provides a distributed architecture. In the distributed architecture each node in the cluster (or each instance of an Oracle database) participates in global lock management and manages a piece of the global lock database. The lock database is distributed among all the participants.

• Provides continual service and maintains the integrity of the lock database in the event of multiple node and instance failures.

• Maintains information about the locks on all nodes that are interested in a given resource. In this situation, the IDLM usually nominates one node to manage all relevant information about the resource and its locks.

• Performs deadlock detection.

• Synchronizes modifications made to the Oracle data files so that no changes are lost.

• Synchronizes disk block access between multiple nodes.

The IDLM is a resource manager and, thus, does not control access to the database.

Example

A node in a cluster needs to modify block number n in the database file. At the same time, another node needs to update the same block n to complete a transaction.

Without the IDLM, both nodes update the same block at the same time. With the IDLM, only one node is allowed to update the block. The other node must wait. The IDLM ensures that only one instance has the right to update a block at any one time. This provides data integrity by ensuring that all changes made are saved in a consistent manner.

Interaction with CM

The IDLM uses CM to determine which instances are active. When the instance is started, the LMON and LMDN processes are started and the IDLM registers with CM. The IDLM deregisters with CM when the database is shutdown.

Parallel Cache Management

Parallel Cache Management (PCM) provides instance locks (with minimal use of the IDLM) that cover one or more data blocks of any class: data block, index blocks, undo blocks, segment headers, and so on. Oracle Parallel Server uses these instance locks to coordinate access to shared resources. The IDLM maintains the status of the instance locks.

PCM locks ensure cache coherency by forcing instances to acquire a lock before modifying or reading any database block. PCM locks allow only one instance at a time to modify a block. If a block is modified by an instance, the block must first be written to disk before another instance can acquire the PCM lock, read the block, and modify it.

Example

If node 1 needs access to data that is currently in node 2's buffer cache, node 1 can submit a request to the IDLM. Node 2 then writes the needed blocks to disk. Only then is Node 1 notified by the IDLM to read updated and consistent data from the disk.

PCM Lock Implementation

You use the initialization parameter GC_FILES_TO_LOCKS to specify the number of PCM locks which cover the data blocks in a data file or set of data files. The smallest granularity is one PCM lock per data block; this is the default. PCM locks usually account for the greatest proportion of instance locks in a parallel server.

PCM locks are implemented in two ways:

PCM Lock	Description
hashed locks	Hashed locks are pre-allocated and statically assigned to one or more blocks at startup time. Hashed locks offer the advantage of no overhead for lock allocation when a data block is accessed. The first instance which starts up creates an IDLM resource and an IDLM lock (in null mode) on the IDLM resource for each hashed PCM lock. The first instance initializes each lock. The instance then proceeds to convert IDLM locks to other modes as required. When a second instance requires a particular IDLM lock, it waits until the lock is available and then converts the lock to the mode required. The total number of locks that can be allocated is limited by system resources. This usually means that multiple blocks have to be covered by the same lock. In other words, there is a low lock granularity. This might result in false pinging. The startup of the instance also requires more time, since all the lock resources have to be allocated at startup time. Typically, hashed locks are never released; each will stay in the mode in which it was last requested. You can, however, specify releasable hashed locks by using the R option with the GC_FILES_TO_LOCKS parameter. Releasable hashed PCM locks are taken from the pool of GC_RELEASABLE_LOCKS.
fine-grain locks	This is the default implementation. With fine-grain locking, locks are dynamically allocated at block-access time. The resources for the lock are only allocated during the time the lock is needed and are released when the lock is released. This makes it possible to achieve very high-lock granularity. If resource minimization is the goal, fine-grain locks can also cover multiple blocks, but are still allocated dynamically. Since locks are allocated only as required, the instance can start up much faster than with hashed locks. An IDLM resource is created and an IDLM lock is obtained only when a user actually requests a block. Once a fine-grain lock has been created, it can be converted to various modes as required by various instances. Typically, fine-grain locks are releasable: An instance can give up all references to the resource name during normal operation. You can, however, allocate fixed locks in a fine grained manner with the GC_FILES_TO_LOCKS parameter. Creating a 1-to-1 ratio of locks-to-blocks creates DBA locking.

See the Oracle8i Parallel Server Concepts and Administration for further information about PCM locks.

It is possible to have both fine-grain and hashed locking enabled at the same time.

Below is a comparison of both PCM locks:

Hash PCM Locks	Fine-Grain PCM Locks
Allocated at instance startup, resulting in a slower startup.	Allocated when user requests a block, resulting in faster instance startup.
Released only at instance shutdown.	Dynamically re-used by other blocks, requiring less memory.
Statically hashed to blocks at startup time, requiring more memory.

Choosing a Lock

Use the table below to choose a PCM lock:

When to use hashed locks...	When to use fine-grain locks...
Data is mostly read-only.	Small amount of data is updated by many instances.
Data can be partitioned.	There is not enough memory for the configuration of hash locking.
Large set of data is modified by a relatively small set of instances.

Oracle Parallel Query

With Oracle Parallel Query, Oracle can divide the work of processing certain types of SQL statements among multiple query server processes.

Architectural Overview

When parallel execution is not being used, a single server thread performs all necessary processing for the sequential execution of a SQL statement. For example, to perform a full table scan (such as SELECT * FROM EMP), one thread performs the entire operation.

Oracle Parallel Query performs the operations in parallel using multiple parallel processes. One process, known as the parallel coordinator, dispatches the execution of a statement to several parallel server processes and coordinates the results from all the server processes to send the results back to the user.

The parallel coordinator breaks down execution functions into parallel pieces and then integrates the partial results produced by the parallel server processes. The number of parallel server processes assigned to a single operation is the degree of parallelism for an operation. Multiple operations within the same SQL statement all have the same degree of parallelism

Oracle Parallel Query on Oracle Parallel Server

Oracle Parallel Server provides the framework for the Parallel Query Option to work between nodes. The Oracle Parallel Query behaves the same way in Oracle with or without the Parallel Server Option. The only difference is that Oracle Parallel Server enables Oracle Parallel Query to ship queries between nodes so that multiple nodes can execute on behalf of a single query. Here, the server breaks the query up it into smaller operations that run against a common database which resides on shared disks. Because it is performed by the server, this parallelism can occur at a low level of server operation, rather than at an external SQL level.

In some applications, an individual query often consumes a great deal of CPU resource and disk I/O (unlike most online insert or update transactions). To take advantage of multi-processing systems, the data server must parallelize individual queries into units of work which can be processed simultaneously.

If the query were not processed in parallel, disks would be read serially with a single I/O. A single CPU would have to scan all rows in a table. With the query parallelized, disks are read in parallel, with multiple I/Os.

Several CPUs can each scan a part of the table in parallel, and aggregate the results. Parallel query benefits not only from multiple CPUs but also from greater I/O bandwidth availability.

Oracle Parallel Query can run with or without the Oracle Parallel Server. Without the Oracle Parallel Server option, Oracle Parallel Query cannot perform multi-node parallelism. Oracle Parallel Server optimizes Oracle8i Enterprise Edition running on clustered hardware, using a parallel cache architecture to avoid shared memory bottlenecks in OLTP and decision support applications.

Figure 1-3 Parallel Query Operation

Oracle Parallel Query Statement

Oracle Parallel Query within Oracle Parallel Server performs parallelism within a node and among nodes via the parallel query slave processes on each node.

A sample SQL statement is shown below:

After you have run a query, you can use the information derived from V$PQ_SYSSTAT to view the number of slave processes used, and other information for the system.

See the Oracle8i Concepts for further information about parallel execution.

---