学步园

Oracle Text Examples

by Jeff Hunter, Sr. Database Administrator

Contents

1. Overview
2. Types of Index
3. Supported Column Types
4. Supported Document Formats
5. Setting Up Your Environment
6. Attention Linux Users!
7. Detecting / Viewing Errors During Index Creation
8. Your first CONTEXT Index Example
9. Demonstrate using BFILE
10. Demonstrate using BLOB
11. Managing DML Operations for a CONTEXT Index
12. Text Index Creation Strategies
13. Example Code Repository

Overview

Oracle Text is a tool that provides mechanisms for developers to build Text Query applications as well as Document Classification applications. Oracle Text provides indexing, word and theme searching , and viewing capabilities for text.

For an explanation of the differences between, Oracle Text (9i), interMedia Text (8i) and Oracle ConText Cartridge (8) see: How Oracle Text (Oracle 9i) Relates To interMedia Text (Oracle 8i)

This article provides a short introduction and example of building a Text Query application.

Types of Index

Oracle Text supports the creation of three types of indexes depending on your application and text source. You use the CREATE INDEX statement to create all Oracle Text index types.

The following table describes these indexes and the type of applications you can build with them. The third column shows which query operator to use with the index.

Index Type	ApplicationType	Query Operator
CONTEXT	Use this index to build a text retrieval application when your text consists of large coherent documents. You can index documents of different formats such as MSWord, HTML, XML, or plain text. With a context index, you can customize your index in a variety of ways.	CONTAINS
CTXCAT	Use this index type to improve mixed query performance. Suitable for querying small text fragments with structured criteria like dates, item names, and prices that are stored across columns.	CATSEARCH
CTXRULE	Use a CTXRULE index to build a document classification application. The CTXRULE index is an index created on a table of queries, where each query has a classification. Single documents (plain text, HTML, or XML) can be classified using the MATCHES operator.	MATCHES

Supported Column Types

With Oracle Text, you can create a CONTEXT index with columns of type VARCHAR2, CLOB, BLOB, CHAR, BFILE and XMLType.

NOTE The column types NCLOB, DATE and NUMBER cannot be indexed using a CONTEXT index.

Supported Document Formats

Because Oracle Text can index most document formats including HTML, PDF, Microsoft Word and plain text, you can load any supported type into the text column.

When you have mixed formats in your text column, you can optionally include a format column to help filtering during indexing. With the format column you can specify whether a document is binary (formatted) or text (non-formatted such as HTML).

Click here for a detailed overview of the supported document formats in Oracle Text. (Taken from the Oracle Text Reference Version 9i)

Setting Up Your Environment

1. Create a user to used for the following examples:

       CREATE USER            ctx_demo
       IDENTIFIED BY        ctx_demo
       DEFAULT TABLESPACE   ctx_demod
       TEMPORARY TABLESPACE temp;
       GRANT connect, resource, ctxapp, dba TO ctx_demo;

2. Set the instance pramater 'text_enable = FALSE'. This parameter had to be set to 'TRUE' in ConText (Pre interMedia Text 8.1.5). Ensure that the parameter now is set to FALSE.

    

3. Include $ORACLE_HOME/ctx/bin in your PATH variable.

    

4. On unix, the environment variable LD_LIBRARY_PATH or the SHLIB_PATH (depending on platform) must be set in the environment of the user using Oracle Text.

   SHLIB_PATH must be set instead of LD_LIBRARY_PATH when working with the HP platform.

   In versions below 8.1.7 it must also be set in the environment of the user who is starting and stopping the listener. Make sure that you set environment settings before starting extproc (lsnrctl start) because extproc looks up those values at startup time and modifications to these variables are thus not visible to the process resulting in indexing errors.

   Set the environment variable to: <ORACLE_HOME>/lib:<ORACLE_HOME>/ctx/lib where <ORACLE_HOME> is the explicit full path for ORACLE HOME. DO NOT use the $ORACLE_HOME environment variable.

   The variable can also be set in the ENVS section of the listener.ora file:

     SID_LIST_listener=
       (SID_LIST=
       (SID_DESC =
       (SID_NAME = PLSExtProc)
       (ORACLE_HOME = /u01/app/oracle/product/8.1.7)
       (ENVS=LD_LIBRARY_PATH = /u01/app/oracle/product/8.1.7/ctx/lib:/u01/app/oracle/product/8.1.7/lib)
       (PROGRAM = extproc)
       )
       (SID_DESC=
       (SID_NAME=OEM1DB)
       (ORACLE_HOME=/u01/app/oracle/product/8.1.7)
       )
       )

Attention Linux Users!

For users of RedHat Linux (Version 7.1 and 7.2), you may have troubles when attempting to use any of the filters. When I first tried to index a Microsoft Word document, I recieved the following errors:

  SQL> select err_index_name, err_text from ctx_user_index_errors;
ERR_INDEX_NAME         ERR_TEXT
--------------------   -------------------------------------------------------
DOCUMENT_BLOB_TAB_T1   DRG-11207: user filter command exited with status 127
DOCUMENT_BLOB_TAB_T1   DRG-11207: user filter command exited with status 127
DOCUMENT_BLOB_TAB_T1   DRG-11207: user filter command exited with status 127

I then tried to use the ctxhx binary to check if there were any errors I could view.

  % ctxhx utl_smtp.doc utl_smtp.html
ctxhx: error while loading shared libraries: /u01/app/oracle/product/9.0.1/ctx/lib/libsc_ut.so: undefined symbol: stat

After working with Oracle on this, they advised me that this symbol stat is referenced from third party inso library libsc_ut.so. The problem is happening becuase of different behavior of linker.

This is recorded as BUG#: 2037255. You can download the following patch to fix the problem.
bug_2037255.tar

Detecting / Viewing Errors During Index Creation

There are times when an index creation operations fail. Whenever the system encounters an error indexing a row, it logs the error into an Oracle Text view.

Ensure you are connected to the database as the user who created the index and query the view CTX_USER_INDEX_ERRORS. You may also view errors on ALL indexes in the database by connecting as CTXSYS and quering the view CTX_INDEX_ERRORS.

  SELECT err_timestamp, err_text
FROM ctx_user_index_errors
ORDER BY err_timestamp DESC;

Your first CONTEXT Index Example

Navigate to the Example Code Repository portion of this article to download the files in the "VARCHAR2 Example" section.

1. Create the Text Table

     SQL> @Create_Text_Table_VARCHAR2.sql

2. Run the example test query

     SQL> @Test_Index_VARCHAR2.sql

Demonstrate using BFILE

Navigate to the Example Code Repository portion of this article to download the files in the "BFILE Example" section.

1. Create the Text Table

     SQL> @Create_Text_Table_BFILE.sql

2. Run the example test query

     SQL> @Test_Index_BFILE.sql

Demonstrate using BLOB

Navigate to the Example Code Repository portion of this article to download the files in the "BLOB Example" section.

1. Create the Text Table

     SQL> @Create_Text_Table_BLOB.sql

2. Run the example test query

     SQL> @Test_Index_BLOB.sql

Managing DML Operations for a CONTEXT Index

DML operations to the base table refer to when documents are inserted, updated or deleted from the base table. This section describes how you can monitor, synchronize, and optimize the Oracle Text CONTEXT index when DML operations occur.

Note: CTXCAT indexes are transactional and thus updated immediately when there is an update to the base table. Manual synchronization as described in this section is not necessary for a CTXCAT index.

Viewing Pending DML

When documents in the base table are inserted, updated, or deleted, their ROWIDs are held in a DML queue until you synchronize the index. You can view this queue with the CTX_USER_PENDING view.

For example, to view pending DML on all your indexes, issue the following statement:

  SELECT pnd_index_name, pnd_rowid , TO_CHAR(pnd_timestamp, 'dd-mon-yyyyhh24:mi:ss') timestamp
FROM ctx_user_pending;

This statement gives output in the form:

  PND_INDEX_NAME PND_ROWID          TIMESTAMP
-------------- ------------------ --------------------
MYINDEX        AAADXnAABAAAS3SAAC 06-oct-1999 15:56:50

Synchronizing the Index

Synchronizing the index involves processing all pending updates, inserts, and deletes to the base table. You can do this in PL/SQL with the CTX_DDL.SYNC_INDEX procedure.

The following example synchronizes the index with 2 megabytes of memory:

  BEGIN
ctx_ddl.sync_index('myindex', '2M');
END;

Setting Background DML

You can set CTX_DDL.SYNC_INDEX to run automatically at regular intervals using the DBMS_JOB.SUBMIT procedure. Oracle Text includes a SQL script you can use to do this. The location of this script is:

  $ORACLE_HOME/ctx/sample/script/drjobdml.sql

To use this script, you must be the index owner and you must have execute privileges on the CTX_DDL package. You must also set the job_queue_ processes parameter in your Oracle initialization file.

For example, to set the index synchronization to run every 360 minutes on myindex, you can issue the following in SQL*Plus:

  SQL> @drjobdml myindex 360

Index Optimization

Frequent index synchronization can fragment your CONTEXT index. Index fragmentation can adversely affect query response time. You can optimize your CONTEXT index to reduce fragmentation and index size and so improve query performance.

To understand index optimization, you must understand the structure of the index and what happens when it is synchronized.

CONTEXT Index Structure

The CONTEXT index is an inverted index where each word contains the list of documents that contain that word. For example, after a single initial indexing operation, the word DOG might have an entry as follows:

  DOG DOC1 DOC3 DOC5

Index Fragmentation

When new documents are added to the base table, the index is synchronized by adding new rows. Thus if you add a new document (DOC 7) with the word dog to the base table and synchronize the index, you now have:

  DOG DOC1 DOC3 DOC5
DOG DOC7

Subsequent DML will also create new rows:

  DOG DOC1 DOC3 DOC5
DOG DOC7
DOG DOC9
DOG DOC11

Adding new documents and synchronizing the index causes index fragmentation. In particular, background DML which synchronizes the index frequently generally produces more fragmentation than synchronizing in batch.

Less frequent batch processing results in longer document lists, reducing the number of rows in the index and hence reducing fragmentation.

You can reduce index fragmentation by optimizing the index in either FULL or FAST mode with CTX_DDL.OPTIMIZE_INDEX.

Document Invalidation and Garbage Collection

When documents are removed from the base table, Oracle Text marks the document as removed but does not immediately alter the index.

Because the old information takes up space and can cause extra overhead at query time, you must remove the old information from the index by optimizing it in FULL mode. This is called garbage collection.

Optimizing in FULL mode for garbage collection is necessary when you have frequent updates or deletes to the base table.

Single Token Optimization

In addition to optimizing the entire index, you can optimize single tokens. You can use token mode to optimize index tokens that are frequently searched, without spending time on optimizing tokens that are rarely referenced.

For example, you can specify that only the token DOG be optimized in the index, if you know that this token is updated and queried frequently.

An optimized token can improve query response time for the token.

Text Index Creation Strategies

Oracle published a small note on MetaLinks (Doc ID: 73605.1) that provides tips on how to manage and control the Oracle Text index creation process, storage preference and space usage.

The note is provided here.

Example Code Repository

VARCHAR2 Example

    Create_Text_Table_VARCHAR2.sql

SQL Script to create Text table for VARCHAR2 column type

    Test_Index_VARCHAR2.sql

SQL Script used to query VARCHAR2 Text table

    Test_Index_VARCHAR2.out

Example output of Test_Index_VARCHAR2.sql

BFILE Example

    Create_Text_Table_BFILE.sql

SQL Script to create Text table for BFILEs

    Test_Index_BFILE.sql

SQL Script used to query BFILEs Text table

    Test_Index_BFILE.out

Example output of Test_Index_BFILE.sql

BLOB Example

    Create_Text_Table_BLOB.sql

SQL Script to create Text table for BLOBs

    Test_Index_BLOB.sql

SQL Script used to query BLOBs Text table

    Test_Index_BLOB.out

Example output of Test_Index_BLOB.sql

 

-- Create_Text_Table_VARCHAR2.sql 
connect ctx_demo/ctx_demo@OEM1DB

/**//*
 ** +---------------------------------+
 ** | DROP TABLE: test_ctx_install    |
 ** +---------------------------------+
*/

DROP INDEX test_ctx_install_t1
/

DROP TABLE test_ctx_install
/

/**//*
 ** +-----------------------------------+
 ** | CREATE TABLE: test_ctx_install    |
 ** +-----------------------------------+
*/

CREATE TABLE test_ctx_install (
    id    NUMBER
  , text  VARCHAR2(80)
)
TABLESPACE ctx_demod
/

ALTER TABLE test_ctx_install
ADD CONSTRAINT test_ctx_install_pk PRIMARY KEY(id)
    USING INDEX
    TABLESPACE ctx_demox
/

/**//*
 ** +-----------------------------------------+
 ** | INSERT TEXT VALUES: test_ctx_install    |
 ** +-----------------------------------------+
*/

INSERT INTO test_ctx_install
  VALUES (1, 'The cat sat on the mat')
/

INSERT INTO test_ctx_install
  VALUES (2, 'The quick brown fox jumped over the lazy dog')
/

COMMIT;

/**//*
 ** +----------------------------------------------+
 ** | CREATE CONTEXT INDEX: test_ctx_install_t1    |
 ** +----------------------------------------------+
*/

CREATE INDEX test_ctx_install_t1
  ON test_ctx_install(text)
  INDEXTYPE IS ctxsys.context
/

 

 

-- Test_Index_VARCHAR2.sql 
connect ctx_demo/ctx_demo@OEM1DB

set pagesize 9000

COLUMN id               HEADING "Document ID"
COLUMN text  FORMAT a50 HEADING "Document Text"

prompt ========================================
prompt --> Query all documents for string "cat"
prompt ========================================

SELECT id, text
FROM   test_ctx_install
WHERE  contains (text, 'cat') > 0
/

prompt ========================================
prompt --> Query all documents for string "fox"
prompt ========================================

SELECT id, text
FROM   test_ctx_install
WHERE  contains (text, 'fox') > 0
/

 

-- Test_Index_VARCHAR2.out 
========================================
--> Query all documents for string "cat"
========================================

Document ID Document Text
----------- --------------------------------------------------
          1 The cat sat on the mat

========================================
--> Query all documents for string "fox"
========================================

Document ID Document Text
----------- --------------------------------------------------
          2 The quick brown fox jumped over the lazy dog

 

--Create_Text_Table_BFILE.sql 
connect ctx_demo/ctx_demo@OEM1DB

/**//*
 ** +---------------------------------+
 ** | CREATE TABLE: document_tab      |
 ** +---------------------------------+
*/

DROP INDEX document_tab_t1
/

DROP TABLE document_tab CASCADE CONSTRAINTS
/

CREATE TABLE document_tab (
    doc_id_no           NUMBER
  , doc_name            VARCHAR2(200)
  , doc_value           BFILE
  , timestamp           DATE
)
TABLESPACE ctx_demod
/

ALTER TABLE document_tab
ADD CONSTRAINT document_tab_pk PRIMARY KEY(doc_id_no)
    USING INDEX
    TABLESPACE ctx_demox
/

/**//*
 ** +---------------------------------------------+
 ** | CREATE TRIGGER: document_tab_timestamp      |
 ** +---------------------------------------------+
*/

CREATE OR REPLACE TRIGGER document_tab_timestamp
  BEFORE INSERT OR UPDATE on document_tab
  FOR EACH ROW
  BEGIN
    :new.timestamp := sysdate;
  END;
/

/**//*
 ** +---------------------------------------------+
 ** | CREATE SEQUENCE: document_tab_seq           |
 ** +---------------------------------------------+
*/

DROP SEQUENCE document_tab_seq
/

CREATE SEQUENCE document_tab_seq
       INCREMENT BY 1
       START WITH 1000
       NOMAXVALUE
       NOCYCLE
/


/**//*
 ** +---------------------------------------------+
 ** | CREATE DIRECTORY: document_tab_dir          |
 ** +---------------------------------------------+
*/

CREATE OR REPLACE DIRECTORY document_tab_dir AS '/u01/app/oracle/Programming/ctx/DocIndex/files'
/

GRANT READ ON DIRECTORY document_tab_dir TO ctxsys
/

COMMIT;

/**//*
 ** +---------------------------------------------+
 ** | INSERT VALUES: document_tab                 |
 ** +---------------------------------------------+
*/


INSERT INTO document_tab (
      doc_id_no
    , doc_name
    , doc_value
  )
  VALUES (
      document_tab_seq.nextval
    , 'Java_Stored_Procedures_Oracle817.pdf'
    , BFILENAME('DOCUMENT_TAB_DIR', 'Java_Stored_Procedures_Oracle817.pdf')
  )
/

INSERT INTO document_tab (
      doc_id_no
    , doc_name
    , doc_value
  )
  VALUES (
      document_tab_seq.nextval
    , 'Index_B_Tree_Management.doc'
    , BFILENAME('DOCUMENT_TAB_DIR', 'Index_B_Tree_Management.doc')
  )
/


INSERT INTO document_tab (
      doc_id_no
    , doc_name
    , doc_value
  )
  VALUES (
      document_tab_seq.nextval
    , 'oracle8i_dbscripts.txt'
    , BFILENAME('DOCUMENT_TAB_DIR', 'oracle8i_dbscripts.txt')
  )
/

COMMIT;

/**/