Go to content Go to navigation and search

Home

Current Oracle Spatial Blog Articles

    Convert Single Geometry to Multi-part Geometry in Oracle Spatial
    Optimized Rectangle to 5 Point Polygon
    Centroid Package now supports Y ordinate seeding
    Convert GeoJSON document to Sdo_Geometry objects
    Implementation Of Travelling Salesman Problem
    Create Polygon From Bearings And Distances
    Function That Returns a Compass Point From a Whole Circle Bearing
    Playing around with Centroids by using different seed values
    GeoRaptor 4.x Update 2
    Simple Oracle C Sprintf or Java String.format
    Some Oriented Point Functions
    Extracting Inner Rings Changed Ordinate Ordering: A Trap For Players Who Don't Read Documentation!
    PLS-00306: wrong number or types of arguments in call to 'SDO_GEOMETRY'
    Converting Google Earth Formatted Longitude/Latitude points to decimal degrees
    Oracle Business Intelligence Warehousing and Analytics - Spatial Summit
    How far inside, is inside? Measuring actual distance.
    Noding and building a polygon from single, overlapping linestrings
    Analyzing Spatial Query Performance Improvements in Oracle Spatial and Graph 12c Through Cross-Vendor Comparison
    ST_VertexN / ST_PointN - Extracting a specific point from any geometry
    Convert Single Point stored in SDO_ORDINATES to SDO_POINT_TYPE
    Aggregate APPEND Islands and XOR polygons
    Circular Arcs in Geodetic Polygons
    Some SDO_GEOMETRY/DIMINFO handling functions
    Applying And Extending Oracle Spatial - Book Released
    Changing all DIMINFO sdo_tolerance values for all metadata records in one go.
    Building Polygons from Incomplete Linestrings using ST_PolygonBuilder
    Computing Cardinal Directions to nearby geometries
    Intersecting two aggregated polygon layers with SC4O
    Spatial and Oracle 12c
    Update Triggers and SDO_GEOMETRY Equality
    Duplicate Geometry data and Data Models
    CENTROID package update
    How to calculate cumulative length of a linestring
    Useful Package of Wrapper Functions for Sdo_Util.AffineTransforms
    Compute Location from known Lat/Long point using delta easting and northing in miles
    SDO_AGGR_SET_UNION
    Sorting SDO_GEOMETRY data using the ORDER BY clause of a SELECT statement
    Creating linestrings from points
    Rounding Coordinates or Ordinates in SDO_GEOMETRY
    Effects of Sdo_Geometry Ordinate Precision on Performance
    Effects of Sdo_Geometry Ordinate Precision on Storage
    The Spatial filtering of geometries: The effect of tolerances on relationships
    Application of Delaunay Triangulation and Inverse Distance Weighting (IDW) in Oracle for Soils Interpolation
    Selecting all SDO_GTYPE values for all tables/sdo_geometry columns in a schema
    CENTROID package - Tips for Use
    Announcing the Spatial Companion For Oracle (SC4O)
    Filtering Rings (Oracle Spatial)
    Splitting a polygon using one or more linestrings
    isValid, isSimple, Dimension and CoordDim methods for SDO_Geometry
    Line Merging or Collecting lines together: ST_LineMerger
    ST_DeleteVertex for Oracle SDO_Geometry based on Jaspa/JTS
    3D/4D and SRID aware Conversion functions for SDO_Geometry: WKT and EWKT
    Topological vs Non-Topological Simplification/Generalization of Aggregated Area Geometies in Oracle
    Filtering very short linestrings via bitmap function index
    CENTROID For Oracle
    Gridding a sdo_geometry line/polygon object (Oracle)
    Finding centre and radius of a circular geometry
    Constraining geometry type for sdo_geometry column in a table.
    CASE Statements and SDO_GEOMETRY
    The Power of Constraints and Indexes for Spatial Constraints: stopping duplicate points
    Replacement for SDO_GEOM.RELATE - ST_Relate based on JTS
    Changing Oracle Spatial Index Parameters on existing index
    Writing Excel Spreadsheets files from within the Oracle database using Java and PL/SQL
    Writing xSV (eg csv) files from within the Oracle database using Java and PL/SQL
    A simple spike finder for Spatial/Locator
    JTS Java class compilation for 11g and above
    Random Spatial Search Procedure
    Geometry Snapping using JTS in Oracle
    Exposing JTS's MinimumBoundingCircle functionality
    Exposing JTS's Densifier functionality
    Using JTS's Comparison Functions - HausdorffSimilarityMeasure & AreaSimilarityMeasure with SDO_GEOMETRY
    Free JTS-based Area/Length Functions
    Handy way of systematically fixing polygon geometries with 13349 and other errors
    Standalone CENTROID package now available for download
    Free Union, Intersection, Xor and Difference Functions for Oracle Locator - Part 4 Processing Geodetic data
    Configurable Buffer: JTS and Oracle
    Free Union, Intersection, Xor and Difference Functions for Oracle Locator - Part 3
    Free Union, Intersection, Xor and Difference Functions for Oracle Locator - Part 2
    Free Union, Intersection, Xor and Difference Functions for Oracle Locator - Part 1
    Building Lines into Polygons in Oracle Locator / Spatial
    Finding Intersection Points between Line and Polygon
    SDO2GeoJSON
    Free version of sdo_length
    Alternative to my SQL based GetNumRings function
    External Tables and SDO_Geometry data.
    layer_gtype keyword issue when indexing linear data on 11g
    String Tokenizer for Oracle
    Free Aggregate Method for Concatenating 2D Lines in Oracle Locator 10g
    Reducing 5 Vertex Polygon to Optimized Rectangle
    Square Buffer
    Converting decimal seconds to string
    SDO_GEOM.VALIDATE_GEOMETRY_WITH_CONTEXT - 13356 Issues
    Valid conversion unit values for Oracle sdo_geom.sdo_length()
    Removing Steps in Gridded Vector Data - SmoothGrid for Oracle
    Oracle Spatial DISJOINT search/filtering
    Creating SDO_Geometry from geometric data recorded in the columns of a table
    Concave Hull Geometries in Oracle 11gR2
    Projecting SDO_GEOM_METADATA DIMINFO XY ordinates
    Instantiating MDSYS.VERTEX_TYPE
    New PL/SQL Packages - Rotate oriented point
    GeoRaptor Development Team
    Fast Refreshing Materialized View Containing SDO_GEOMETRY and SDO_GEOM.SDO_AREA function
    Performance of PL/SQL Functions using SQL vs Pure Code
    Implementing the BEST VicGrid Projection in Oracle 10gR2
    Making Sdo Geometry Metadata Update Generic Code
    ORA-13011 errors when using SDO_GEOM.VALIDATE_LAYER_WITH_CONTEXT()
    Extract Polygons from Compound Polygon
    Detecting sdo_geometries with compound (3-point Arcs) segments
    GEOMETRY_COLUMNS for Oracle Spatial
    Convert GML to SDO_Geometry in Oracle 10gR2
    Spatial Sorting of Data via Morton Key
    Swapping Ordinates in an SDO_GEOMETRY object
    New To_3D Function
    Extend (Reduce/Contract/Skrink) Function for Oracle
    Loading and Processing GPX 1.1 files using Oracle XMLDB
    Loading Spatial Data from an external CSV file in Oracle
    Calling the Oracle Spatial shapefile loader from within the Oracle database itself
    Implementing SDO_VertexUpdate/ST_VertexUpdate for Oracle
    Implementing SDO_RemovePoint/ST_RemovePoint for Oracle
    Implementing SDO_AddPoint/ST_AddPoint for Oracle
    ESRI ArcSDE Exverted and Inverted Polygons and Oracle Spatial
    Funky Fix Ordinates By Formula
    Implementing a SetPoint/ST_SetPoint function in Oracle
    Implementing an ST_SnapToGrid (PostGIS) function for Oracle Spatial
    Generating random point data
    Implementing an Affine/ST_Affine function for Oracle Spatial
    Implementing a Scale/ST_Scale function for Oracle Spatial
    Implementing a Parallel/ST_Parallel function for linestring data for Oracle Spatial
    Implementing a Rotate/ST_Rotate function for Oracle Spatial
    Limiting table list returned when connecting to Oracle Database using ODBC
    ST_Azimuth for Oracle: AKA Cogo.Bearing
    Implementing a Translate/ST_Translate/Move function for Oracle Spatial
    Elem_Info_Array Processing: An alternative to SDO_UTIL.GetNumRings and querying SDO_ELEM_INFO itself
    Minumum Bounding Rectangle (MBR) Object Type for Oracle
    How to extract elements from the result of an sdo_intersection of two polygons.
    How to restart a database after failed parameter change
    Fixing failed spatial indexes after import using data pump
    generate_series: an Oracle implementation in light of SQL Design Patterns
    Multi-Centroid Shootout
    Oracle Spatial Centroid Shootout
    On the use of ROLLUP in Oracle SELECT statements
    Surrounding Parcels
    Spatial Pipelining
    Using Oracle's SDO_NN Operator - Some examples
    Converting distances and units of measure in Oracle Locator
    Split Sdo_Geometry Linestring at a known point
    Forcing an Sdo_Geometry object to contain only points, lines or areas
    Unpacking USER_SDO_GEOM_METADATA's DIMINFO structure using SQL
    Generating multi-points from single point records in Oracle Spatial
    Object Tables of Sdo_Geometry
    Oracle Locator vs Oracle Spatial: A Reflection on Oracle Licensing of the SDO_GEOM Package
    FAST REFRESHing of Oracle Materialized Views containing Sdo_Geometry columns
    Australian MGA/AMG Zone Calculation from geographic (longitude/latitude) data
    Loading Shapefiles (SHP) into Oracle Spatial
    Oracle Spatial Mapping and Map Rendering Performance Tips
    The significance of sdo_lb/sdo_ub in USER_SDO_GEOM_METDATA: Do I need it?
    Oracle Spatial Forum - Melbourne April 2007
    Layer_GTypes for spatial indexes
    Oracle's SQL/MM Compliant Types
    Tips and Tricks

Search

    Browse

RSS / Atom

Email me

textpattern

Creative Commons License
All Blog Articles, Data Models and Free Source Code by Simon Greener, The SpatialDB Advisor is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Calling the Oracle Spatial shapefile loader from within the Oracle database itself

Thursday February 19 2009 at 06:07

Oracle makes available for download from its website a shapefile loader called shp2sdo. Loading data using this tool is a four step process:

  1. Execute shp2sdo to create a set of sqlplus and sqlldr files.
  2. Use SQLPlus to run the SQL file generated by shp2sdo to create the table and user_sdo_geom_metadata entries.
  3. Load the actual data (stored in files generated by shp2sdo) into the created table using sqlldr.
  4. Finally, run SDO_GEOM.VALIDATE_GEOMETRY to identify incorrect geometries, correct them using SDO_UTIL.RECTIFY_GEOMETRY and other tools, processing and, finally, create spatial indexes etc.

The shp2sdo tool is an operating system tool and so runs outside of the Oracle database processes. To effect a load of a shapefile from within the database one needs a method of being able to execute the process. There are a number of ways to do this but this blog article will concentrate on one: the use of the Java Virtual Machine that ships with every Oracle database.

Firstly, we need to write a Java class that will allow us to execute an external program. There are many ways to do this as any Googling of the relevant terms shows. I did this and chose a simple approach that does not handle stdin, stdout or stderr (and so is limited as to the error notification that can be generated).

Here is my class.

package com.spatialdbadvisor.gis.oracle;
import java.io.IOException;
import java.sql.DriverManager;
import java.sql.SQLException;

public class utilities
{
  public static int RunCommand(String command)
  {
    int exitVal = 0;
    try
    {
      Runtime rt = Runtime.getRuntime();
      Process proc = rt.exec(command);
      proc.waitFor();
      exitVal = proc.exitValue();
    } catch (Exception e)
    {
      System.out.println(e.getMessage());
      exitVal = -1;
    }
    // By convention, 0 indicates normal termination.
    return exitVal;
  }
}

Note that the function expects a single command string so if you want to pass parameters to an executable you have to put them all together with their command. See later for example.

We compile this using the java compiler that is shipped with the Oracle database (as the versions have to be exactly the same).

Since I did this work on Windows I created a cmd file which contained the following:

@ECHO OFF
SET ORACLE_DB_HOME=C:\oracle\product\10.2.0\db_1
SET CLASSPATH=%ORACLE_DB_HOME%\jdk\jre\lib
SET JAVA_HOME=%ORACLE_DB_HOME%\jdk
SET PATH=%JAVA_HOME%\bin;%ORACLE_DB_HOME%\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem

%JAVA_HOME%\bin\javac -classpath .;%ORACLE_DB_HOME%\lib\xmlparserv2.jar;%ORACLE_DB_HOME%\jdbc\lib\ojdbc14.jar;%ORACLE_DB_HOME%\md\lib\sdoutl.jar;%ORACLE_DB_HOME%\md\lib\sdoapi.jar -d C:\Temp\DBUtilities\classes C:\temp\DBUtilities\src\com\spatialdbadvisor\gis\oracle\utilities.java 

Once this is compiled, we then use the Oracle loadjava tool to load the java class we have just compiled into Oracle. Again, I wrapped this inside a WIndows cmd tool.

ECHO OFF
SET ORACLE_DB_HOME=C:\oracle\product\10.2.0\db_1
SET CLASSPATH=%ORACLE_DB_HOME%\jdk\jre\lib
SET JAVA_HOME=%ORACLE_DB_HOME%\jdk
SET PATH=%JAVA_HOME%\bin;%ORACLE_DB_HOME%\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem

IF %ORACLE_SID%_ EQU _ SET ORACLE_SID=GISDB

SET /P ousr=Enter codesys username (codesys):
IF %ousr%_ EQU _ SET ousr=CODESYS
SET /P opwd=Enter %ouser% password (codemgr):
IF %opwd%_ EQU _ SET opwd=CODEMGR

SET /P osid=Enter TNSName (%ORACLE_SID%):
IF %osid%_ EQU _ SET osid=%ORACLE_SID%

loadjava -user %ousr%/%opwd%%osid% -r -v -grant public -f C:\Temp\DBUtilities\classes\com\spatialdbadvisor\gis\oracle\utilities.class 

Which, if successful, should give messages like this:

E:\Temp\DBUtilities\PLSQL>load_java_class
Enter codesys username (codesys):
Enter  password (codemgr):
Enter TNSName (GISDB):
arguments: '-user' 'CODESYS/CODEMGR@GISDB' '-r' '-v' '-grant' 'public' '-f' 'C:\
temp\DBUtilities\classes\com\spatialdbadvisor\gis\oracle\utilities.cl
ass'
creating : class com/spatialdbadvisor/gis/oracle/utilities
loading  : class com/spatialdbadvisor/gis/oracle/utilities
granting : execute on class com/spatialdbadvisor/gis/oracle/utilities to public
resolving: class com/spatialdbadvisor/gis/oracle/utilities
Classes Loaded: 1
Resources Loaded: 0
Sources Loaded: 0
Published Interfaces: 0
Classes generated: 0
Classes skipped: 0
Synonyms Created: 0
Errors: 0

So far, so good.

We now have a compiled Java class in the Oracle JVM,

But how do we execute it?

Firstly we have to create a PL/SQL function wrapper over the top of it as follows.

create or replace Function RunCommand( p_command in varchar2 )
    Return Number
        As language java name 
           'com.spatialdbadvisor.gis.oracle.utilities.RunCommand(java.lang.String) return oracle.sql.string';

Now, we have the ability to execute an external program.

Or do we?

The Oracle Database is a very secure application. No organisation or the DBA it entrusts to ensure data security, wants to jeopordise database secutiry. Oracle’s JVM’s security is managed from the database. We can’t just have our new tool execute any program reading and writing data anywhere on disk. It has to be controlled.

Oracle does this via DBMS_JAVA’s GRANT_PERMISSION procedure. This procedure allows the DBA to grant read, write, execute, delete permissions directly to a user or via a role.

So, we need to use this to grant appropriate permissions.

For this article I copied the Oracle shp2sdo.exe application and the GeoScience Australia GeoData250K shapefiles (GeoData250K_Tas_roads.*) to the C:\Temp directory. I will do all my processing in this directory.

So, first, we need to grant appropriate permissions. As the “system” user I executed:

execute dbms_java.grant_permission( 'CODESYS', 'SYS:java.io.FilePermission', 'c:\Temp\*', 'read,write,execute,delete' );

Now we are ready to load our data.

Remember that our load is a multi-step process in which three of the processes have to be executed externally and one internally. So we now have some choices.

  1. Do we execute all the steps individually or
  2. Do we wrap all the external processes up into a single process and execute it followed by the post-load internal processes?

Because there are a number of issues relating to flow of control, operating system environment settings etc, I decided to use step two and put all three processing steps into a single command shell (or Linux/Unix shell).

Now, Windows shell programming is inflexible (Linux shell programming is far better), so please do not get too critical of what follows! To control the external processes I created a Windows command tool called shp2sdo.cmd. Here is what it does.

echo off
REM Script Usage: <user> <password> <sid> <shapefile> <tablename> <geometry column> <id column> <start_id> <xmin> <xmax> <ymin> <ymax> <srid>
rem shp2sdo.cmd GeoData250K_Tas_roads GeoData250K_Tas_roads geom gid 1 -180 180 -90 90 0.05 8311
REM

SET _directory=%1
IF NOT EXIST %_directory% GOTO NOWINDIR
SHIFT
SET _ousr=%1
SHIFT
SET _opwd=%1
SHIFT
SET _osid=%1
SHIFT
SET _shapefile=%1
IF NOT EXIST %_directory%\%_shapefile%.shp GOTO NOSHAPEFILE
SET _tablename=%2
SET _geomcolumn=%3
SET _idcolumn=%4
SET _start_id=%5
SET _xmin=%6
SET _xmax=%7
SET _ymin=%8
SET _ymax=%9
shift
SET _tolerance=%9%
shift
SET _srid=%9%

REM Note that for some strange reason Oracle will execute this batch file from: %ORACLE_HOME%\DATABASE 
REM eg c:\oracle\product\10.2.0\db_1\DATABASE so we must ensure that the data And the Shp2Sdo 
REM executable all exist within the directory we have read/write/execute permissions from within Oracle.

REM This is NOT cross-platform as it is only an example for publication
cd %_directory%

SET > %_directory%\%shapefile%shapefile%log
echo shp2sdo -o %_shapefile% %_tablename% -g %_geomcolumn% -i %_idcolumn% -n %_start_id% -d -x (%_xmin%,%_xmax%) -y (%_ymin%,%_ymax%) -t %_tolerance% -s %_srid% >> %_directory%\%shapefile%.log

REM USAGE: shp2sdo [-o] <shapefile> <tablename> -g <geometry column>
REM                -i <id column> -n <start_id> -p -d
REM                -x (xmin,xmax) -y (ymin,ymax) -s <srid>
REM 
%_directory%\shp2sdo -o %_directory%\%_Shapefile% %_tablename% -g %_geomcolumn% -i %_idcolumn% -n %_start_id% -d -x (%_xmin%,%_xmax%) -y (%_ymin%,%_ymax%) -t %_tolerance% -s %_srid%

REM Create the table
echo sqlplus %_ousr%/%_opwd%%_osid% %_tablename%.sql >> %shapefile%.log
Rem The Oracle tool does not put an EXIT or QUIT at the end of %_tablename%.sql so add it.
echo EXIT; >> %_tablename%.sql 
sqlplus %_ousr%/%_opwd%%_osid% %_directory%\%_tablename%.sql

echo sqlldr %_ousr%/%_opwd%%_osid% %_directory%\%_tablename% >> %shapefile%.log
REM Finally, load the data using sql*loader.
sqlldr %_ousr%/%_opwd%@%_osid% %_directory%\%_tablename%

GOTO EXITSCRIPT
:NOSHAPEFILE
ECHO %_directory%\%_shapefile%.shp does not exist >> %_directory%\%shapefile%.log
GOTO EXITSCRIPT
:NOWINDIR
ECHO %_directory% does not exist >> %_directory%\%shapefile%.log
:EXITSCRIPT
exit
REM ===================================================================================

Which we would execute this way:

declare
  v_err integer;
begin
  v_err := RunCommand('c:\temp\shp2sdo.cmd c:\temp gis gis GISDB GeoData250K_Tas_roads GeoData250K_Tas_roads geom gid 1 -180 180 -90 90 0.05 8311' );
  dbms_output.put_line('RunCommand (0==SUCCESS) = ' || v_err);
end;

RunCommand (0==SUCCESS) = 0

After executing RunCommand, we can check the results in SQL Developer.

Table created by shp2sdo external processing.

And then go on to create spatial indexes etc. (This could be added to the above processing: I leave that as an exercise for the interested used.)

This is pretty neat and simple processing (I do not present myself as a seasoned or experienced Java programmer). However, note that most DBAs get very worried about this sort of processing so, if you are thinking about implementing this sort of processing in your database, make sure you provide your DBA with lots of chocolates and beer.

I hope this article is useful to someone.

Creative Commons License

post this at del.icio.uspost this at Diggpost this at Technoratipost this at Redditpost this at Farkpost this at Yahoo! my webpost this at Windows Livepost this at Google Bookmarkspost this to Twitter

Comment [2]

Filling SDO_POINT of an POLYGON (2003) with its centroid increse performance of spatial queries?

— Luciana · 26 June 2009, 17:24 · #

No.
Simon

Simon · 10 December 2009, 06:55 · #