More

Using GDAL to get vertices of a multipolygon

Using GDAL to get vertices of a multipolygon


I am successfully using GDAL to get the vertices of a polygon from a ShapeFile using .NET code. This link showed me how to do that:

http://geoinformaticstutorial.blogspot.com/2012/10/accessing-vertices-from-polygon-with.html

However if the shape type is a multipolygon, then this process does not get any vertices. How can I get the vertices of a multipolygon?


Here is the code I used to get this to work. This is VB.NET code:

Public Function SchoolDistrictBoundary() As List(Of List(Of Coordinate)) Dim district As Feature = schoolDistrict() Dim result = New List(Of List(Of Coordinate)) If district IsNot Nothing Then Dim geometry = district.GetGeometryRef() Dim type = geometry.GetGeometryType() If type = wkbGeometryType.wkbPolygon Then 'only one polygon Dim coordSet = New List(Of Coordinate) Dim ring = geometry.GetGeometryRef(0) For x = 0 To ring.GetPointCount() - 1 Dim point() As Double = {0, 0} ring.GetPoint(x, point) Dim coord = New Coordinate() coord.Longitude = point(0) coord.Latitude = point(1) coordSet.Add(coord) Next result.Add(coordSet) ElseIf type = wkbGeometryType.wkbMultiPolygon Then 'multiple polygons Dim boundaries = geometry.GetBoundary() For shape = 0 To boundaries.GetGeometryCount() - 1 Dim coordSet = New List(Of Coordinate) Dim polygon = boundaries.GetGeometryRef(shape) For x = 0 To polygon.GetPointCount() - 1 Dim point() As Double = {0, 0} polygon.GetPoint(x, point) Dim coord = New Coordinate() coord.Longitude = point(0) coord.Latitude = point(1) coordSet.Add(coord) Next result.Add(coordSet) Next End If End If Return result End Function

I suppose that you are happy with your script but you may find use for the Spatialite SQL dialect in the future.

Let's take a shapefile with one multipolygon to start with.

ogrinfo multipolygon.shp -al INFO: Open of 'multipolygon.shp' using driver 'ESRI Shapefile' successful. Layer name: multipolygon Geometry: Polygon Feature Count: 1 Extent: (403784.354400, 6924130.250945) - (409732.186830, 6926861.991572) Layer SRS WKT: (unknown) OGRFeature(multipolygon):0 MULTIPOLYGON (((403784.35439963144 6926261.3961154055,405838.0033819516 692686 1.9915724993,407000.44620213285 6925680.1747053144,405140.53768984287 6924692.09 83081609,403784.35439963144 6926261.3961154055)),((407484.79737720836 6926745.74 72904809,409732.18682955875 6926745.7472904809,409344.70588949829 6924323.991415 103,406922.95001412078 6924130.2509450726,407484.79737720836 6926745.7472904809) ))

Next utilize the ST_DissolvePoints function of Spatialite https://www.gaia-gis.it/gaia-sins/xmlBlob/spatialite-sql-latest.html through SQLite SQL dialect of GDAL http://www.gdal.org/ogr_sql_sqlite.html. You must have GDAL that is compiled with new enough Spatialite because the ST_DissolvePoints function is pretty new.

ogrinfo multipolygon.shp -dialect sqlite -sql "select dissolvepoints(geometry) from multipolygon" INFO: Open of 'multipolygon.shp' using driver 'ESRI Shapefile' successful. Layer name: SELECT Geometry: Unknown (any) Feature Count: 1 Extent: (403784.354400, 6924130.250945) - (409732.186830, 6926861.991572) Layer SRS WKT: (unknown) Geometry Column = dissolvepoints(geometry) OGRFeature(SELECT):0 MULTIPOINT (403784.35439963144 6926261.3961154055,405838.0033819516 6926861.99 15724993,407000.44620213285 6925680.1747053144,405140.53768984287 6924692.098308 1609,403784.35439963144 6926261.3961154055,407484.79737720836 6926745.7472904809 ,409732.18682955875 6926745.7472904809,409344.70588949829 6924323.991415103,4069 22.95001412078 6924130.2509450726,407484.79737720836 6926745.7472904809)

I know that the result is not exactly what you aim at but perhaps you will find some use for the SQLite dialect in the future.


Here's a C# version,dsis aOSGeo.OGR.Driver driverandlayerIdXis the layer you want:

OSGeo.OGR.Feature feature = null; OSGeo.OGR.Geometry geo = null, ring = null; double[] pointList = { 0, 0, 0 }; int layerIdX = 2; OSGeo.OGR.Layer layer = ds.GetLayerByIndex(layerIdX); if( geoType == "wkbMultiPolygon") { for( int i = 0; i < numberOfFeatures; ++i ) { feature = layer.GetFeature(i); if( feature != null ) { geo = feature.GetGeometryRef(); for( int j = 0; j < geo.GetGeometryCount(); ++j ) { var inGeo = geo.GetGeometryRef(j); for( int k = 0; k < inGeo.GetGeometryCount(); ++k ) { ring = inGeo.GetGeometryRef(k); int pointCount = ring.GetPointCount(); for( int l = 0; l < pointCount; ++l ) { ring.GetPoint( l, pointList ); // pointList[0] is the Longitude. // pointList[1] is the Latitude. // pointList[2] is the Altitude. } } } } } }

Ytdjtkui

Where did Heinlein say "Once you get to Earth orbit, you're halfway to anywhere in the Solar System"?

Should I outline or discovery write my stories?

Did arcade monitors have same pixel aspect ratio as TV sets?

Count the occurrence of each unique word in the file

On a tidally locked planet, would time be quantized?

What was the exact wording from Ivanhoe of this advice on how to free yourself from slavery?

Create all possible words using a set or letters

Longest common substring in linear time

Electoral considerations aside, what are potential benefits, for the US, of policy changes proposed by the tweet recognizing Golan annexation?

Loading commands from file

Melting point of aspirin, contradicting sources

Freedom of speech and where it applies

GraphicsGrid with a Label for each Column and Row

Why is so much work done on numerical verification of the Riemann Hypothesis?

Store Credit Card Information in Password Manager?

Lowest total scrabble score

Is the U.S. Code copyrighted by the Government?

Question about the proof of Second Isomorphism Theorem

How could a planet have erratic days?

Creating Multi-Part Polygons that touch internally using ArcGIS Desktop?

Converting adjacent singlepart polygons to multipart?Merging parcels but keeping Geometry separate in ArcGIS Desktop?Merge without dissolve in ArcGIS for Desktop?Finding parcel polygon features with frontage on multiple streets using ArcGIS Desktop?Merging parcels but keeping Geometry separate in ArcGIS Desktop?Splitting multi-part polygon into single polygons using ArcGIS Desktop?How to merge single polygons to multi part polygons by maximum distances?Unsplit/Dissolve multiple touching lines in Stream Network using ArcGIS Desktop?Merging lines (that might cross) at common endpoints without multi-part features in the resultDelineating boundary around many small polygons to reduce number of polygons but retain relative coverage using ArcGIS Desktop?Merging or dissolving points while maintaining attributes using ArcGIS Desktop?Dissolving feature/field by interval using ArcGIS Desktop?Is there a “Merge Parts” tool in QGIS?


Mdhntd

What does "To die quietly of old age would be to go there on foot" mean?

Busted my bike hub & derailleur (I think) - how bad is it?

Vertical equivalence arrow in the middle between two lines

What was the stated reason for giving Trump this award?

How to reward good service in Japan?

Thicken stew/sauce twice with flour

Making bread with few ingredients

Kitchen rewire gone wrong

Why do we need dedicated launch vehicles for small satellites?

I feel I've done enough for a PhD but my supervisor disagrees

My co-worker withholds information about his systems and blames me for "bugs" which aren't my fault

Is it a Hamiltonian cycle on a grid?

Why didn't Hogwarts shut down after these type of events?

7-Zip seems unable to compress zero-filled virtual disk image after 1.8 GB

If I attempt to use the Teleport spell to teleport into an area warded by the Forbiddance spell, is my spell slot expended?

Why don't combat aircraft have rear-facing *laser* weapons?

Has someone with no experience at all ever taken off in a plane? Could they?

Why do companies pay fixed dividends instead of fluid ones?

Urgently need pre-paid debit card to pay debt collector

Why is reverb typically called "wet' or "drippy"?

How to create numeronyms in bash

Sentinel2 : coordinate system of band shows none jp2 in python

Geographic Coordinate System to Projected Coordinate SystemCreating composite band mosaic of sentinel 2 data with arcpyHandle .JP2 Sentinel dataChange shapefile coordinate system using PythonNetCDF with separate lat/lon bands to GeoTiff with Python/GDALRasterio write_band output has no CRSHow to download Sentinel2 L1C using AWS Python SDK (boto3)

I am using python 3.7. I have downloaded sentinel2 images using sentinelsat python API. I have used geojsonfile multipolygon as footprint with coordinate reference system, EPSG:32632.
I am currently trying to do sentinel image analysis. So, I needed to access the coordinate. The information I have obtain is None.

The result is:
None
does anybody have possible solution?

I am using python 3.7. I have downloaded sentinel2 images using sentinelsat python API. I have used geojsonfile multipolygon as footprint with coordinate reference system, EPSG:32632.
I am currently trying to do sentinel image analysis. So, I needed to access the coordinate. The information I have obtain is None.

The result is:
None
does anybody have possible solution?

I am using python 3.7. I have downloaded sentinel2 images using sentinelsat python API. I have used geojsonfile multipolygon as footprint with coordinate reference system, EPSG:32632.
I am currently trying to do sentinel image analysis. So, I needed to access the coordinate. The information I have obtain is None.

The result is:
None
does anybody have possible solution?

I am using python 3.7. I have downloaded sentinel2 images using sentinelsat python API. I have used geojsonfile multipolygon as footprint with coordinate reference system, EPSG:32632.
I am currently trying to do sentinel image analysis. So, I needed to access the coordinate. The information I have obtain is None.

The result is:
None
does anybody have possible solution?


Bumped to the homepage by Community ♦ 15 mins ago

This question has answers that may be good or bad the system has marked it active so that they can be reviewed.

Is using ogr2ogr an acceptable option for you?

– user30184
Jul 16 '18 at 21:53

@user30184 Not really. I believe that's a command line tool, where I need it embedded in a website so a user can upload a shapefile and it gets processed (preferably using C#).

How about GDAL generally if used through C# or Python bindings? But it does feel like a heavy tool if the only problem is with ring orientations.

Yeah - I'm so close with NetTopologySuite loading the data in, there's just this one small issue, which MakeValid is supposed to fix. So I'm wondering if there's something wrong with how I'm doing it, rather than looking for a new tool.

The data I've loaded from a shapefile into a geometry column in SQL Server (using NetTopologySuite) is not producing valid GeoJSON when I export it because it doesn't follow the right-hand rule.

I thought this would be easy to fix with the MakeValid() function, but it's not correcting the issue.

I tried with a simplified geometry as below:

SQL Server says it's valid (24400: Valid), so presumably that's why it's not fixing it in the MakeValid function.

What's the best way to resolve this? I've seen an option to use:
UPDATE table SET geom = geom.STUnion(geom.STStartPoint()) which works for simple polygons, but I'm not sure how well it will work for more complex geometries.

The data I've loaded from a shapefile into a geometry column in SQL Server (using NetTopologySuite) is not producing valid GeoJSON when I export it because it doesn't follow the right-hand rule.

I thought this would be easy to fix with the MakeValid() function, but it's not correcting the issue.

I tried with a simplified geometry as below:

SQL Server says it's valid (24400: Valid), so presumably that's why it's not fixing it in the MakeValid function.

What's the best way to resolve this? I've seen an option to use:
UPDATE table SET geom = geom.STUnion(geom.STStartPoint()) which works for simple polygons, but I'm not sure how well it will work for more complex geometries.


Ytdjtkui

Why is it a bad idea to hire a hitman to eliminate most corrupt politicians?

Finitely generated matrix groups whose eigenvalues are all algebraic

What Exploit Are These User Agents Trying to Use?

Machine learning testing data

How to prevent "they're falling in love" trope

What is required to make GPS signals available indoors?

Does the Idaho Potato Commission associate potato skins with healthy eating?

Convert seconds to minutes

Car headlights in a world without electricity

How to compactly explain secondary and tertiary characters without resorting to stereotypes?

What is the most common color to indicate the input-field is disabled?

Is there a hemisphere-neutral way of specifying a season?

Knowledge-based authentication using Domain-driven Design in C#

Bullying boss launched a smear campaign and made me unemployable

OP Amp not amplifying audio signal

Processor speed limited at 0.4 Ghz

Unlock My Phone! February 2018

How to show a landlord what we have in savings?

What does the same-ish mean?

What is the fastest integer factorization to break RSA?

How do I exit BASH while loop using modulus operator?

Mathematica command that allows it to read my intentions

Do creatures with a speed 0ft., fly 30ft. (hover) ever touch the ground?

What do you call someone who asks many questions?

Spatial overlays : Union between multiline and multipolygon with geopandas

Find pairs of near features with geopandas, fiona, GDAL etcfrom geometrycollection to polygon geometry with pythonDifference between lines and polygons using geopandasgeopandas dissolve versus postGIS postgresql aggregated groupby dissolveKnown Intersecting Polygons returning false for .intersects() in geopandasFilter a GeoPandas dataframe for points within a specific countryBuffer with dissolve - Geopandas - unary_union multipolygonOverlay Union Geopandas improve performanceWhy is Union in ArcMap much faster than other approaches?How to create a shapefile [polygon type] from a Geodataframe, returned from a Oracle Spatial cursor with geometry column type=cx_Oracle.LOB?

I try to overlays (apply union) my multipolygon on the multiline with GeoPandas, but it seem like the GeoPandas overlay function work only with (Multi)polygon. Is there any way to make that with multiline and multipolygon with GeoPandas. Here is my code :

What output are you expecting overlaying polygons with lines? Split lines with polygon attributes?

Yeah exactly, that what i want to do, to split lines with polygon attributes

Can you hard-code some sample geometries using shapely objects?

Yes, i think i can dp that

I try to overlays (apply union) my multipolygon on the multiline with GeoPandas, but it seem like the GeoPandas overlay function work only with (Multi)polygon. Is there any way to make that with multiline and multipolygon with GeoPandas. Here is my code :

What output are you expecting overlaying polygons with lines? Split lines with polygon attributes?

Yeah exactly, that what i want to do, to split lines with polygon attributes

Can you hard-code some sample geometries using shapely objects?

Yes, i think i can dp that

I try to overlays (apply union) my multipolygon on the multiline with GeoPandas, but it seem like the GeoPandas overlay function work only with (Multi)polygon. Is there any way to make that with multiline and multipolygon with GeoPandas. Here is my code :

I try to overlays (apply union) my multipolygon on the multiline with GeoPandas, but it seem like the GeoPandas overlay function work only with (Multi)polygon. Is there any way to make that with multiline and multipolygon with GeoPandas. Here is my code :

What output are you expecting overlaying polygons with lines? Split lines with polygon attributes?

Yeah exactly, that what i want to do, to split lines with polygon attributes

Can you hard-code some sample geometries using shapely objects?

Yes, i think i can dp that

What output are you expecting overlaying polygons with lines? Split lines with polygon attributes?

Yeah exactly, that what i want to do, to split lines with polygon attributes

Can you hard-code some sample geometries using shapely objects?

Yes, i think i can dp that

What output are you expecting overlaying polygons with lines? Split lines with polygon attributes?

What output are you expecting overlaying polygons with lines? Split lines with polygon attributes?

Yeah exactly, that what i want to do, to split lines with polygon attributes

Yeah exactly, that what i want to do, to split lines with polygon attributes

Can you hard-code some sample geometries using shapely objects?

Can you hard-code some sample geometries using shapely objects?

Yes, i think i can dp that

Yes, i think i can dp that


Sffyju

Can stored/leased 737s be used to substitute for grounded MAXs?

How to ask rejected full-time candidates to apply to teach individual courses?

IC on Digikey is 5x more expensive than board containing same IC on Alibaba: How?

calculator's angle answer for trig ratios that can work in more than 1 quadrant on the unit circle

Is there a verb for listening stealthily?

Does GDPR cover the collection of data by websites that crawl the web and resell user data

Who's this lady in the war room?

Can gravitational waves pass through a black hole?

A German immigrant ancestor has a "Registration Affidavit of Alien Enemy" on file. What does that mean exactly?

How can I introduce the names of fantasy creatures to the reader?

As a dual citizen, my US passport will expire one day after traveling to the US. Will this work?

How does Billy Russo acquire his 'Jigsaw' mask?

Who's the Giant Batman in the back of this dark knights metal Batman picture?

Can I feed enough spin up electron to a black hole to affect it's angular momentum?

Why not use the yoke to control yaw, as well as pitch and roll?

Why did Bronn offer to be Tyrion Lannister's champion in trial by combat?

How to achieve cat-like agility?

Should man-made satellites feature an intelligent inverted "cow catcher"?

Weaponising the Grasp-at-a-Distance spell

Making Shapely box larger using GeoPandas?

I'm creating a box that covers the the area of a GeoPandas GeoDataFrame,
so that I can use it as an inverted map to later cover up unwanted spill-over data.

I need to make the box about 5% bigger to cover all unwanted data.

What is a good aproach to make a shapely box bigger?

Have a look at the buffer method: shapely.readthedocs.io/en/latest/manual.html#object.buffer By playing with the cap_style or join_style arguments, you should be able to make a buffer around that box that still results in a square box (and not with rounded corners, which will be the default)

I'm creating a box that covers the the area of a GeoPandas GeoDataFrame,
so that I can use it as an inverted map to later cover up unwanted spill-over data.

I need to make the box about 5% bigger to cover all unwanted data.

What is a good aproach to make a shapely box bigger?

Have a look at the buffer method: shapely.readthedocs.io/en/latest/manual.html#object.buffer By playing with the cap_style or join_style arguments, you should be able to make a buffer around that box that still results in a square box (and not with rounded corners, which will be the default)

I'm creating a box that covers the the area of a GeoPandas GeoDataFrame,
so that I can use it as an inverted map to later cover up unwanted spill-over data.

I need to make the box about 5% bigger to cover all unwanted data.

What is a good aproach to make a shapely box bigger?

I'm creating a box that covers the the area of a GeoPandas GeoDataFrame,
so that I can use it as an inverted map to later cover up unwanted spill-over data.


Using GDAL to get vertices of a multipolygon - Geographic Information Systems

This chapter provides conceptual and usage information about loading, storing, accessing, and working with spatial data in a Big Data environment.


    Oracle Big Data Spatial and Graph features enable spatial data to be stored, accessed, and analyzed quickly and efficiently for location-based decision making.
    Oracle Big Data Spatial and Graph supports the storage and processing of both vector and raster spatial data.
    Oracle Spatial Hadoop Image Processing Framework allows the creation of new combined images resulting from a series of processing phases in parallel.
    The first step to process images using the Oracle Spatial and Graph Hadoop Image Processing Framework is to actually have the images in HDFS, followed by having the images separated into smart tiles.
    Once the images are loaded into HDFS, they can be processed in parallel using Oracle Spatial Hadoop Image Processing Framework.
    The framework provides a raster processing API that lets you load and process rasters without creating XML but instead using a Java application. The application can be executed inside the cluster or on a remote node.
    When you create custom processing classes. you can use the Oracle Spatial Hadoop Raster Simulator Framework to do the following by "pretending" to plug them into the Oracle Raster Processing Framework.
    Oracle Big Data Spatial Raster Processing for Apache Spark is a spatial raster processing API for Java.
    Oracle Big Data Spatial Vector Analysis is a Spatial Vector Analysis API, which runs as a Hadoop job and provides MapReduce components for spatial processing of data stored in HDFS.
    Oracle Big Data Spatial Vector Analysis for Apache Spark is a spatial vector analysis API for Java and Scala that provides spatially-enabled RDDs (Resilient Distributed Datasets) that support spatial transformations and actions, spatial partitioning, and indexing.
    Oracle Big Data Spatial Vector Hive Analysis provides spatial functions to analyze the data using Hive.
    You can use the Oracle Big Data SpatialViewer Web Application (SpatialViewer) to perform a variety of tasks.

2.1 About Big Data Spatial and Graph Support for Spatial Data

Oracle Big Data Spatial and Graph features enable spatial data to be stored, accessed, and analyzed quickly and efficiently for location-based decision making.

Spatial data represents the location characteristics of real or conceptual objects in relation to the real or conceptual space on a Geographic Information System (GIS) or other location-based application.

The spatial features are used to geotag, enrich, visualize, transform, load, and process the location-specific two and three dimensional geographical images, and manipulate geometrical shapes for GIS functions.

2.1.1 What is Big Data Spatial and Graph on Apache Hadoop?

Oracle Big Data Spatial and Graph on Apache Hadoop is a framework that uses the MapReduce programs and analytic capabilities in a Hadoop cluster to store, access, and analyze the spatial data. The spatial features provide a schema and functions that facilitate the storage, retrieval, update, and query of collections of spatial data. Big Data Spatial and Graph on Hadoop supports storing and processing spatial images, which could be geometric shapes, raster, or vector images and stored in one of the several hundred supported formats.

Oracle Spatial and Graph Developer's Guide for an introduction to spatial concepts, data, and operations

2.1.2 Advantages of Oracle Big Data Spatial and Graph

The advantages of using Oracle Big Data Spatial and Graph include the following:

Unlike some of the GIS-centric spatial processing systems and engines, Oracle Big Data Spatial and Graph is capable of processing both structured and unstructured spatial information.

Customers are not forced or restricted to store only one particular form of data in their environment. They can have their data stored both as a spatial or nonspatial business data and still can use Oracle Big Data to do their spatial processing.

This is a framework, and therefore customers can use the available APIs to custom-build their applications or operations.

Oracle Big Data Spatial can process both vector and raster types of information and images.

2.1.3 Oracle Big Data Spatial Features and Functions

The spatial data is loaded for query and analysis by the Spatial Server and the images are stored and processed by an Image Processing Framework. You can use the Oracle Big Data Spatial and Graph server on Hadoop for:

Cataloguing the geospatial information, such as geographical map-based footprints, availability of resources in a geography, and so on.

Topological processing to calculate distance operations, such as nearest neighbor in a map location.

Categorization to build hierarchical maps of geographies and enrich the map by creating demographic associations within the map elements.

The following functions are built into Oracle Big Data Spatial and Graph:

Indexing function for faster retrieval of the spatial data.

Map function to display map-based footprints.

Zoom function to zoom-in and zoom-out specific geographical regions.

Mosaic and Group function to group a set of image files for processing to create a mosaic or subset operations.

Cartesian and geodetic coordinate functions to represent the spatial data in one of these coordinate systems.

Hierarchical function that builds and relates geometric hierarchy, such as country, state, city, postal code, and so on. This function can process the input data in the form of documents or latitude/longitude coordinates.

2.1.4 Oracle Big Data Spatial Files, Formats, and Software Requirements

The stored spatial data or images can be in one of these supported formats:

Both Geodetic and Cartesian data

Other GDAL supported formats

You must have the following software, to store and process the spatial data:

GCC Compiler - Only when the GDAL-supported formats are used

2.2 Oracle Big Data Vector and Raster Data Processing

Oracle Big Data Spatial and Graph supports the storage and processing of both vector and raster spatial data.

2.2.1 Oracle Big Data Spatial Raster Data Processing

For processing the raster data, the GDAL loader loads the raster spatial data or images onto a HDFS environment. The following basic operations can be performed on a raster spatial data:

Mosaic: Combine multiple raster images to create a single mosaic image.

Subset: Perform subset operations on individual images.

Raster algebra operations: Perform algebra operations on every pixel in the rasters (for example, add, divide, multiply, log, pow, sine, sinh, and acos).

User-specified processing: Raster processing is based on the classes that user sets to be executed in mapping and reducing phases.

This feature supports a MapReduce framework for raster analysis operations. The users have the ability to custom-build their own raster operations, such as performing an algebraic function on a raster data and so on. For example, calculate the slope at each base of a digital elevation model or a 3D representation of a spatial surface, such as a terrain. For details, see Oracle Big Data Spatial Hadoop Image Processing Framework for Raster Data Processing.

2.2.2 Oracle Big Data Spatial Vector Data Processing

This feature supports the processing of spatial vector data:

Loaded and stored on to a Hadoop HDFS environment

Stored either as Cartesian or geodetic data

The stored spatial vector data can be used for performing the following query operations and more:

Sevetal data service operations are supported for the spatial vector data:

In addition, there is a limited Map Visualization API support for only the HTML5 format. You can access these APIs to create custom operations. For details, see "Oracle Big Data Spatial Vector Analysis."

2.3 Oracle Big Data Spatial Hadoop Image Processing Framework for Raster Data Processing

Oracle Spatial Hadoop Image Processing Framework allows the creation of new combined images resulting from a series of processing phases in parallel.

It includes the following features:

HDFS Images storage, where every block size split is stored as a separate tile, ready for future independent processing

Subset, user-defined, and map algebra operations processed in parallel using the MapReduce framework

Ability to add custom processing classes to be executed in the mapping or reducing phases in parallel in a transparent way

Fast processing of georeferenced images

Support for GDAL formats, multiple bands images, DEMs (digital elevation models), multiple pixel depths, and SRIDs

Java API providing access to framework operations useful for web services or standalone Java applications

Framework for testing and debugging user processing classes in the local environment

The Oracle Spatial Hadoop Image Processing Framework consists of two modules, a Loader and Processor, each one represented by a Hadoop job running on different stages in a Hadoop cluster, as represented in the following diagram. Also, you can load and process the images using the Image Server web application, and you can use the Java API to expose the framework’s capabilities.

For installation and configuration information, see:

2.3.1 Image Loader

The Image Loader is a Hadoop job that loads a specific image or a group of images into HDFS.

While importing, the image is tiled and stored as an HDFS block.

GDAL is used to tile the image.

Each tile is loaded by a different mapper, so reading is parallel and faster.

Each tile includes a certain number of overlapping bytes (user input), so that the tiles cover area from the adjacent tiles.

A MapReduce job uses a mapper to load the information for each tile. There are 'n' number of mappers, depending on the number of tiles, image resolution and block size.

A single reduce phase per image puts together all the information loaded by the mappers and stores the images into a special .ohif format, which contains the resolution, bands, offsets, and image data. This way the file offset containing each tile and the node location is known.

Each tile contains information for every band. This is helpful when there is a need to process only a few tiles then, only the corresponding blocks are loaded.

The following diagram represents an Image Loader process:

2.3.2 Image Processor

The Image Processor is a Hadoop job that filters tiles to be processed based on the user input and performs processing in parallel to create a new image.

Processes specific tiles of the image identified by the user. You can identify one, zero, or multiple processing classes. These classes are executed in the mapping or reducing phase, depending on your configuration. For the mapping phase, after the execution of processing classes, a mosaic operation is performed to adapt the pixels to the final output format requested by the user. If no mosaic operation was requested, the input raster is sent to reduce phase as is. For reducer phase, all the tiles are put together into a GDAL data set that is input for user reduce processing class, where final output may be changed or analyzed according to user needs.

A mapper loads the data corresponding to one tile, conserving data locality.

Once the data is loaded, the mapper filters the bands requested by the user.

Filtered information is processed and sent to each mapper in the reduce phase, where bytes are put together and a final processed image is stored into HDFS or regular File System depending on the user request.

The following diagram represents an Image Processor job:


Description of the illustration image_processor_job.png

2.4 Loading an Image to Hadoop Using the Image Loader

The first step to process images using the Oracle Spatial and Graph Hadoop Image Processing Framework is to actually have the images in HDFS, followed by having the images separated into smart tiles.

This allows the processing job to work separately on each tile independently. The Image Loader lets you import a single image or a collection of them into HDFS in parallel, which decreases the load time.

The Image Loader imports images from a file system into HDFS, where each block contains data for all the bands of the image, so that if further processing is required on specific positions, the information can be processed on a single node.

2.4.1 Image Loading Job

The image loading job has its custom input format that splits the image into related image splits. The splits are calculated based on an algorithm that reads square blocks of the image covering a defined area, which is determined by

area = ((blockSize - metadata bytes) / number of bands) / bytes per pixel.

For those pieces that do not use the complete block size, the remaining bytes are refilled with zeros.

Splits are assigned to different mappers where every assigned tile is read using GDAL based on the ImageSplit information. As a result an ImageDataWritable instance is created and saved in the context.

The metadata set in the ImageDataWritable instance is used by the processing classes to set up the tiled image in order to manipulate and process it. Since the source images are read from multiple mappers, the load is performed in parallel and faster.

After the mappers finish reading, the reducer picks up the tiles from the context and puts them together to save the file into HDFS. A special reading process is required to read the image back.

2.4.2 Input Parameters

The following input parameters are supplied to the Hadoop command:

  • SOURCE_IMGS_PATH is a path to the source image(s) or folder(s). For multiple inputs use a comma separator. This path must be accessible via NFS to all nodes in the cluster.
  • HDFS_OUTPUT_FOLDER is the HDFS output folder where the loaded images are stored.
  • OVERLAPPING_PIXELS is an optional number of overlapping pixels on the borders of each tile, if this parameter is not specified a default of two overlapping pixels is considered.
  • GDAL_LIB_PATH is the path where GDAL libraries are located.
  • GDAL_DATA_PATH is the path where GDAL data folder is located. This path must be accessible through NFS to all nodes in the cluster.
  • THUMBNAIL_PATH is an optional path to store a thumbnail of the loaded image(s). This path must be accessible through NFS to all nodes in the cluster and must have write access permission for yarn users.
  • -expand controls whether the HDFS path of the loaded raster expands the source path, including all directories. If you set this to false , the .ohif file is stored directly in the output directory (specified using the -o option) without including that directory’s path in the raster.
  • -extractLogs controls whether the logs of the executed application should be extracted to the system temporary directory. By default, it is not enabled. The extraction does not include logs that are not part of Oracle Framework classes.
  • -logFilter <LINES_TO_INCLUDE_IN_LOG> is a comma-separated String that lists all the patterns to include in the extracted logs, for example, to include custom processing classes packages.
  • -pyramid <OUTPUT_DIRECTORY, LEVEL, [RESAMPLING]> allows the creation of pyramids while making the initial raster load. An OUPUT_DIRECTORY must be provided to store the local pyramids before uploading to HDFS pyramids are loaded in the same HDFSA directory requested for load. A pyramid LEVEL must be provided to indicate how many pyramids are required for each raster. A RESAMPLING algorithm is optional to specify the method used to execute the resampling if none is set, then BILINEAR is used.

For example, the following command loads all the georeferenced images under the images folder and adds an overlapping of 10 pixels on every border possible. The HDFS output folder is ohiftest and thumbnail of the loaded image are stored in the processtest folder.

By default, the Mappers and Reducers are configured to get 2 GB of JVM, but users can override this settings or any other job configuration properties by adding an imagejob.prop properties file in the same folder location from where the command is being executed. This properties file may list all the configuration properties that you want to override. For example,

Java heap memory ( java.opts properties) must be equal to or less than the total memory assigned to mappers and reducers ( mapreduce.map.memory and mapreduce.reduce.memory ). Thus, if you increase Java heap memory, you might also need to increase the memory for mappers and reducers.

For GDAL to work properly, the libraries must be available using $LD_LIBRARY_PATH. Make sure that the shared libraries path is set properly in your shell window before executing a job. For example:

2.4.3 Output Parameters

The reducer generates two output files per input image. The first one is the .ohif file that concentrates all the tiles for the source image, each tile may be processed as a separated instance by a processing mapper. Internally each tile is stored as a HDFS block, blocks are located in several nodes, one node may contain one or more blocks of a specific .ohif file. The .ohif file is stored in user specified folder with -out flag, under the /user/<USER_EXECUTING_JOB>/OUT_FOLDER/<PARENT_DIRECTORIES_OF_SOURCE_RASTER> if the flag 𠄾xpand was not used. Otherwise, the .ohif file will be located at /user/<USER_EXECUTING_JOB>/OUT_FOLDER/ , and the file can be identified as original_filename.ohif .

The second output is a related metadata file that lists all the pieces of the image and the coordinates that each one covers. The file is located in HDFS under the metadata location, and its name is hash generated using the name of the ohif file. This file is for Oracle internal use only, and lists important metadata of the source raster. Some example lines from a metadata file:

If the -thumbnail flag was specified, a thumbnail of the source image is stored in the related folder. This is a way to visualize a translation of the .ohif file. Job execution logs can be accessed using the command yarn logs -applicationId <applicationId> .

2.5 Processing an Image Using the Oracle Spatial Hadoop Image Processor

Once the images are loaded into HDFS, they can be processed in parallel using Oracle Spatial Hadoop Image Processing Framework.

You specify an output, and the framework filters the tiles to fit into that output, processes them, and puts them all together to store them into a single file. Map algebra operations are also available and, if set, will be the first part of the processing phase. You can specify additional processing classes to be executed before the final output is created by the framework.

The image processor loads specific blocks of data, based on the input (mosaic description or a single raster), and selects only the bands and pixels that fit into the final output. All the specified processing classes are executed and the final output is stored into HDFS or the file system depending on the user request.

2.5.1 Image Processing Job

The image processing job has different flows depending on the type of processing requested by the user.

Default Image Processing Job Flow: executed for processing that includes a mosaic operation, single raster operation, or basic multiple raster operation.

Multiple Raster Image Processing Job Flow: executed for processing that includes complex multiple raster algebra operations.

2.5.1.1 Default Image Processing Job Flow

The default image processing job flow is executed when any of the following processing is requested:

Basic multiple raster algebra operation

The flow has its own custom FilterInputFormat , which determines the tiles to be processed, based on the SRID and coordinates. Only images with same data type (pixel depth) as the mosaic input data type (pixel depth) are considered. Only the tiles that intersect with coordinates specified by the user for the mosaic output are included. For processing of a single raster or basic multiple raster algebra operation (excluding mosaic), the filter includes all the tiles of the input rasters, because the processing will be executed on the complete images. Once the tiles are selected, a custom ImageProcessSplit is created for each image.

When a mapper receives the ImageProcessSplit , it reads the information based on what the ImageSplit specifies, performs a filter to select only the bands indicated by the user, and executes the list of map operations and of processing classes defined in the request, if any.

Each mapper process runs in the node where the data is located. After the map algebra operations and processing classes are executed, a validation verifies if the user is requesting mosaic operation or if analysis includes the complete image and if a mosaic operation is requested, the final process executes the operation. The mosaic operation selects from every tile only the pixels that fit into the output and makes the necessary resolution changes to add them in the mosaic output. The single process operation just copies the previous raster tile bytes as they are. The resulting bytes are stored in NFS to be recovered by the reducer.

A single reducer picks the tiles and puts them together. If you specified any basic multiple raster algebra operation, then it is executed at the same time the tiles are merged into the final output. This operation affects only the intersecting pixels in the mosaic output, or in every pixel if no mosaic operation was requested. If you specified a reducer processing class, the GDAL data set with the output raster is sent to this class for analysis and processing. If you selected HDFS output, the ImageLoader is called to store the result into HDFS. Otherwise, by default the image is prepared using GDAL and is stored in the file system (NFS).

2.5.1.2 Multiple Raster Image Processing Job Flow

The multiple raster image processing job flow is executed when a complex multiple raster algebra operation is requested. It applies to rasters that have the same MBR, pixel type, pixel size, and SRID, since these operations are applied pixel by pixel in the corresponding cell, where every pixel represents the same coordinates.

The flow has its own custom MultipleRasterInputFormat , which determines the tiles to be processed, based on the SRID and coordinates. Only images with same MBR, pixel type, pixel size and SRID are considered. Only the rasters that match with coordinates specified by the first raster in the catalog are included. All the tiles of the input rasters are considered, because the processing will be executed on the complete images.

Once the tiles are selected, a custom MultipleRasterSplit is created. This split contains a small area of every original tile, depending on the block size, because now all the rasters must be included in a split, even if it is only a small area. Each of these is called an IndividualRasterSplit , and they are contained in a parent MultipleRasterSplit .

When a mapper receives the MultipleRasterSplit , it reads the information of all the raster´s tiles that are included in the parent split, performs a filter to select only the bands indicated by the user and only the small corresponding area to process in this specific mapper, and then executes the complex multiple raster algebra operation.

Data locality may be lost in this part of the process, because multiple rasters are included for a single mapper that may not be in the same node. The resulting bytes for every pixel are put in the context to be recovered by the reducer.

A single reducer picks pixel values and puts them together. If you specified a reducer processing class, the GDAL data set with the output raster is sent to this class for analysis and processing. The list of tiles that this class receives is null for this scenario, and the class can only work with the output data set. If you selected HDFS output, the ImageLoader is called to store the result into HDFS. Otherwise, by default the image is prepared using GDAL and is stored in the file system (NFS).

2.5.2 Input Parameters

The following input parameters can be supplied to the hadoop command:

  • MOSAIC_CONFIG_PATH is the path to the mosaic configuration xml, that defines the features of the output.
  • GDAL_LIBRARIES_PATH is the path where GDAL libraries are located.
  • GDAL_DATA_PATH is the path where the GDAL data folder is located. This path must be accessible via NFS to all nodes in the cluster.
  • IMAGE_CATALOG_PATH is the path to the catalog xml that lists the HDFS image(s) to be processed. This is optional because you can also specify a single raster to process using 𠄿ile flag.
  • USER_PROCESS_JAR_PATH is an optional user-defined jar file or comma-separated list of jar files, each of which contains additional processing classes to be applied to the source images.
  • THUMBNAIL_PATH is an optional flag to activate the thumbnail creation of the loaded image(s). This path must be accessible via NFS to all nodes in the cluster and is valid only for an HDFS output.
  • USER_NATIVE_LIBRARIES_PATH is an optional comma-separated list of additional native libraries to use in the analysis. It can also be a directory containing all the native libraries to load in the application.
  • USER_PARAMETERS is an optional key/value list used to define input data for user processing classes. Use a semicolon to separate parameters. For example: azimuth=315altitude=45
  • SINGLE_RASTER_PATH is an optional path to the .ohif file that will be processed by the job. If this is set, you do not need to set a catalog.

For example, the following command will process all the files listed in the catalog file input.xml file using the mosaic output definition set in testFS.xml file.

By default, the Mappers and Reducers are configured to get 2 GB of JVM, but users can override this settings or any other job configuration properties by adding an imagejob.prop properties file in the same folder location from where the command is being executed.

For GDAL to work properly, the libraries must be available using $LD_LIBRARY_PATH. Make sure that the shared libraries path is set properly in your shell window before executing a job. For example:

2.5.2.1 Catalog XML Structure

The following is an example of input catalog XML used to list every source image considered for mosaic operation generated by the image processing job.

A <catalog> element contains the list of <image> elements to process.

Each <image> element defines a source image or a source folder within the <raster> element. All the images within the folder are processed.

The <bands> element specifies the number of bands of the image, The datatype attribute has the raster data type and the config attribute specifies which band should appear in the mosaic output band order. For example: 3,1,2 specifies that mosaic output band number 1 will have band number 3 of this raster, mosaic band number 2 will have source band 1, and mosaic band number 3 will have source band 2. This order may change from raster to raster.

Parent topic: Input Parameters

2.5.2.2 Mosaic Definition XML Structure

The following is an example of a mosaic configuration XML used to define the features of the output generated by the image processing job.

The <mosaic> element defines the specifications of the processing output. The exec attribute specifies if the processing will include mosaic operation or not. If set to “false” , a mosaic operation is not executed and a single raster is processed if set to “true” or not set, a mosaic operation is performed. Some of the following elements are required only for mosaic operations and ignored for single raster processing.

The <output> element defines the features such as <SRID> considered for the output. All the images in different SRID are converted to the mosaic SRID in order to decide if any of its tiles fit into the mosaic or not. This element is not required for single raster processing, because the output rster has the same SRID as the input.

The <directory> element defines where the output is located. It can be in an HDFS or in regular FileSystem (FS), which is specified in the tag type.

The <tempFsFolder> element sets the path to store the mosaic output temporarily. The attribute delete=”false” can be specified to keep the output of the process even if the loader was executed to store it in HDFS.

The <filename> and <format> elements specify the output filename. <filename> is not required for single raster process and if it is not specified, the name of the input file (determined by the -file attribute during the job call) is used for the output file. <format> is not required for single raster processing, because the output raster has the same format as the input.

The <width> and <height> elements set the mosaic output resolution. They are not required for single raster processing, because the output raster has the same resolution as the input.

The <algorithm> element sets the order algorithm for the images. A 1 order means, by source last modified date, and a 2 order means, by image size. The order tag represents ascendant or descendant modes. (These properties are for mosaic operations where multiple rasters may overlap.)

The <bands> element specifies the number of bands in the output mosaic. Images with fewer bands than this number are discarded. The config attribute can be used for single raster processing to set the band configuration for output, because there is no catalog.

The <nodata> element specifies the color in the first three bands for all the pixels in the mosaic output that have no value.

The <pixelType> element sets the pixel type of the mosaic output. Source images that do not have the same pixel size are discarded for processing. This element is not required for single raster processing: if not specified, the pixel type will be the same as for the input.

The <crop> element defines the coordinates included in the mosaic output in the following order: startcoordinateX , pixelXWidth , RotationX , startcoordinateY , RotationY , and pixelheightY . This element is not required for single raster processing: if not specified, the complete image is considered for analysis.

The <process> element lists all the classes to execute before the mosaic operation.

The <classMapper> element is used for classes that will be executed during mapping phase, and the <classReducer> element is used for classes that will be executed during reduce phase. Both elements have the params attribute, where you can send input parameters to processing classes according to your needs.

The <operations> element lists all the map algebra operations that will be processed for this request. This element can also include a request for pyramid operations for example:


6 Answers 6

Not a command line solution, but since you tagged the question QGIS:

I recommend using QGIS PostGIS Manager plugin which is a GUI for shp2pgsql which allows you to upload Shapefiles to a remote database.

I think that if you qgis installed you will have also ogr2ogr (It's provided with the gdal-bin package). With the following command you will create a table called yousahpefile in your database:

There are some workarounds if you find troubles with the encoding.

except replace localhost with the IP address of the remote PostGIS database, e.g. host=192.168.123.12 or whatever

You'll need to install shp2psql on the client machine or copy the shapefile to a machine with it installed.

I think you just need to copy the relevant files. I've done this on ewindows. Haven't tried on Linux. For windows I copy the shp2pgsql.exe, libpq.dll, libiconv-2.dll, comerr32.dll, msvcr71.dll , a couple of others but those are the key ones -- these are located in the PostgreSQL bin folder

Linux ones would be different but libpq and shp2pgsql are definitely required

You can either use shp2pgsql on your local copy of the database, create a dump of the new database and upload it, .
or install shp2pgsql on the remote machine, scp the shapefile over, and shp2pgsql from there (as iant recommends).

AFAIK there is no way to do this remotely in one step. This is because even if you have access to the remote database from where you are, the functions you are using (such as shp2pgsql) are the ones installed on the server, regardless of what is on your local machine.

I'm developing a web map application that uses geoserver + express as its server and postgreSQL + postGIS as its database. In one part, I want to add some functionality that users can upload their .shp remotely. After spending days, I found that it's possible to execute shell command just using exec function that both php and express provides for us.

Although I use javascript at client and server, in the following I've given a simple piece of code for both php and express based on your need:


Ytdjtkui

How do spell lists change if the party levels up without taking a long rest?

Can we generate random numbers using irrational numbers like π and e?

Working through the single responsibility principle (SRP) in Python when calls are expensive

Didn't get enough time to take a Coding Test - what to do now?

One-dimensional Japanese puzzle

US Healthcare consultation for visitors

What to do when moving next to a bird sanctuary with a loosely-domesticated cat?

Can each chord in a progression create its own key?

Do I have Disadvantage attacking with an off-hand weapon?

How did passengers keep warm on sail ships?

University's motivation for having tenure-track positions

Python - Fishing Simulator

Word for: a synonym with a positive connotation?

Accepted by European university, rejected by all American ones I applied to? Possible reasons?

60's-70's movie: home appliances revolting against the owners

different output for groups and groups USERNAME after adding a username to a group

"is" operation returns false even though two objects have same id

Is it ok to offer lower paid work as a trial period before negotiating for a full-time job?

My body leaves my core can stay

Identify 80s or 90s comics with ripped creatures (not dwarves)

Student Loan from years ago pops up and is taking my salary

should truth entail possible truth

Has anyone used ESRI JSON data with json-schema validation?

Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)ESRI JSON Polygon Ring Orientation?Converting ESRI Web Client JSON to Standard Feature Class With Python GeoprocessorData interoperability json refreshHow to create a JSON object in Geocortex to be used in Crystal ReportsJavscript API esri/dijit/search JSON ResponseParsing and rendering JSON data in Java with GeoToolsTransforming Esri JSON to Shapefile without using ArcPy?Working with complex JSON attributes in QGISjson file with geo-coordinatesGetting data from JSON data API?


Bumped to the homepage by Community ♦ 21 mins ago

This question has answers that may be good or bad the system has marked it active so that they can be reviewed.

– aldo_tapia
Oct 7 '17 at 17:50

All the DEMs were generated in Leica Photogrammetry Suite using same set of GCPs taken from a single orthorectified image as reference. Also, I have generated respective orthorectified images from all the multidate DEMs. Those are coming perfectly coregistered.

If you are looking at changes in sediment within rivers, suggest you have a look at this work first.

I have 10 multidate Cartosat-1 stereo pair derived raster DEMs of dates ranging over a decade over a river sand area. I want to derive volumes of sand deposited in that period within that area. Is it possible?


3 Answers 3

Let's imagine a big grid mesh, like one we might use for terrain. We'll render n triangles worth of it, covering say half our 1080p screen, in a single draw call.

If we weld all of our vertices and have no smoothing/texturing seams, then each triangle has 3 vertices and each vertex is shared by 6 triangles, so we have n/2 vertices.

To render this we need to:

Run the vertex shader at least n/2 times

("at least" because our cache for vertex results is only so big. Sometimes we'll end up evicting a vertex we already transformed, then need it again for a later triangle that shares it and so re-run the vertex shader on it. So we don't get quite as much savings as it looks like on paper)

Rasterize & interpolate over at least 1920x1080/2 or about 1 million pixels of the frame buffer (since we said our terrain covers about half the screen).

("at least" because of the way GPUs work on quads of pixels, some fragments just outside the edges of polygons still get rasterized but then masked, meaning we process fragments twice. For a bumpy mesh we'll also get overdraw anywhere the mesh occludes itself, if we're not lucky enough to draw the frontmost polygon into the depth buffer first)

Run the fragment shader for all those >= 1 million fragments.

1 million results into the frame & depth buffers.

Okay, now let's unweld all of our vertices so now we have 3n vertices to render, six times more than before! Our steps are.

Run the vertex shader 3n times.

(No asterisks due to caching since every vertex is used only once, though this means the cache can't save us any time)

Rasterize & interpolate over at least 1920x1080/2 or about 1 million pixels of the frame buffer.

Run the fragment shader for all those >= 1 million fragments.

1 million results into the frame & depth buffers.

. wait, every step except the first one is the same! So most of the work that the GPU does in a typical draw call is not directly related to the number of vertices used. The amount of screen coverage, overdraw, and total triangle count make up much more of the cost.

That doesn't mean vertices are completely free. If you share vertices when you can you get some modest savings from caching, especially if your vertex shaders are complicated or your hardware's vertex pipeline is weak (as was the case on some older consoles). But given that vertex count tracks proportional to triangle count plus or minus a constant factor, it's usually not as interesting a metric of overall mesh cost.


Watch the video: 201 - Working with geotiff files using rasterio in python also quick demo of NDVI calculation