View in PDF format

Bigger, Bigger, and Bigger Projects.

Introduction.

The innovation in the TNT products discussed above is interesting and potentially useful in some of your applications.  But it is certain that the size and scope of your geospatial analysis tasks for the TNT products will grow and grow and continually require our development efforts to keep up.  You can handle these large geodata tasks in a lot of little orthoimage pieces or map units using a batch-like strategy, but this may impose limitations, such as edge effects.  You can also approach these tasks for your nation, state, or the world using large, single objects and Project Files. 

It should be clear to you that just as the outer boundaries of your projects expand toward some physical, geographical, or political boundary, the inner detail required will also continue to increase.  Available image resolution and extent will increase, GPS precision for line oriented materials will increase, datum and local coordinate reference systems will be refined, and so on. 

This is why the Coordinate Reference System (CRS) standardization and related modifications have been released in RV7.0.  These standardized coordinate systems are the base needed to build these obvious future “really big jobs” and meet their accuracy requirements.  For example, the new High Accuracy Reference Network (HARN) datums and transformations in and out of older datums are being introduced into geodata sets in Japan,  Europe, and the US to support new accuracy requirements. 

Future geospatial applications require standardized, industry wide, exchangeable, and accurate CRSs.  It may be as simple as resolving the current arguments between image providers about how to standardize the access to the CRS of JPEG2000 compressed images (*.jp2 files) moved around the Internet.  Or, it may be as complicated as some future application in automatically driving a vehicle.

As you need them, these new demands are met by our TNT upgrades released to respond to these new requirements in size and accuracy.  Often, it is you clients who are professional geospatial analysts (in other words, those using TNTmips as part of earning their living) who identify to us these new requirements by asking for these kinds of “bigger and better features.”  These improvements can then be applied by all clients whether you are developing geodata for a nation, state, or river basin or simply requiring very high precision for preserving the spatial relationship of features in managing a municipal infrastructure or recording and analyzing a small archaeological site.

Big Project Strategy. 

Throughout its existence, MicroImages has focused on getting the really large project completed as efficiently as possible on your desktop.  Our working premise is that if the TNT products can do the really large geodata storage, access, and analysis in acceptable times, the small activities of these types will appear to be nearly instantaneous. 

Your concept of what is a “really large project” for your TNT products has continuously ratcheted up over their 20-year evolution.  What is a “really large project” has primarily been, and continues to be defined by the availability to you of larger and larger inexpensive hard drives; CDs, then DVDs, and soon HDVDs as a publication media; the limits of your operating system; and other limits these hardware factors place on TNT objects and Project File sizes.

Our original conceptual design of the Project File, its objects, and their subsequent adjustments has enabled us to adjust the Project FIle to keep well ahead of these improvements in your storage, operating system, and processing power and the ever larger objects they permit you to use in your projects.  Today you do not hesitate to commit to undertake massive city, county, province, and country sized projects in TNTmips using single objects for each data layer.  

Your personal time is often most efficiently used if your project can be approached in this fashion, even if you need to run a complex TNT analysis task in the background while completing other work in a word processor or even when it continues running unattended overnight. Then often, upon completion of the project, you tile out the objects for export into other formats in smaller units so they can be used in less robust products.

Certainly your desktop hardware speed (processor, bus and drive access, memory, and so on) and the efficiency of a required TNT process control your efficiency in completing the task.  However, usually it is the storage media that enables you to consider undertaking it as a “really big project” rather than inefficiently in pieces or not at all.  Yes, you most certainly let us know when a particular process runs for hours or days.  We then go to work to determine if that process can be made more efficient and faster.  Sometimes it is a matter of the basic limitations of the current hardware.  However, in either case, you often move ahead with your “really big project” running slow tasks overnight or over the weekend.

Sample Past Improvements.

The Global Data Set DVD released as part of V6.9 and the Lincoln Property Viewer TNTatlas released with RV7.0 demonstrate that your large object approach to large projects can be viable and cost effective and differentiate the TNT products from others.  Keeping pace with your expectations in this area does require that new or completely revised building blocks and strategies are needed and must be gradually introduced and perfected.  Some of these “really large project” software strategies introduced over the past years in the TNT products are:

• image pyramiding for very fast display of images at any scale,

• periodic implementation of new and better means of raster compression,

• optimizing the internal structure of a polygonal vector object,

• providing simpler topologies when polygonal topology is not needed,

• using geodata in objects closer to their original CAD and shape designs,

• indexing large database tables, and

• continuing effort to speed up topology validation.

How well have these past features you have had for years met these “really big project” goals as we have evolved forward?  Inserted here is a web comment addressing the last feature listed above.  It was picked up from a discussion on a forum for an inexpensive GIS product by a user of TNTmips 6.4 (circa 2000) and the latest version of the other product.  The TNTmips 6.4 version mentioned is now more than 4 years old, a long time in this rapidly evolving business.  The inclusion of this quote is not intended to be critical of the other low-cost products mentioned (they have their own design objectives), but to present a totally objective comment on how users of products react to how our products or their other products use their time. This posting is obviously a comment from a user who remains satisfied with the capabilities of V6.4 or has a lot of time and little money for upgrades.

From a posting to manifold-l@lists.directionsmag.com on 18 November 2004 

“Another interesting thing I've just been looking at (and I know this has featured in a few discussions, and [a name] mentioned that they're working on it) is the display time for a large drawing (75mb e00 file) of watershed basins (i.e. areas). I double clicked the drawing and it opened up and is still working away at displaying the basins (quite a while now) - in the meantime I've opened up TNTmips 6.4 and displayed it in there. The initial time was about 1 minute, thereafter redrawing took less than ten seconds (while Manifold is still not showing anything but the red dot in the right hand corner). I then overlayed the flowpaths (couple of tens of thousands of lines) on the basins in TNTmips, and again the initial display time was about a minute, and thereafter redisplaying, zooming in/out, etc/ takes less than 10 seconds. Selecting basins or flowpaths is almost instantaneous (recolouring the line/area as well as showing the attribute data in the table). How do they do it? I know this has been one of TNTmips' features for many years - very fast drawing/displaying of vectors and rasters. I've seen ArcView battling with less, and so do all the open source products (especially the Java products like Jump and OpenMap) that I've worked with. It takes ages for larger vector objects to display. I'm interested in how it's done - curious. I know there's a checkbox in TNTmips, when you import vector data: Optimize vector for display. Also, a process for optimizing old (pre 6.4, I think) vector layers. Any theories? What voodoo art do they use to get MSWindows to display these things so very quickly? 

Mmmh, no drawing yet - Manifold's still oozing along... :-) 

V7.0 Results.

A major portion of the effort expended in getting you RV7.0 is to alter and improve TNT features to accommodate your bigger, bigger, and bigger project materials.  These activities are summarized here in this context of largeness and robustness improvements in RV7.0 and are covered in much more detail later in this MEMO’s corresponding technical sections.

Mosaicking: making big raster objects bigger.

Assembling mosaics of large areas from good quality orthoimages and collarless, good quality, scanned maps is becoming common.  Often the source material is large in number (for example, thousands) and/or large in uncompressed size, but provided in a compressed form such as JPEG, MrSID, or PNG.  By necessity, the target is also compressed, such as in a JPEG2000 object or a JP2 file.  Mosaic can now accept as direct input any linked raster files, does not import them, and outputs the mosaicked object in any supported raster type including a compressed JPEG2000, JPEG, or standard lossless object.  To accomplish this kind of task, even if it is so big that it takes hours, has required a lot of attention to be given to the RV7.0 version of Mosaic.

Compression; making big raster objects small again.

JPEG2000 compression is now completely integrated for use in raster objects in RV7.0 of the TNT products.  For example, you can now mosaic directly into JPEG2000 compressed raster objects.  These objects can then be used as texture layers in your TNTsim3D, greatly increasing the texture raster dimensions of the Landscape File that can be distributed on a DVD.  JPEG2000 compressed images (as well as linked MrSID, and ECW) can be used to put much larger images into your TNTatlas on DVD and correspondingly for use by your TNTserver.  JPEG2000 files can even be served up by TNTserver to reduce transmission time to a TNTclient.

Shape Object: making shapefile layers faster.

Large, single shapefiles are beginning to appear of over a gigabyte, usually due to attachments to large database structures.  The use of the new shape object concept has been advanced in RV7.0 to greatly accelerate the ability of the TNT products to directly link to large shapefiles and use them just like any other layer in a TNT view.  Accompanying this has been the addition of procedures into this automatic link to show the shapefile’s legend entries and styles in the TNT LegendView for the linked object.  

Coordinate Reference Systems (CRSs): supporting the precision needed.

You can not do many types of large projects without accurate, standard CRSs and conversions between them and their possible datums.  RV7.0 introduces a completely revised CRS based on ISO standard 19111:2003.  For example, your large project might have a small geographic extent, such as a city, but require very high accuracies.  These could be difficult specifications to accommodate without using the new High Accuracy Reference Network (HARN) datums and datum transformations enabled by this new CRS management introduced first in RV7.0.

Merging and Combining:  assembling bigger, more complex geometric objects.

Copy/Paste, extracting, merging, and combining large vector, CAD, shape, and TIN objects can now be set up to run with fewer intermediate steps using the new Geometric Conversion Engine introduced in RV7.0.  However, the larger and more complex the objects combined directly into a polygonal vector object or converted later, the greater the possibility that conflation errors (in other words, microscopic topology errors) might occur.  This improved capability and your tendency toward creating larger geometric objects, particularly vector objects, have required substantial improvements in the validation of topology to detect and resolve these errors and in speed to accommodate all the additional computation and checking this necessitates.

Scale Control: preventing meaningless, slow displays of large geometric objects.

Even a trivial thing like displaying a large vector at meaningless map/view scales has been addressed.  Now you are given a warning in the form of a Dense Layer Verification window and options if the geometric object you are selecting to display at the current view scale is so dense that it will simply fill in the view in a meaningless mass of crisscrossing lines.

This Dense Layer Verification window is illustrated in the accompanying color plate entitled Managing Display of Large Vectors.  It will appear when you select a vector object for the current view that exceeds an element density threshold.  When this warning window appears, you are being informed that at the current scale of the view the vector will be slow to render and will solidly or densely fill in the area it covers.  You can choose to dispose of this window using several toggles.

Do not add layer.

Select this toggle if you simply want to skip this layer for the moment.  You can then use some other layer(s) to zoom into your view to the local area of interest, and then come back and add in this vector when it will display with a lower element density.

Add with full visibility.

This toggle overrides this warning window and displays the vector object.

Add with scale range of.

This is the toggle that is initially on by default for each new large vector object you display. Along with this toggle you are provided with data entry boxes to set the scale range of the view window over which this layer will be drawn in the view.  Each box will have a default scale value in it that you can edit.  The larger value will determine how far you zoom into the view for this layer to begin appearing.  It is determined to be a reasonable scale from the element density.  The lower value is initially zero.  If it is set to some other value, it will determine when the layer will cease to appear as you zoom in further and further.  If you set this toggle, the current values in this window will be saved with this vector object and appear as defaults in the Dense Layer Verification window the next time the vector is added as a layer in any view.  If you want to change the scale range previously set for any layer, reset it the next time it is added or choose  Show Scale Ranges from the Options menu in the Group or Layout Controls window, and change it there.  

Add initially hidden.

This toggle adds the vector to the layer list as a hidden layer.  You can then toggle the layer on when you have zoomed into the view to a scale at which it is needed and will not be so dense that it obscures all other previously added layers.

DV7.1. Making it even better.

In DV7.1 we plan to experiment to see how other kinds of modifications can improve, or at least keep pace with, your ever increasing project sizes.  At least two areas of investigation are related directly to how fast you can work in your TNT products.

Buffering Individual Layers.

As you know, big geometric objects can take time to add to a composite view each time any layer is turned off and back on.  Using a new memory buffering approach, it may be possible to add individual new layers to a view or toggle those already showing off and then back on without regenerating all the other layers in that view. 

Selecting File Opens a View. 

Another feature of possible wide interest, which has already been implemented in DV7.1 in its initial form, is the ability to double click on a file with a supported extension (for example, *.jpg, *.jp2, *.shp, *.sid, and so on) and automatically open the TNT product registered in Windows for this file, the X server and the display process, link to the object, and display it. You can also click and open to run an SML script (*.tkp, *.sml), or a Landscape File (*.sim) for TNTsim3D, or a Project File (*.rvc).  Selecting a Project File in this fashion will open it in the registered TNT product using the first layout, the first group, or the first object it finds in that Project File.  Methods for determining and controlling which of these objects to open when more than one is present are currently being explored. This new capability is currently limited to Windows but will soon be made available for Mac OS X and other TNT supported operating systems.

Yes, this click and go even works for a single layer if the free TNTatlas/X is installed and is registered in Windows as the software used to open any of these extensions. TNTatlas/X is downloaded and installed as part of every TNTlite and does not have the object size limitations of the other TNT products included in TNTlite.  Thus, any supported file type or TNT object can be opened in this fashion by a lite user and use all the features and tools in TNTatlas/X, but only one object at a time.   If you want to view more than one layer or layout then install and register these extensions for use with the new, lower cost TNTview product.  Using the mouse to select a *.jp2 or *.sid file of many gigabytes before or after compression can automatically open TNTatlas (or your other TNT product) to view the raster in seconds!  Shapefiles may take a few more seconds but work just the same.  Will this new feature make the RV7.1 TNTatlas the most powerful and useful FREE geodata viewer available?

New Feature Priorities.

Will the feature be there when you need it?

The TNT products probably cover the widest range of geospatial application areas and uses of any single product without expensive extensions or options.  However, in a specific project, you often concentrate your use and requests for new features in a specific area.  For example, your interest is primarily in using TNTatlas as a geodata publishing tool.  You may go along in this fashion for years with this activity and may not feel you need to keep your TNTmips current.  Suddenly you encounter a situation, say a new operating system, new hardware lacking an earlier feature (for example, a portable with no parallel port), a data source such as MrSID, large mosaics, a detailed project area with a special local coordinate reference system, and so on.  Any one of these may require you to take advantage of the newest features in the latest version of TNTmips to make your TNTatlas.  Depending upon the potential generic nature of your request, you will find it has been addressed in a recent version of the TNT products.  If you do not find that feature in the current version, you begin to lobby for it or even for a completely new direction in product development (for example, the introduction of manifolds in this release).  Every request we receive is documented and assigned a priority.

It’s a case of setting priorities. 

At the present time we have 2750 new feature requests of varying priorities, which have accumulated over the last 20 years. The majority of these were not logged by you, but internally by MicroImages’ staff as part of our internal design decisions.  RV7.0 completed 281 new feature requests from this list while 404 were added since RV6.9.   To manage our resources and product development we have to establish priorities for which new features we add and which we do not add for each new release. Once you have received back the code number assigned to any new feature you have submitted, you can check the priority we give to it at www.microimages.com/support/features/.  The initial priority we assign to a new feature request is based upon our perception of whether or not that feature represents a commonly needed feature for our clients.    However, assigning a high priority to any new feature does not mean that it will be in the next release.  Often this is because it requires some lower level, underlying, and not obvious developments in the TNT code and must await these developments; it can already be accomplished by other TNT procedures; it is large and complex to implement; or it simply becomes less important due to other internal software or operating system changes.

Increasing a priority.

Finding that a less than high priority has been assigned to your new feature request does not mean you should give up on getting it.  Sometimes you have to be persistent and convince us that it is in our interest and that of other clients to raise its priority.  Or, you may want to discuss the possibility of contracting with MicroImages to add that feature to the TNT products.  Often we will cost share the expenses of such developments.  But, even if it is added in this contract fashion, the feature will not be proprietary and will become available to everyone in the next TNT release.

  Opportunity costs. 

Many commercial product companies will not respond to custom feature requests.  Why?  Because the true cost, not your perceived cost of commercial software development, is not obvious.  These are called “lost opportunity costs.”  This is the cost(s) of not completing the most important features of common and wide interest at the earliest date with a limited resource, in this case the time of the software engineering manpower available. Software engineers experienced and capable of doing TNT programming are a limited resource that can not be replaced simply because extra money is made available.

Also keep in mind that any new feature added to our commercial product whether by our initiative, your lobbying, or by a contract must be maintained indefinitely on all platforms, may restrict other future developments, must be supported for all, requires documentation (for example, at least color plates), may delay widely usable generic developments, can delay a release and thus, subscription fulfillment, and so on. Consider these hidden costs of software development to the extreme by reflecting on how large the Microsoft employee base has grown.  Yet clearly this gradual increase in their size was not accompanied by a proportional increase in the number of software engineers actually coding their products.  As they grew larger and larger, the proportion of their staff members writing product code is quite small in comparison.

Design costs.  

However, the biggest single hidden cost to us in adding more complex new features via contract can be even less obvious.  It’s the time required to figure out what you want by our management and our software engineers, especially when we speak different languages both literally and professionally and reside in different nations necessitating written communications.  We are a software development, production, and marketing company and are not staffed as you are by geospatial analysts or professionals in the various application disciplines.  Yes, we have some professionals from the application areas, but they are responsible for preparing written materials for all users and associated general software feature design and testing.  Whether you are lobbying for a new feature, or possibly contracting for its addition, considerable momentum and waste of time can be overcome if you simply arrange a visit with us.  And, of course, you always have the opportunity to satisfy a special or unique need in the TNT products using TNT scripts (SML) or our TNTsdk. 

Hardware News  

Geologic Mapping Station.

Periodically this section brings to your attention hardware configurations of particular interest to you as a geospatial analyst, such as good portables, good US$3000 workstations, and so on.  This time it discusses a high-end Apple system designed for a specific purpose that is being used with available geodata and the latest RV7.0 32-bit and 64-bit builds of TNTmips.  This specific purpose is only one example of the many large project uses for such a workstation.

Apple Based High Resolution Workstation.

Apple has available large, high resolution flat panel monitors in 20" (1680 by 1050 pixels), 23" (1920 by 1200 pixels), and 30" (2560 by 1600 pixels) sizes.  A MicroImages client site has recently explained their configuration and use of TNTmips systems on Mac OS X based workstations using these Apple monitors.  Their stations are based on Apple Power Macs with dual 2.5 GHz G5 processors (US$3000), a 30" monitor (US$3000), and a 23" monitor (US$1800).  The 30" monitor requires an additional NVIDIA GeForce 6800 GT or Ultra DDL display card (US$600) to support its extra high frequency refresh rates.  Total cost of this Apple workstation is US$8300 plus some additional US$ for more memory.  The top of the line Power Mac has 8 memory slots for 8 GB and Apple promises very soon (V10.4 = Tiger?) to let a single TNTmips process use more than 4 GB.  The TNTmips Mac OS X system for this workstation would cost $5000 to $7500 depending on its location and the need for the large format printing option.

Application.

The application of these workstations is to compile geologic maps and other forms of geologic information to guide the search for coal, oil, gas, and minerals in a nation.  The total cost of each system is equal to a single day of field work for a professional geologist in costly, remote, and/or potentially dangerous locales.  To reduce or eliminate field work time in hazardous areas, they are using these workstations for their office preparation or the entire compilation task.  The results will be compiled and distributed at 1:100,000 scale in printed and electronic form and then recompiled from this scale to a 1:250,000 series.

Geodata Available.

Maps.

An existing reconnaissance level geologic map of the nation was originally compiled on 1:250,000 base maps and is available in a single vector object with attributes. 

More than 1000 1:50,000-scale topographic maps are available in ~150 dpi scanned form for the entire nation in JPEG format (*.jpg) and georeferenced with a world file (*.jgw).  The collars, or marginalia, for these maps have been trimmed away.  RV7.0 can link to and display these JPEG maps directly from this format. They can also be mosaicked directly from this JPEG format and the result saved as a single JPEG2000 compressed raster in one step for the whole nation under Mac OS X. 

Elevation Models

SRTM 30-meter elevation data is available for the nation.  RADAR-shadow holes in the SRTM data have been patched using elevation data from other sources and some proprietary software.

Imagery.

Landsat. 

Landsat ortho imagery is available for the entire nation at 30 meters.  It has been extensively processed to bring up the geologic detail, such as to remove as much vegetation as possible.  A proprietary process has been used to remove terrain induced radiance effects using the DEM.  For high resolution viewing, the natural color and special image color enhancements, such as ratioing, have been pan sharpened to about 15-meter resolution.

The complete 15-meter natural color imagery for this site is in MrSID format (*.sid) in 1 by 1 degree units.  This same kind of 30-meter Landsat image coverage of most of the world can be downloaded from NASA in MrSID format (for details see the section below entitled Reference Geodata).  In either case, RV7.0 on Windows, Mac OS X, and Linux/UNIX can link to and display multiple MrSID images directly from this format. These images can also be mosaicked directly from the MrSID format and the result saved as a single JPEG2000 compressed raster in one step for the whole nation under Mac OS X.

High Resolution.  

Higher resolution, 1-meter, pan sharpened imagery from IKONOS and QuickBird is available for spot locations.

Other.

Gravimetric and magnetic surveys are also available for some areas, as well as other miscellaneous geodata of more detail for spot areas. 

Working at Map Scale on the 30" Monitor.

A 30" monitor is used for the 2D composite view of the georeferenced 1:50,000 color map quadrangle (white areas transparent) superimposed on the 15-meter images of various types in the TNT Spatial Data Editor. This is the base upon which the detailed map units are interpreted and drawn.

Map Views “To Scale.”

The single 1:50,000 topographic map scanned at >150 dots per inch in color yielded 5134 columns and 3707 lines and can be viewed at 1:50,000 on the 30" monitor.   The horizontal maximum fit of this on the 30" monitor is 5134 map cells / 2560 pixels or ~2.0 map cells or dots per pixel.  The vertical maximum fit of this on the 30" monitor is 3707 cells / 1600 pixels or ~2.3 map cells or dots per inch.  Thus, fitting the entire map on the screen would sample 2.5 map cells into a screen pixel allowing for the vertical dimension and the marginalia of the TNT view.  This translates to viewing the map at a resolution of about >150 dpi / 2.5 dots per cell or the equivalent of viewing the map at about a 75 to 100 dot per inch scan resolution.  When resampled from the 2X pyramid layer formed in the link file for these JPEG files, it produces a readable overlay of map features including the contours.  

Alternatively, this base map could be viewed on this 30" monitor at about 1:100,000 at the full 150 dpi scan resolution. Then the 1X zoom icon would bring up a composite view of ¼ of the area of the map in seconds to about 1:50,000 design scale noted above.

Image View “To Scale.”

Several layers of Landsat images are added to this view before the transparent topographic map is added (best enhancement for materials A, best enhancement for materials B,… natural color, …).  This permits toggling them on and off and using the View-in-View types of tools.  These ~15 meter Landsat images require about 3300 pixels horizontally to display at 1:50,000.  So for the 30-inch monitor and a map scale of 1:50,000, you are viewing nearly the full resolution of these Landsat images overlaid by a readable ~75 dpi map.  For a 2X zoom the imagery is zoomed ~2x and the map is displayed at nearly its 150 dpi design scan resolution.

Any Area “To Scale.”

The 30" monitor nicely fits the familiar map scales and boundaries as outlined.  However, you are not restricted to working map by map.  Working in this same range of scales you can just as easily get a 2D TNT view of the images and map at the same scales where their 4 corners mosaic together, but without first mosaicking them.  However a more direct approach would be to use the sketch tool in the GeoToolbox to draw features across object boundaries in a 2D view and save them as a CAD object.  Later this CAD object can be merged, refined with attributes, styles, and so on in TNTedit.  

This “sketch it first” is the classic approach to all visual image interpretation (geology, forestry, …).  First concentrate on how the big picture fits together and capture the important linear features in the easiest way possible by sketching them on mylar overlays, which translates into sketching in a TNT view in display.  Then, as your understanding of the site and its 3D structure matures, you edit and refine that initial line work in the TNT Spatial Data Editor as a CAD object or as converted to a vector object.  In this step you unify and encode the structure and finally identify it (for example, add attributes and styles) within a schema and presentation standards used by others in a printed and/or electronic form.

Referencing 3D and Control via the 23" monitor.

The 23" monitor is used for TNT process control together with 3D views of the same or other combinations of these image and map materials to act as a substitute for being on-the-ground.  This view is open, altered to various viewpoints, and used concurrently with the mapping activities on the 2D map base in the Display, Spatial Data Editor, or other TNT processes.  Various interrelationships between 2D and 3D views can be established, such as concurrent layer control and a gadget that shows a trapezoidal outline in the 2D view of the edges of the area in the perspective 3D to help understand its orientation when the 3D viewpoint is changed.  

The 3D view can also use other combinations of layers such as a Landsat natural color image for realism or the enhanced false color images; color coded elevation, shaded relief, contours, and topographic maps; a vector overlay of the reconnaissance geologic map, lineaments, or the new interpretations; and so on.  As these 3D views are rotated around, they help in visualizing how the surface structural features and their manifestations indicate subsurface structures and trends.  These 3D views can then be used to 2D sketch in the detail between the surface features seen only as edges in the 2D view.   DigitalGlobe and IKONOS images of higher resolution available for some small areas are used to help identify the edge features that can be seen and traced out for the total area being mapped.   They act as a substitute for some of the ground truth and also reduce the cost of time in the field or improve its value.   

Stereo.

The new and improved stereo capabilities in RV7.0 and related hardware are discussed and illustrated in a later section.  These have not yet been factored into this geologic mapping project. However, the use of mirror stereoscopes or the Sharp and SeeReal direct view 3D monitors may be of use to improve the 3D understanding of the area being mapped. Similarly, the value of manifolds for constructing and visualizing geologic profiles in these 3D views is just becoming available to this project in this RV7.0 release.

Rendering Speeds.

2D Views.

TNTmips is the fastest system available on the Mac OS X or a Windows-based platform for handling composite views of multiple layers to scale.  Pyramiding rasters, fast JPEG2000 decompression, vector optimization, and scale control are just some of the examples of how multiple objects of any size are rapidly read to form a composite view.  However, if you want to delete or add a layer to the composite view, it is all redrawn from these various sources.  At this time for DV7.1 MicroImages is experimenting with using separate real memory buffers of 32 bits each (RGB and alpha) for each layer.  This would not change the time to create the original composite view, but might significantly reduce the time needed to toggle layers on and off or add a new layer.  In this example application, multiple Landsat layers are loaded at the outset as noted above.  If they could be rapidly toggled on to become the image exposed under the transparent map layer, this would improve their geologic interpretation.  This might be possible using this new buffer per layer by simply rotating their position in the display order.  

3D Views.

As discussed below, making composite 3D views is now not only of high quality, but in RV7.0 also has new features and all previous features restored, for example, layer transparency.  Even reorientation of the viewpoint is faster in RV7.0, for example 10 to 20 seconds.  This redisplay speed is important in this type of geological application and in other plans for the TNT products.  Redisplaying a 3D view is already 2 to 4 times faster in DV7.1 and work in this area is being actively continued. 

Windows High End Workstation.

The following dual display subsystems can be put in the Windows PC of your choice to use for a similar application to that outlined above.  These high end Windows workstations are also currently being used by other TNTmips clients for applications where the highest quality image display is important.

Highest Resolution Color Monitor.

ViewSonic 22" VP2290b (3840 by 2400 pixels called QUXGA-Wide) (US$6000) www.viewsonic.com/support/desktopdisplays/lcddisplays/proseries/vp2290b/.

The same monitor is also available from IIyama and IBM.  It is used in place of the 30" Apple monitor noted above for the 2D image/map display.  This monitor uses 4 VDI video inputs to create the refresh rates needed for this resolution.  Therefore, it must be coupled with the Matrox Parhelia HR256 (US$2500) quad out display board (see www.matrox.com/mga/products/p_hr256/home.cfm).

Best 3D Companion Monitor.

These monitors are very similar in screen design and capability to the Apple 23" monitor used for this purpose above.

HP 23" monitor HP f2304 (1920 by 1200 pixels) High-Definition LCD. (~US$2000).

Sony 23" monitor SDM-P234/B (1920 by 1200 pixels) (from Dell at US$1900).

The Matrox Parhelia HR256 board used for the ViewSonic VP2290b uses a PCI bus slot.  Thus, one of these 23" monitors can be added via the display board in the standard AGP slot of the PC, assuming it supports their higher resolution. 

Other Considerations.

If you set out to assemble a Windows/PC based equivalent of the Apple station noted above, please keep in mind the following features automatically available in the Apple system.  You will need a high powered PC with a big power supply and plenty of cooling.  SATA serial drives are faster and best.  Your memory for each application is limited by Windows XP and a 32-bit processor to 2 GB until a formal release of XP-64 is available for use with an AMD F64 based processor or the future Intel equivalent.  DDR2 memory is used in the best PCs whereas only DDR memory is usable in the Power Macs.

Starting TNT products from a Portable Drive

If you are using Mac OS X 10.x (presumably the latest release) you can use a USB2.0, Firewire400, Firewire800, or cartridge hard drive or flash card as an installation drive for your TNT products making them physically portable across G4 and G5 based Macs.  This will permit you to move both TNTmips software and your preferences along with the USB TNT Software Authorization Key between your Apple portable, base system, classroom units, and so on. 

External tri-interface drives supporting USB2.0, Firewire400, and Firewire800 are now available and give the most portable drive flexibility. If the portable Firewire drive is an 800 instead of a 400 (800 connectors are only on the PowerMacs at this time), it will be just as fast as an internal drive.  On a USB2.0, Firewire400, or flash card, the startup and the loading of processes may be about ½ as fast or faster compared to the speed of an installation on an internal drive.  A faster result than ½ will depend upon the type of flash memory, USB2.0 support, and so on. 

A 1 GB USB2.0 thumb drive or memory stick could also be used to carry a TNT product and geodata around in your pocket between Apple systems that are not networked or can access a TNT floating license but do not have the TNT product locally installed.  The first thumb sized Firewire flash memory drives have now also appeared but are Firewire400 and, thus, perform at about the same speed as the much more widely available and cheaper USB2.0 drives.

The above portability idea does not work quite as well for Windows-based systems.  Without a direct install to the internal hard drive of each computer, it is unlikely that all the Windows libraries and many other factors will permit this portability.

Serial ATA drives.

Serial interface  SATA hard drives are now only 10% more expensive than IDE interface drives and are approaching cost parity with them.  You should make sure any new PC you buy uses these SATA drives, which are faster, more flexible in use, and will soon be cheaper than IDE drives.

Software News  

Further Confusion over Wavelet Compression.

LizardTech  vs. ER Mapper.

After an earlier apparent resolution by a Federal Judge of a LizardTech appeal, five years of litigation has just been resumed between LizardTech and ER Mapper with regard to their patent dispute over their proprietary wavelet compression products.  The dispute has focused upon how limited memory is managed when compressing large rasters.   When insufficient memory is available to hold the wavelet coefficients for the entire input raster because it is very large, the source image can be broken down and compressed in tiles.  If a large, lossy compression ratio is targeted and each tile is compressed separately, the slight differences in the lossy result can occur at the edges between these tiles and may be visible.  

An approach to the application of consistent wavelet compression to match the edges of a series of tiles is particularly significant when mosaicking many large pieces into an even larger mosaic.  For example, you might wish to mosaic hundreds of uncompressed or compressed orthoimages into a compressed province or national level image.  The management of these edge effects determines whether or not a large, uncompressed mosaic must be temporarily created on a hard drive before it can then be lossy compressed to 10 to 1 or 20 to 1 or more and if the large, temporary uncompressed intermediate image can indeed be compressed.

How to unify or mitigate this edge effect in the compression procedure is the subject of this legal dispute.  More information on this topic and the current position taken by www.lizardtech.com/press/news.php?item=11-01-2004a, and www.ermapper.com (...link obsolete...). Please note that LizardTech is now wholly owned by Celartem Technology USA, Inc., which is part of Celartem Technology, Inc. in  Japan

File formats can not be patented but the software used to create them may use patented or copyrighted techniques.  This is the basis for legal disputes when the software used to create the wavelet compressed file is deemed to be proprietary and using it is under control of a license.  LizardTech and ER Mapper create their own compression and decompression files in this manner that have no relationship to JP2 files and require a license from their patent holder to use the software they provide to perform the compression.  Thus, in these legal disputes, it becomes a matter of deciding if a patented technology has been used without a license to create their proprietary compressed formats in a competitor’s product.  

An internal TNT raster object is compressed into and out of JPEG2000 using the Kakadu library and not by proprietary wavelet code or methods created by MicroImages.  You can use these raster objects without a license in the TNTserver, the FREE TNTsim3D, and FREE TNTatlas or as external JP2 files in any manner you choose.  However, you do need a TNTmips, TNTedit, or TNTserver to create JPEG2000 compressed raster objects or JP2 files.

Proprietary Approaches Versus JPEG2000.

Since the release of RV6.9 of the TNT products in early 2004, both LizardTech and ER Mapper have announced the addition to their products of support for the creation and reading of JPEG2000 compressed Part 1 compliant JP2 files.  Their support of these JP2 files is being added in parallel to their disputed proprietary wavelet compressed file formats.  The question of how their support of this standard JPEG2000 JP2 file fits in with their proprietary file’s performance, legal claims, and marketability is best addressed to them. 

LizardTech is using the Kakadu library for this purpose.  This is the same library MicroImages selected for the first JPEG2000 features introduced in V6.7 2.5 years ago and continues to use.  ER Mapper’s approach to supporting JPEG2000 compression is unknown. The latest version of the Kakadu libraries supports a form of tiling during compression for memory management for very large images.  RV7.0 of the TNT products does not utilize this potentially contested procedure.  Using the amount of real memory common on your desktop computers (0.5 up to 2 gigabytes), large images up to 250 gigabytes can be JPEG2000 compressed without this tiling in the TNT products.  The MEMO entitled Release of the RV6.9 Products and dated 31 December 2003 discusses this JPEG2000 compression of large raster objects in detail in the section entitled JPEG2000 Compression.  This earlier MEMO can be reviewed in HTML or Microsoft Word formats at www.microimages.com/relnotes/v69/rel69.doc, and www.microimages. com/relnotes/v69/rel69.htm respectively.  How MicroImages will handle JPEG2000 compression of even larger images is being investigated.  An example of the need for this would be the assembly and compression of the 15-meter NASA Landsat imagery of entire continents for public distribution as a single compressed image.  

LuraTech located in Germany was heavily involved in the development of the JPEG2000 standard and has a licensed proprietary wavelet-based compression product called Lurawave Smart Compress, which creates files in the LuraTech Wavelet Format (*.lwf).  This proprietary format and compression library has been popular in Europe and also now licenses a toolkit for JPEG2000 software development.  In the following published paper Carsten Heiermann, President of LuraTech in  Germany concludes the following regarding the future of LuraTech’s proprietary compression.

“To close, I asked Heiermann to look ahead five years.  Would the company still be offering its proprietary solution?  He suggests that by then, there will be no new business in that area.  Even now, he notes, the company is not actively selling its proprietary solution.  Existing customers are moving to JPEG solutions, sometimes running both concurrently as they make the transition.  New customers invariably purchase the open standards-based solution over the proprietary one.”

Quoted from Image Compression Embraces Open Standards: A Conversation with Carsten Heiermann of LuraTech, by Andena Schutzberg, EOM magazine, November 2004, pp 24-26.   

Generic Requirements of JPEG2000.

The JPEG2000 specifications dictate how a JP2 file must be structured, but not how this is most efficiently accomplished.  The Kakadu library is one procedure for wavelet compression of rasters into a standard JP2 file.  There are other libraries and possibly other patented approaches used for this same purpose. Since a file format is not patentable, patent disputes over wavelet compression primarily focus upon the procedures and efficiencies employed in assembling and compressing JP2 files.

MicroImages and now LizardTech use the Kakadu library to create and read JPEG2000 compressed rasters.  Part 1 of the JPEG2000 specifications defines the features that can be incorporated into a stand alone JP2 file.  The Kakadu library insures that the JP2 file created in or exported from these and the TNT products meets the Part 1 compliance standard.  It is important to understand that a product can not claim to support the creation and/or reading of a Part 1 standard JP2 file unless it can be decompressed and used by other JPEG2000 Part 1 compliant products.  Variations are permitted by Part 1 depending upon the type and size of raster material used.  Even with this variability the JP2 file created using this permitted variability must still be useable in other products.  However, this does not guarantee how efficiently a Part 1 compliant program can read a specific JP2 file.  For example, the pyramid structure inherent in a Part 1 JP2 file may be ignored by the reading program making zoomed-out viewing quite slow because it is sampling.

Size Limitations of JP2 Files in Other Products. 

For faster operation many low-cost commercial products that do not use pyramiding are designed to work with full sized, uncompressed images stored directly in real memory and go really slow if virtual memory has to be substituted when the file size is large.  As a result, if these programs support using a JP2 image, they automatically decompress the full resolution image into real memory and then into virtual memory as needed.  The pyramid structure automatically built into JP2 rasters is simply ignored.  If the raster is small, this is not a problem.  However, this places limitations on how big a JP2 raster used in these products can be or how long it will take to decompress and load it.  Some simply will not load a JP2 image over a few megabytes uncompressed or automatically revert to virtual memory which, from a performance viewpoint, is extremely slow and more or less equivalent to not working.  Examples of products with these size limitations on using JP2 images would be Photoshop, QuickTime, and all known browser plug-ins.

LizardTech MrSID Compression to Be Supported in DV7.1.

As you will learn elsewhere in this MEMO, RV7.0 of the TNT products can now import or link to and directly use MrSID formatted files on Windows, Mac OS X, and Linux/UNIX.  Recently LizardTech and MicroImages have come to an agreement whereby the RV7.1 of TNT products will be able to export raster geodata into the MrSID format for use in the TNT products such as your FREE TNTatlases or in other software that can use MrSID files.  These MrSID files will be created with the accompanying georeference information in a companion world file (*.sdw) and/or embedded in the MrSID metadata.  

The integration of the LizardTech compression engine will be available in the early DV7.1 releases of TNTmips and TNTedit. There will be no extra charge by MicroImages for providing you access to this proprietary compression procedure beyond the charge to upgrade to RV7.1.  It is planned that it will be available for all TNT supported operating systems including Windows, Mac OS X, Linux, and Sun.  It may be likely that this may be the first product to create MrSID compressed files using the Mac OS X operating system.

Charges for MrSID Compression.

As you may already know, LizardTech’s stand alone GeoExpress mosaicking and compressor product and any other product, such as TNTmips, that provides compression into a  MrSID format, requires that you pay a per byte charge to LizardTech.  Since LizardTech does not charge anyone for decompressing and using compressed MrSID files, this is their mechanism for charging end users for using their proprietary compression engine.  This use fee is metered by a software data cartridge that you buy from LizardTech or one of its dealers. You purchase it with an amount of compression encoded into it.  You then install it and every compression into a MrSID file by your TNT product reduces the capacity of the meter in your MrSID data cartridge until it reaches zero or until you purchase and add additional capacity.  

GeoExpress permits a MrSID file to be read and recompressed into a new MrSID file without charging their data cartridge.  For example, you can convert a lossless MrSID file into a more compressed lossy MrSID file without any charge.  In contrast, as part of MicroImages’ agreement with LizardTech, if you choose MrSID files for input to a TNT process, such as mosaic, and designate MrSID as the output format, the data cartridge will be charged for its compression according to the decompressed size of the input objects you have selected.

Data cartridge metering is based on the uncompressed input bytes you send into the compression engine from the uncompressed equivalent of the TNT source raster object and not on the size of the compressed raster object or MrSID file.  Please keep in mind that a data cartridge is metering bytes and not raster cells, thus a 24-bit color composite image will use 3 times as many bytes as its pixel count.  Additional information about the operation of their data cartridge can be found at www.lizardtech.com/products/ geo/faq.php or can be addressed directly to LizardTech.  LizardTech does not directly publish the prices of GeoExpress or data cartridges, however you can get an idea of their prices from those charged to U.S. Government agencies from the General Services Agency (GSA) price schedule at www.saic-gsa.com  (...link obsolete...)

Charges for ECW Compression.

ER Mapper’s approach to compression into their wavelet based proprietary compression engine has exactly the opposite cost strategy.  A license for using this compression engine can be obtained free-of-charge for the use of this engine in other products.  This free license limits the compression to the input of single uncompressed rasters of 500 MB or less.  It is under this license that TNTmips and TNTedit can export and compress raster objects into the ECW format (*.ecw) within this size limitation. 

To compress rasters of greater than 500 MB uncompressed, a software developer must pay a large, upfront, one-time charge and large annual charge.  Both of these licensing charges can be found at www.ermapper.com/pricing.aspx.  Once their compression engine has been added to another software developer’s product, its end user can create ECW compressed files from uncompressed rasters greater than 500 MB without paying any per byte fee.

©MicroImages, Inc. 2010 Published in the United States of America
11th Floor - Sharp Tower, 206 South 13th Street, Lincoln NE 68508-2010   USA
Business & Sales: (402)477-9554  Support: (402)477-9562  Fax: (402)477-9559
Business info@microimages.com  Support support@microimages.com  Web webmaster@microimages.com

---