Visualizing NetCDF files, that are bigger than system RAM

  6 posts   Feedicon  
Replies: 5 - Last Post: August 02, 2010 00:02
by: Bergtroll
showing 1 - 6 of 6
 
Posted: July 14, 2010 11:18 by Bergtroll

I shall try to visualize some of the measuring results of the Coltrims experiment (for your interest: http://hsb.uni-frankfurt.de/web/research/atomic/coltrims/). Unfortunately the data files come in chunks of more than 10 GB each. I found that there are classes called VirtualData, FileFlatField, etc. and hope, that my guess is right. Are they are somehow intended to be used for accessing datasets that do not fit into memory? How do I utilize them for fast retrieving of slices in dimension of 500MB? I do not have the heart yet trying to access one of those files, because I did have not really understood, how to use them, and I do not want to freeze my system instantly :-D

 
Posted: July 14, 2010 22:04 by Bergtroll

Quoted from VisAD mailing list, Answer from Tom Rink, 14.07.2010, 15:36

I'll try to get you some specifics on what was done with caching, our goal is to be able to animate perhaps several hundred, roughly 1000x1000, satellite images. Of course, caching saves memory, but is performance hit.

Tom


 
Posted: July 14, 2010 22:12 by Bergtroll

Quoted from VisAD mailing list, Answer from Bill Hibbard, 14.07.2010, 14:50

Jeff McWhirter recently created the CachedFlatField and related "Cache" classes, that I think are better than the olf FileFlatField. Hopefully he or someone can comment on these.


 
Posted: July 23, 2010 19:26 by Bruce Schubert
Were you successful in your attempt to visualize the large NetCDF files? I'm curious because I do not have any experience NetCDF files yet. But I've considering them for storing and visualizing wildland fire behavior data -- due to their use with weather data which is also temporal and spatially oriented.

-- Bruce
 
Posted: August 02, 2010 00:02 by Bergtroll

Hi Bruce,

I did not found the time to give it a try yet. If I have something usefull, I will tell :-)

Kind regards, Marius

 
Posted: July 20, 2010 08:38 by Bergtroll

Quoted from VisAD mailing list, Answer from Jeff McWhirter, 16.07.2010, 16:38

We've done a number of things with caching. We use the class: '''visad.data.DataCacheManager''' as a singleton to manage the cache. It can 1d and 2d arrays of all of the primitive types. The CachedFlatField just uses the >DataCacheManager to store its range values. The DCM will hold the arrays in memory until a memory limit is exceeded then it writes the arrays out to disk and clears them from memory (least recently used strategy).

The first thing you need to do is to set the directory to cache to, e.g., from the IDV: visad.data.DataCacheManager.getCacheManager().setCacheDir( new File(getDataCacheDirectory()));

And set the memory threshold: visad.data.DataCacheManager.getCacheManager() .setMemoryPercent(dataContext.getIdv().getStateManager() .getPreferenceOrProperty(PROPCACHEPERCENT, 0.25));

Then just create the CachedFlatField like a regular FlatField. The IDV does some tricky things with delayed data reading. >See, for example, ucar.visad.data.GeoGridFlatField to see how this is used.

Using the CachedFlatField allows you to, for example, have lengthy time series of grids or images. If you do a lazy loading >(like the IDV does) then you build up your FieldImpl of empty CachedFlatFields and pass it to the renderer. When a time step is rendered then the data is read. As the cache fills up the arrays are dumped to disk in case of subsequent accesses.

Another way we use this is with the visad.data.CachedBufferedByteImage. This is a BufferedImage but plays tricks with the >base class so the actual raster data is stored by the CacheManager. This is used by visad.java3d.VisADImageTile to render images. That way the image data in the scene graph is transparently cached though when animating and if your cached image is on disk it can take some time to load.

To enable this the system property visad.java3d.imageByRef needs to be set to true.

-Jeff


showing 1 - 6 of 6
Replies: 5 - Last Post: August 02, 2010 00:02
by: Bergtroll
  • Mysql
  • Glassfish
  • Jruby
  • Rails
  • Nblogo
Terms of Use; Privacy Policy;
© 2014, Oracle Corporation and/or its affiliates
(revision 20150626.29986a4)
 
 
Close
loading
Please Confirm
Close