First Submission email from Gustav:
"This submission adds a --cache option to osgconv and osgviewer that enables setObjectCacheHint(osgDB::Options::CACHE_ALL); It greatly reduces memory usage when a .osg file has lots of external references with ProxyNode:s that points to the same file.
Options are also added to the osg plugin. The code was already mostly implemented but there was no way to change the options.
includeExternalReferences
writeExternalReferenceFiles
A counter is added to keep track if an external file has already been written down to avoid writing the same file over and over again. If it has already been written once then it is not written again.
The counter is added to the Output class in osgDB.
"
Second Submission email from Gustav:
"This is a continuation to my previous submission.
I noticed that the same problem that I fixed in ProxyNode.cpp for the osg plugin (external files being written over and over again) also existed in the ive plugin. I attached a submission where the ive plugin remembers which external files that have already been written and do not write them again."
Changes to the above done by Robert Osfield,
changed command line parameter to --enable-object-cache
changed set/get methods in osgDB::Output and ive/DataOutputStream.cpp to be s/getExternalFileWritten(const std::string&)
cleaned up set up of osgDB::Options.
Introduced a new callback osgDB::FindFileCallback that overrides the default behavior of findDataFile/findLibraryFile.
Introduced support for assigning ReaderWriter::Options directory to PagedLOD.
Introduced new osgDB::FileLocationCallback for assistancing the DatabasePager to know when a file is hosted on a local or remote file system.
Introduced a new FindFileCallback to Registry to compliement the existing ReadFileCallback and WriteFileCallback.
Added support for assign Find, Read and WriteFileCallbacks to osdDB::Options to enable plugins/applications to override the callbacks just for that
read/write call and any nested file operations
Converted Present3D across from using libxml2 to using the new osgDB::XmlNode/XmlNode::Input classes from Xml Parsing.
This changes removes the dependency on libxml2, and allows the present3D application and p3d to work on all platforms.
(http://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg23098.html)
Background: when you access a file over HTTP, you cannot rely on a file extension being present; instead the file's mime-type is conveyed in the HTTP Content-Type response header. This facility adds a mime-type-to-extension map to the registry to handle this.
There are two new osgDB::Registry functions which are pretty self-explanatory:
void addMimeTypeExtensionMapping( mime-type, extension )
ReaderWriter* getReaderWriterForMimeType( mime-type )
I also added the file osgDB/MimeTypes.cpp which houses a hard-coded list of built-in types. I took the list from here (http://www.webmaster-toolkit.com/mime-types.shtml) and then pared it down to include mostly image and video types, editing them to map to existing plugins where possible.
In addition, I updated the CURL plugin to a) install a set of built-in mime-type mappings, and b) use them to look up an extension in the event that the target filename does not have an extension.
Here is a test case. This URL pulls down a JPEG (without a file extension):
osgviewer --image "http://us.maps3.yimg.com/aerial.maps.yimg.com/ximg?v=1.8&s=256&t=a&r=1&x=0&y=0&z=2"
"
Single features are implemented as bits asother enums in ReaderWriter class already does, so that combinations are possible and fast comparison operations are possible
By default all features are enabled.
I have added this virtual method to the dot plugin to get an idea how to use these features.
With this patch osgconv --formats shows an additional line 'features' for each plugin"
"Attached you'll find a proposal for using different
protocols. The idea behind the new code is:
1.) plugins/apps register protocols which they can handle. This is done
via osgDB::Registry::registerProtocol(aProtocolName). Plugins register
supported protocols as usual via ReaderWriter::supportsProtocol(..), the
Registry is updated accordingly.
2.) osgDB::containsServerAddress checks first for an appearance of "://"
in the filename and then checks the protocol against the set of
registered protocols via Registry::isProtocolRegistered(aProtocollName)
3.) the other getServer*-functions changed as well, there's even a
getServerProtocol-function
With these changes filenames/Urls get routed to loaded plugins even with
different protocols than 'http'."
a capping the number of PagedLOD to a sepcified maximum, with pruning of inactive
PagedLOD when the total number of inactive and active PagedLOD goes above the maximum.
To enable the mode set the env var OSG_MAX_PAGEDLOD to a value something like 1000.
New attribute DatabasePager::_expiryFrames sets number of frames a PagedLOD child is kept in memory. The attribute is set with DatabasePager::setExpiryFrames method or OSG_EXPIRY_FRAMES environmental variable.
New attribute PagedLOD::PerRangeData::_
frameNumber contains frame number of last cull traversal.
Children of PagedLOD are expired when time _AND_ number of frames since last cull traversal exceed OSG_EXPIRY_DELAY _AND_ OSG_EXPIRY_FRAMES respectively. By default OSG_EXPIRY_FRAMES = 1 which means that nodes from last cull/rendering
traversal will not be expired even if last cull time exceeds OSG_EXPIRY_DELAY. Setting OSG_EXPIRY_FRAMES = 0 revokes previous behaviour of PagedLOD.
Setting OSG_EXPIRY_FRAMES > 0 fixes problems of children reloading in lazy rendering applications. Required behaviour is achieved by manipulating OSG_EXPIRY_DELAY and OSG_EXPIRY_FRAMES together.
Two interface changes are made:
DatabasePager::updateSceneGraph(double currentFrameTime) is replaced by DatabasePager::updateSceneGraph(const osg::FrameStamp &frameStamp). The previous method is in #if 0 clause in the header file. Robert, decide if You want to include it.
PagedLOD::removeExpiredChildren(double expiryTime, NodeList &removedChildren) is deprecated (warning is printed), when subclassing use PagedLOD::removeExpiredChildren(double expiryTime, int expiryFrame, NodeList &removedChildren) instead. "
multi-threaded paging, where the Pager manages threads of reading local
and http files via seperate threads. This makes it possible to smoothly
browse large databases where parts of the data are locally cached while
others are on a remote server. Previously with this type of dataset
the pager would stall all paging while http requests were being served,
even when parts of the models are still loadable virtue of being in the
local cache.
Also as part of the refactoring the DatabaseRequest are now stored in the
ProxyNode/PagedLOD nodes to facilitate quite updating in the cull traversal,
with the new code avoiding mutex locks and searches. Previous on big
databases the overhead involved in make database requests could accumulate
to a point where it'd cause the cull traversal to break frame. The overhead
now is negligable.
Finally OSG_FILE_CACHE support has been moved from the curl plugin into
the DatabasePager. Eventually this functionality will be moved out into
osgDB for more general usage.
ReaderWriter::ReadResult now has a FILE_REQUEST enum.
ReaderWriter::Options now has a s/getAsynchronousFileReadHint() parameter methods.
libcurl based plugin now detects enabing of the AsynchronousFileReadHint, but
as yet does not handle async requests - handling everything syncronously.
DatabasePager now by default will enable AsynchronousFileReadHint for http
based file requests
from: DEEP_COPY_STATESETS = 8,
to: DEEP_COPY_STATESETS = 1<<3,
showing clearly that this isn't the _value_ 8, but the _bit_ 8. this is an old pattern i see (and like to promulgate) to make code a bit more readable and maintainable.
"
as not expiring subgraphs quick enough to enable reasonable load balancing.
New version isn't perfect and will need further work, but does at least reduce
the memory footprint by as much as half on test paths on big databases.
The rewritten method no longer uses the the MaximumNumOfRemovedChildPagedLODs
and MinimumNumOfInactivePagedLODs variables so these and associated methods
for accessing them have been removed.
- /** Set the maximum number of PagedLOD child to remove per frame */
- void setMaximumNumOfRemovedChildPagedLODs(unsigned int number) { _maximumNumOfRemovedChildPagedLODs = number; }
-
- /** Get the maximum number of PagedLOD child to remove per frame */
- unsigned int getMaximumNumOfRemovedChildPagedLODs() const { return _maximumNumOfRemovedChildPagedLODs; }
-
- /** Set the minimum number of inactive PagedLOD child to keep */
- void setMinimumNumOfInactivePagedLODs(unsigned int number) { _minimumNumOfInactivePagedLODs = number; }
-
- /** Get the minimum number of inactive PagedLOD child to keep */
- unsigned int getMinimumNumOfInactivePagedLODs() const { return _minimumNumOfInactivePagedLODs; }
in the DatabasePager. The practical effects of these are to greatly reduce startup time
and the time to load an individual scenery tile in FlightGear.
- From my log message:
Minimize the number of StateSets and drawables that are compiled by checking
if they have already been compiled or will be elminated by the
SharedStateManager.
Move the sorting of the dataToCompile queue out of compileGLObjects
into the man pager run function.
Change the SharedStateManager to use maps instead of vectors."
- Get or set the target number of PagedLOD children to remove per frame.
- Get or set the minimum number of inactive PagedLOD to keep.
Corresponding environment variables have been added too.
The default values reproduce the previous DatabasePager behavior."
I've done some additional small modification regarding constness in ReaderWriter and added
mutable on _pluginData so passing data back would be possible too.
Have updated the collada plugin (ReaderWriterDAE.cpp) to use the map to handle options and
have attached the changes.
The stuff in daeReader.h and daeWriter.h are just cosmetic changes to get rid of a warning."
Added and cleaned up DeleteHandler calls in osgViewer to help avoid crashes on exit.
Changed DatabasePager across to dynamically checcking osg::getCompileContext(..)
Updated wrappers.
"By repurpose, I'm creating a new plugin that uses much of the .osg fileformat,
but with some changes. Specifically, I'm creating a ".osgfs" plugin, which
represents the scenegraph hierarchy as a filesystem of nested subdirectories and
individual files for each node, rather than nested braces with everything in a
single monolithic file. I intend to incorporate file alteration monitor events
to watch the filesystem for modifications and automatically reload.
The problem I'm running into is osgDB is too tightly coupled to the .osg format.
osgDB::Output::writeObject() contains literal .osg format-specific strings like
"{" to represent the Object hierarchy at too low a semantic level. I propose
using virtual methods; my plugin can then derive from osgDB::Output and
represent Object hiearchy differently.
"
~~~~~~~~~~~~~~~~~~~~~~~~~
This is a simple change to permit databases other than those named
"*.osga" to be used. It is hardcoded in read() at present.
It is non-critical and does not affect existing program functionality.
Registry and Registry.cpp
~~~~~~~~~~~~~~~~~~~~~~~~~
Added a new typedef: typedef std::vector< std::string>
ArchiveExtensionList;
a list of extensions: ArchiveExtensionList _archiveExtList;
and an "add" method: addArchiveExtension(const std::string ext)
This is initialised by adding "osga" in Registry() and used in
Registry::read() where the list is searched for the extension used.
Archive.cpp
~~~~~~~~~~~
This submission is a little more tentative. openArchive() is modified to
automatically add the filename extension to the Registry extension list.
"
To set up extension aliases using a config file, an app calls:
osgDB::Registry::instance()->readPluginAliasConfigurationFile(), passing in the file name as the parameter. (Of course this should be done before loading any files whose names depend on the mapping.) osgDB will search for the file using OSG_FILE_PATH.
The file should contain a line for each mapping, with the "map" extension first, followed by a space or tab, then the plugin identifier. For example, a file containing this line:
flt OpenFlight
would map the ".flt" extension to the OpenFlight plugin."
copy constructors that take a const osg::CopyOp rather than a
osg::CopyOp&, forcing an unnecessary copy. The attached header fixes
this, based off OSG 1.2.
Also fixed duplicate "or" openArchive()'s comment."
Tweaks by Robert Osfield - Moved methods to FileNameUtils, added unix
implementation of getRealPath add extern and OSGDB_EXPORT to function declarations.
default Registry Options object, but it would be useful to be able to
request loading with a different Options object. The attached files
allow you to do that (based off the OSG 1.2 source).
For example, I'm implementing a loader that requires context information
when it pages in subgraphs, which becomes significantly complicated when
multiple scenegraphs are requesting subgraph loads with different
contexts (the loader needs to know which context to use, and the
Registry Options needs to be carefully managed so the context settings
don't clobber each other, especially in multithreaded situations). Being
able to pass an Options instance along with the Node request resolves
this problem."
"My patch is a slight refactoring of the mac specific code in
Registry.cpp and FileUtils.cpp, specifically around the library and
resource file path initilialization methods. This patch cleans up a
lot of the mac specific code by moving repeated code into separate
local functions in FileUtils.cpp that are only compiled on mac builds.
It also adds one function to the API,
appendPlatformSpecificResourceFilePaths in FileUtils. This function
will mirror the already existing
appendPlatformSpecificLibraryFilePaths except for resource file paths.
Currently this function is empty except when built on the mac, in
which case it will add the application bundle's internal Resources
folder and the bundle's parent folder. Previously this code was
implemented as a separate mac specific #ifdef block in Registry.cpp
around the initDataFilePathList method. However, it now is implemented
in appendPlatformSpecificResourceFilePaths in FileUtils.cpp and the
initDataFilePathList method is now the same on all platforms. This
patch should behave the same as before on non-mac platforms.
This patch already includes the fix that Eric mentioned earlier. This
patch is based off of the 0.99 release code. I have tested this patch
using the following testing scheme:
Make a proper bundled application.
While Run from the Finder:
Test that it finds plugins in its internal plugins path.
Test that it finds resources in its internal resources path.
Test that it finds resources in the bundle's parent directory
Test that it finds plugins in the user's Application Support Directory
Test that it finds plugins in the system's Application Support Directory
Test that it finds plugins in the Network Application Support Directory
Check the plugin and resource path lists after they have been
initialized to see if they are in the correct order
While Run from the command line (both from it's parent directory and
from inside the /Contents/MacOS directory) and repeat the above tests.
Check that it also finds plugins and resources within the paths
defined by various environment variables.
Now, Make an application that is NOT bundled/command line only
Test that it does NOT try to look in an internal bundle
plugin/resource directory for plugins or resources.
Test that it finds plugins/resources in the paths defined by the
environment variables.
"
Drawables,StateSet, and osgDB::Registry.
Added cleanup_frame() from to osgProducer::OsgCamerGroup to help with proper
clean of OpenGL objects before exit, and modified osgviewer, osghangglider,
osgwindows examples to do the extra frame call to cleanup_frame() before exit.
much time is allocated to compiling and deleting OpenGL objects, also
added support into osgProducer::OsgSceneHandler.cpp for these new parameters.
The new cotrols are:
DatabasePager::s/getTargetFrameRate(..)
DatabasePager::s/getMinimumTimeAvailableForGLCompileAndDeletePerFrame()
DatabasePager::s/getMaximumNumOfObjectsToCompilePerFrame()
