FGFS-4.1/OpenSceneGraph
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

From Wojiech Leandowski, "I earlier wrote about my hassles with archives under Windows. I implemented

Browse Source 
64 bit binary compatible OSGA archive reader/writer using mixed
stdio/iostream calls. But during this work I learned that it can be made in
much simpler way.

Attached is result of this new attempt. I hope its appropriate for inclusion
into OSG codebase. It was compiled and tested with latest SVN OSG, Windows
XP 32 bit and Windows Vista business 64 bit. OSG was built using VS 2005
Express SP1 for 32 bit environment and VS 2005 Std for 64 bit.
---
Solution description (there were two problems involved):
---
Problem 1: implicit conversions beetween file positions and 32 bit int. This
could be considered a MS compiler bug because this 32 bit int was
additionally implicitly converted to/from 64 bit. As far as I know compiler
is allowed to make only one implict conversion (but maybe this rule does not
refer to simple types).

Its actually possible to address OSGA files above 4 GiB range using 32 bit
windows iostreams. MS Iostreams in practice offer the same level of
functionality as stdio functions. There are functions fsetpos and fgetpos in
stdio lib which use 64 bit file pointers (fpos_t). These functions are
internally called by seekp( streampos ), seekg( streampos ), tellp(), and
tellg() methods. So its also possible to change and retrieve file postions
using iostream calls. But the problem lies in implicit handling of streampos
type.

streampos type is actually a template class used as seekp, seekg parameter
and returnd from tellp, tellg. Its capable of storing 64 bit file pointers.
But streampos can be also converted to/from simple type streamoff. It has
proper constructor and cast operator. In Win 32 environment streamoff is
defined as long (~32 bit int). So when seekp, and tellp arent used with
exact streampos objects but OSGA_Archive::pos_type complier makes implicit
casts to 32 bit int types loosing important bits of information.

So above problem could be easily handled by making conversion calls
explicit. My code defines 2 functions used to convert back and forth beetwen
64 bit OSGA_Archive::pos_type and std::streampos objects:

       OSGA_Archive::pos_type ARCHIVE_POS( const std::streampos & pos );
       std::streampos STREAM_POS( OSGA_Archive::pos_type & pos );

Rest of the OSGA implementation code was modified to call these conversions
explicitly with seekp, seekg, tellp, tellg.

---
Problem 2: seekp and seekg have two variants. Only one of these variants is
actually 64 bit proof.

When I solved my first problem and made use of explicit streampos conversion
functions, OSGA archive was able to read my example 11 GiB archive. But
there were still problems with write and append. I found that the reason for
this was pair of seekp( 0, std::ios_base::end ) and tellp() calls. It turned
out that use of seekp, seekg( offset, direction ) function variants was
setting file pos pointer to EOF when file was larger than 4GiB. But I
noticed that one arg seekp, seekg ( streampos ) versions worked correctly.
So the solution was to change OSGA write logic a little, and replace
seekp( offset, direction ) with seekp( absolute_pos ) calls.
I achieved this by modifing IndexBlock write method to record and restore
file pos after IndexBlock was written. This modification has the effect that
put pointer is generally kept at the end of file, so there is no need to
repostion to the end before writing the files. This allowed me to get rid of
those problematic seekp( 0, std::ios_base::end ) calls.

There was one place where I could not easily get rid of seekp( 0,
std::ios_base::end ). It was situation where existing OSGA was opened for
appending. I resolved this by computing file length by finding max position
from index block and file block endings. Then I replaced former seekp( 0,
std::ios_base::end ) with  seekp( STREAM_POS( found_file_length ).
---

Description of these changes may sound bit hacky but in practice these were
fairly simple and straightforward modifications. I hope they pass your
review. There is one complex preprocessor condition which I based on few
lines taken from boost positioning.hpp. Boost licence does allow such
reproduction. In case of problems this condition may be easily simplified to
windows only implementation.
"
This commit is contained in:
Robert Osfield 2007-12-17 09:58:55 +00:00
parent 3fcba1bc51
commit 9507ee6c7d
1 changed files with 135 additions and 27 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

162
src/osgPlugins/osga/OSGA_Archive.cpp
View File

@ -8,6 +8,90 @@
using namespace osgDB;
/*
Functions to convert between
std::streampos ( typedef'ed as iostream::pos_type
used as seekp,seekg argument and tellp,tellg return value )
and
OSGA_Archive::pos_type (64 bit file position index)
Purpose:
To allow using OSGA files larger than 4GiB in Windows.
std::streampos is used as argument to iostreams seekp and seekg methods
and is returned as result from iostream tellp and tellg methods.
std::streampos can be implicitly cast from/to std::streamoff as
std::streampos class defines appropriate constructor and cast operator.
Since std::streamoff is usually defined as simple int,
it is possible to call seekp( ), seekg( ) with int argument and
assign tellp(), tellg() result to int type.
But this implicit methods fail when std::streamoff is 32 bit and
std::streampos actually points past 32 bit addressable range (2 GiB).
Even if std::streamoff is 32 bit and incapable of representing 64 bit file
positions, original std::streampos may be prefectly able to handle them.
But, when such situation occurs more elaborate conversion methods from/to
std::streampos are needed. Functions below employ such methods.
I made this fix for use with 32 bit Windows OSG. Acutally this
solution is not dependent on OS but standard C++ library.
Microsoft SDKs always use some version of Dinkumware libs.
Practically this tweak is made for Dinkumware C++ libs. I hope it can
be easily extended to other 32bit systems supporting 64bit files, provided
their std::streampos implementations are similar.
I based my solution on a small portion of boost iostreams code.
For additional reference look at:
http://boost.org/boost/iostreams/positioning.hpp
*/
/*
Recognize Dinkumware std C++ lib implementation. Its used by Microsoft,
but method is more generic - should work in all Dinkumware environments.
Complex condition below was taken from
http://boost.org/boost/iostreams/positioning.hpp
Great thanks to J.Tukanis and G. Sylvester-Bradley for figuring it out.
*/
#if ((defined(_YVALS) && !defined(__IBMCPP__)) || defined(_CPPLIB_VER)) && \
!defined(__SGI_STL_PORT) && !defined(_STLPORT_VERSION) \
&& !defined(__QNX__)
inline std::streampos STREAM_POS( const OSGA_Archive::pos_type pos )
{
return std::streampos( std::mbstate_t(), pos );
}
inline OSGA_Archive::pos_type ARCHIVE_POS( const std::streampos & pos )
{
#if defined(_CPPLIB_VER)//newer Dinkumware(eg: one included with VC++ 2003,2005)
fpos_t position = pos.seekpos();
#else // older Dinkumware (eg: one included in Win Server 2003 Platform SDK )
fpos_t position = pos.get_fpos_t();
#endif
std::streamoff offset = pos.operator std::streamoff( ) - _FPOSOFF( position );
return OSGA_Archive::pos_type( position + offset );
}
#else // non Dinkumware std C++ lib implementations
// do the old school streampos <-> streamoff casts
inline std::streampos STREAM_POS( const OSGA_Archive::pos_type pos )
{
return std::streampos( pos );
}
inline OSGA_Archive::pos_type ARCHIVE_POS( const std::streampos & pos )
{
return OSGA_Archive::pos_type( pos );
}
#endif // Dinkumware std C++ lib
////////////////////////////////////////////////////////////////////////////////
float OSGA_Archive::s_currentSupportedVersion = 0.0;
const unsigned int ENDIAN_TEST_NUMBER = 0x00000001;
@ -49,7 +133,7 @@ OSGA_Archive::IndexBlock* OSGA_Archive::IndexBlock::read(std::istream& in, bool
if (!in) return 0;
osg::ref_ptr<IndexBlock> indexBlock = new IndexBlock;
indexBlock->_filePosition = in.tellg();
indexBlock->_filePosition = ARCHIVE_POS( in.tellg() );
in.read(reinterpret_cast<char*>(&indexBlock->_blockSize), sizeof(indexBlock->_blockSize));
in.read(reinterpret_cast<char*>(&indexBlock->_filePositionNextIndexBlock), sizeof(indexBlock->_filePositionNextIndexBlock));
in.read(reinterpret_cast<char*>(&indexBlock->_offsetOfNextAvailableSpace), sizeof(indexBlock-> _offsetOfNextAvailableSpace));
@ -162,22 +246,27 @@ bool OSGA_Archive::IndexBlock::getFileReferences(FileNamePositionMap& indexMap)
void OSGA_Archive::IndexBlock::write(std::ostream& out)
{
pos_type currentPos = ARCHIVE_POS( out.tellp() );
if (_filePosition==pos_type(0))
{
osg::notify(osg::INFO)<<"OSGA_Archive::IndexBlock::write() setting _filePosition"<<std::endl;
_filePosition = out.tellp();
_filePosition = currentPos;
}
else
{
out.seekp(_filePosition);
out.seekp( STREAM_POS( _filePosition ) );
}
osg::notify(osg::INFO)<<"OSGA_Archive::IndexBlock::write() to _filePosition"<<out.tellp()<<std::endl;
osg::notify(osg::INFO)<<"OSGA_Archive::IndexBlock::write() to _filePosition"<< ARCHIVE_POS( out.tellp() )<<std::endl;
out.write(reinterpret_cast<char*>(&_blockSize), sizeof(_blockSize));
out.write(reinterpret_cast<char*>(&_filePositionNextIndexBlock), sizeof(_filePositionNextIndexBlock));
out.write(reinterpret_cast<char*>(&_offsetOfNextAvailableSpace), sizeof(_offsetOfNextAvailableSpace));
out.write(reinterpret_cast<char*>(_data),_blockSize);
if( _filePosition < currentPos ) // move file ptr to the end of file
out.seekp( STREAM_POS( currentPos ) );
osg::notify(osg::INFO)<<"OSGA_Archive::IndexBlock::write() end"<<std::endl;
}
@ -251,17 +340,44 @@ bool OSGA_Archive::open(const std::string& filename, ArchiveStatus status, unsig
{
if (status==WRITE && open(filename,READ))
{
pos_type file_size( 0 );
_input.seekg( 0, std::ios_base::end );
file_size = ARCHIVE_POS( _input.tellg() );
if( _input.is_open() && file_size <= 0 )
{ // compute end of file postition manually ...
// seekp( 0, ios::end ), tellp( ) fails in 32 bit windows with files > 4 GiB
size_t BlockHeaderSize =
sizeof( unsigned int /*_blockSize*/ ) +
sizeof( pos_type /*_filePositionNextIndexBlock*/ ) +
sizeof( unsigned int /*_offsetOfNextAvailableSpace*/ );
for(IndexBlockList::iterator itr=_indexBlockList.begin();
itr!=_indexBlockList.end();
++itr)
{
pos_type end = (*itr)->getPosition() + BlockHeaderSize + (*itr)->getBlockSize();
if( file_size < end ) file_size = end;
}
for(FileNamePositionMap::iterator mitr=_indexMap.begin();
mitr!=_indexMap.end();
++mitr)
{
pos_type end = mitr->second.first + mitr->second.second;
if( file_size < end ) file_size = end;
}
}
_input.close();
_status = WRITE;
_output.open(filename.c_str(), std::ios_base::binary | std::ios_base::in | std::ios_base::out);
osg::notify(osg::INFO)<<"File position after open = "<<(int)_output.tellp()<<" is_open "<<_output.is_open()<<std::endl;
// place write position at end of file.
_output.seekp(0, std::ios::end);
osg::notify(osg::INFO)<<"File position after open = "<<ARCHIVE_POS( _output.tellp() )<<" is_open "<<_output.is_open()<<std::endl;
osg::notify(osg::INFO)<<"File position after seekp = "<<(int)_output.tellp()<<std::endl;
// place write position at end of file.
_output.seekp( STREAM_POS( file_size ) );
osg::notify(osg::INFO)<<"File position after seekp = "<<ARCHIVE_POS( _output.tellp() )<<std::endl;
osg::notify(osg::INFO)<<"OSGA_Archive::open("<<filename<<") open for writing"<<std::endl;
@ -284,12 +400,7 @@ bool OSGA_Archive::open(const std::string& filename, ArchiveStatus status, unsig
_indexBlockList.push_back(indexBlock);
}
osg::notify(osg::INFO)<<"File position after write = "<<(int)_output.tellp()<<std::endl;
// place write position at end of file.
_output.seekp(0,std::ios::end);
osg::notify(osg::INFO)<<"File position after seekp = "<<(int)_output.tellp()<<std::endl;
osg::notify(osg::INFO)<<"File position after write = "<<ARCHIVE_POS( _output.tellp() )<<std::endl;
return true;
}
@ -337,7 +448,7 @@ bool OSGA_Archive::_open(std::istream& input)
_indexBlockList.push_back(indexBlock);
if (indexBlock->getPositionNextIndexBlock()==pos_type(0)) break;
input.seekg(indexBlock->getPositionNextIndexBlock());
input.seekg( STREAM_POS( indexBlock->getPositionNextIndexBlock() ) );
}
// now need to build the filename map.
@ -464,7 +575,7 @@ bool OSGA_Archive::addFileReference(pos_type position, size_type size, const std
// if not one available create a new block.
if (!indexBlock)
{
if (previousBlock.valid()) previousBlock->setPositionNextIndexBlock(_output.tellp());
if (previousBlock.valid()) previousBlock->setPositionNextIndexBlock( ARCHIVE_POS( _output.tellp() ) );
indexBlock = new IndexBlock(blockSize);
indexBlock->write(_output);
@ -549,7 +660,7 @@ ReaderWriter::ReadResult OSGA_Archive::read(const ReadFunctor& readFunctor)
if (_status!=READ)
{
osg::notify(osg::INFO)<<"OSGA_Archive::readObject(obj, "<<readFunctor._filename<<") failed, archive opened as read only."<<std::endl;
osg::notify(osg::INFO)<<"OSGA_Archive::readObject(obj, "<<readFunctor._filename<<") failed, archive opened as write only."<<std::endl;
return ReadResult(ReadResult::FILE_NOT_HANDLED);
}
@ -563,15 +674,15 @@ ReaderWriter::ReadResult OSGA_Archive::read(const ReadFunctor& readFunctor)
ReaderWriter* rw = osgDB::Registry::instance()->getReaderWriterForExtension(getLowerCaseFileExtension(readFunctor._filename));
if (!rw)
{
osg::notify(osg::INFO)<<"OSGA_Archive::readObject(obj, "<<readFunctor._filename<<") failed to find appropriate plugin to write file."<<std::endl;
osg::notify(osg::INFO)<<"OSGA_Archive::readObject(obj, "<<readFunctor._filename<<") failed to find appropriate plugin to read file."<<std::endl;
return ReadResult(ReadResult::FILE_NOT_HANDLED);
}
osg::notify(osg::INFO)<<"OSGA_Archive::readObject(obj, "<<readFunctor._filename<<")"<<std::endl;
_input.seekg(itr->second.first);
_input.seekg( STREAM_POS( itr->second.first ) );
// set up proxy stream buffer to proide the faked ending.
// set up proxy stream buffer to provide the faked ending.
std::istream& ins = _input;
proxy_streambuf mystreambuf(ins.rdbuf(),itr->second.second);
ins.rdbuf(&mystreambuf);
@ -663,15 +774,12 @@ ReaderWriter::WriteResult OSGA_Archive::write(const WriteFunctor& writeFunctor)
osg::notify(osg::INFO)<<"OSGA_Archive::write(obj, "<<writeFunctor._filename<<")"<<std::endl;
// place write position at end of file.
_output.seekp(0,std::ios::end);
pos_type position = _output.tellp();
pos_type position = ARCHIVE_POS( _output.tellp() );
WriteResult result = writeFunctor.doWrite(*rw,_output);
pos_type final_position = _output.tellp();
size_type size = size_type(final_position-position);
pos_type final_position = ARCHIVE_POS( _output.tellp() );
size_type size = size_type( final_position-position );
if (result.success()) addFileReference(position, size, writeFunctor._filename);