Attached is a fixed version of OverlayNode.cpp. I fixed CustomPolytope::cut( osg::Plane ) method. Bug was apparent in such scenario:
Let P1 be some random frustum polytope
Let P2 be the polytope that was created from P1 bounding box (P2 contains P1 entirely)
Then ignoring precision errors: P1.cut( P2 ) == P2.cut( P1 ) == P1. But this condition was not always met. Cut failed when some of the polytope reference points happened to lie exactly on some intersecting planes in both P1 & P2 (plane distance was = 0).
I only use CustomPolytope for my shadowing stuff so I did not test how this affects rest of OverlayNode.cpp.
----2----
Also attached is a minor precision improvement for osg::Plane intersect method (double version).
----3----
I have also one observation regarding osg::Plane - There are two intersect vertices methods (float and double flavour):
inline int intersect(const std::vector<Vec3>& vertices) const
inline int intersect(const std::vector<Vec3d>& vertices) const
I guess osg::Plane won't compile when someone changes default vec3 typedef to vec3d. Shouldn't the first method be changed to use vec3f explicitly ? Ie:
inline int intersect(const std::vector<Vec3f>& vertices) const"
Performance tests on big models did not indicate any performance penalty in using doubles over floats,
so the move to doubles should mainly impact precision improvements for whole earth databases.
Also made improvements to osgUtil::PlaneIntersector and osgSim::ElevationSlice classes
"I've added a Plane constructor which accepts a normal and a point.
I also removed calculateUpperLowerBBCorners() from the Plane(const
Vec3& v1, const Vec3& v2, const Vec3& v3) since the constructor is
using the function set(const Vec3& v1, const Vec3& v2, const Vec3& v3)
which already computes the upper and lower bounding box."
