Dynamic Loading of Ogre Resources
This article was written in March 2009 and relates to OGRE 1.6.1 [Shoggoth]. Maybe some of this is changed or improved in later versions. Updated 20090620 with discussion and code on reloading of textures.
I am working on a project which uses Ogre as the 3D renderer of objects and textures fetched from a server. Ogre, though, is built around a game design workflow and the ability to dynamically load and insert objects into an Ogre rendered scene is not obvious.
This article describes my solution. I will put some code which is a chopped up version of my running code. Any code on this page is licensed under the BSD license. If you have any questions, you can contact me at misterblue at misterblue dot com.
Default Ogre Resource Workflow
Ogre is designed around preloaded resources. The usual application is a game that has been authored and is loaded onto the computer. All of the Ogre resources (meshes, textures, materials, …) are on the disk and are read and indexed at application load time. The resources.cfg file lists the directories to find resources and Ogre::ResourceGroupManager::InitializeAllResourceGroups() causes all of the resources to be read in and initialized. There after, when a mesh is needed, for instance, Ogre::MeshManager is called to load the mesh and, viola, the mesh is available to be rendered. (Materials are a little different which is discussed below).
Dynamic Resource Loading
My application has nothing preloaded. resources.cfg points only to the barest of directories which contain only default textures and materials. The application connects to a server and starts receiving meshes and object locations for display. As objects are received, I need to hook the mesh into the scene graph and load the materials and textures needed to display that mesh. So, for my application, I have to handle complete dynamic loading of Ogre resources.
Looking over the forums, the power user solution to this problem is to edit the Ogre sources and replace the existing resource manager. This creates a custom version of Ogre that perfectly fits the required solution.
I wished to use an unmodified Ogre binary so I set out to find a solution that does not require rebuilding Ogre. My design is: when Ogre notices it does not have a necessary resource, it displays a default resource while requesting the real resource. When the real resource is available, Ogre is made to reload the resource and thus start displaying the real shape/texture/… I also chose to cache everything on disk. It is a “simple matter of programming” to add some database storage system if needed.
Meshes
Meshes are pretty easy. I defined my own resource group and added an archive class to catch the file reads. Since, in my application, meshes (displayed objects) are pushed into the scene, I can declare the mesh and thus cause Ogre::MeshManager to do all the right things with fetching the resource from the archives. My archive handler at the bottom catches the mesh request the reads the mesh resource. Here is the code that starts it all:
void RendererOgre::AddEntity(Ogre::SceneManager* sceneMgr, Ogre::SceneNode* sceneNode,
char* entName, char* meshName) {
Ogre::ResourceGroupManager::getSingleton().declareResource(meshName,
"Mesh", MyResourceGroupName
);
Ogre::MovableObject* ent = sceneMgr->createEntity(entName, meshName);
sceneNode->attachObject(ent);
return;
}
I declare the mesh to Ogre::ResourceGroupMananger and declare it to be in my resource group. This is a nicety that really doesn’t effect anything. Then, an Ogre::Entity is created on an existing scene node and the mesh (in name only) is attached to it. Here we only specify the name of the mesh and the declare to the manager will mean the mesh is known and needs to be loaded.
Let’s take a moment to understand (at least as far as I do) how Ogre resources work. At the top level is Ogre::ResourceGroupManager. This manager is the controller of all the resource managers. Ogre::ResourceGroupManager keeps a list of all declared resources and, when initialized, it calls the appropriate managers to create and prepare the resources. After that, the group manager is only good for making general calls down into the various managers.
Most of the work is done by the type specific resource managers (Ogre::MeshManager, Ogre::MaterialManager, ...). In our above code, we put the name of a mesh on the entity node. Later, the rendering thread will call the mesh manager to find and load a mesh by that name. The mesh manager will create a mesh object and then call the archive handlers to read it in. Remember the lines you put in resource.cfg? Saying something like “Filesystem=../media/meshes” has the effect of creating an instance of Ogre::FilesystemArchive initialized to recursively search the directory “../media/meshes”. The resource managers eventually call the archive instances to find the datastream to read the resource contents from. This can be short circuited by specifying a manual loader for a resource but that’s for another article.
I inserted myself into this process by creating an archive class that is a wrapper around a filesystem archive. The wrapper checks to see if the file in the filesystem exists. If it does not, it generates the request to dynamically load the resource and then it returns a default datastream. So, for meshes, my archive returns a default mesh (a sphere) while requesting that the real mesh be loaded into the filesystem. [[TODO: insert a discussion of how resource names correspond to filenames and what that implies here.]]
Here is some pseudo code that shows the calling sequence. This is a mixture of calling sequences (one routine calling the next is shown as indented code) and decision logic. Loading a mesh calls the manager who finds the mesh who is loaded and its loading causes calls to the archives to get the data from where ever it is stored.
MeshManager::prepare(meshName, groupName)
MeshManager::createOrRetreive(meshName, groupName)
mesh = ResourceManager::createOrRetreive(meshName, groupName)
mesh.prepare()
Resource::prepare()
Mesh::prepareImpl()
ResourceGroupManager::getSingleton().openResource(meshName, groupName)
if (loadingListener)
loadingListener->resourceLoading(meshName, groupName, resourceBeingLoaded)
grp = getResourceGroup(groupName)
grp->resourceIndexCaseSensitive(resourceName)
if (found)
stream = foundArchive->open(resourceName)
if (loadingListener)
loadingListener->resourceStreamOpened(resourceName, groupName, resourecBeingLoaded, stream)
return stream
else
grp->resoureceIndexCaseInsensitive(resoureceName.toLower())
if (found)
stream = foundArchive->open(resourceName)
if (loadingListener)
loadingListener->resourceStreamOpened(resourceName, groupName, resourceBeingLoaded, stream)
return stream
else
foreach (archive in grp->locationList)
if (archive->exists(resourceName))
stream = archive->open(resourceName)
if (loadingListener)
loadingListener->resourceStreamOpened(resourceName, groupName, resourecBeingLoaded, stream)
return stream
if (shouldSearchAllGroups)
grp = findGroupContainingResoureceImpl(resourceName)
doTheSameCodeAboveForEachGroup
memoryCopy = new MemoryDataStream(stream)
MeshManager::load(meshName, groupName)
MeshManager::createOrRetreive(meshName, groupName)
mesh = ResourceManager::createOrRetreive(meshName, groupName)
mesh.load()
Resource::load()
Mesh::loadImpl()
MeshSerializer.importMesh(memoryCopy)
What you should get from all that is the resource system searches across all the archives several different ways in an attempt to find the data for the mesh. Once a datastream is found, it is read into memory and later imported as the mesh data. So, if I just slip in a new archive, I can load anything I want into the mesh.
A copy of sample archive code is HERE. Essentially, it creates an instance of Ogre::FilesystemArchive to do the actual filesystem operations and otherwise looks at the requested resource name and, if it’s one of mine, it says “yes it exists”. If some thing actually tries to load the resource, the routine starts the process of downloading the actual resource while, in the mean time, returning a stream of default contents. In my case, new meshes show up as a sphere made of a glittery substance until the real mesh is loaded and the displayed mesh is updated.
My updating works by storing the downloaded mesh into the filesystem in the correct cache location (where the filesystem archive will find it) and then, between frames, does a mesh->reload(). This eventually causes another entry into my archive routine for the mesh of that name and this time it is on the disk. The second time, the mesh is loaded from the disk and all is good.
Materials Are Resources But They Are Different
Once I got all that working, I figured it would be easy to do materials. Well, a few weeks later I have finally mastered that materials, while they are Ogre::Resources, they do not follow the same resource management workflow as meshes. Let me talk about some of the differences.
Materials are scripts. They are read and parsed by one of the script parsers. The invocation of this parsing is not done in Ogre::MaterialManager. The parsing is called in Ogre::ResourceGroupManager::InitializeGroupResources. That is, by default materials are only read and parsed at startup time. So, if you want to create new materials, you either have to build them up with code or you have to call the parser yourself to read from a datastream.
“Well, I’ll just use a manual loader then”. Nope. Material specifically cannot be loaded manually. The code even has a special error message to tell you this.
Materials are also different in that they have their names embedded in them. For meshes, I could fool the upper systems by passing them any ol’ datastream as a default shape. For materials, you’ll discover that the parsing routines don’t take a material name because they expect to read the name of the material out of the material file.
Thus, materials are always expected to be there. Material loading and unloading only changes the mapping of the material techniques onto the hardware. If a material is not defined when it is needed, the material does not go through the same search-through-all-the-archives code as described for meshes. Also, this means that just reloading a material does not update the display since the material is hidden down under the mesh and does not cause a scene recalculation.
These together mean the archive trick does not work for materials. So, what to do? You might have noticed in the above calling sequence example several checks for listeners. If the listener was specified, the routine calls out to allow modifications to the code flow. Well, some listeners were added to Ogre which allow material loading.
Materials are used in meshes. When a mesh is read in, Ogre::MeshSerializer accepts an optional listener which is called when the mesh is being parsed and a material name is encountered. While initializing Ogre, I do:
Ogre::MeshManager::getSingleton().setListener(new MyMeshSerializerListener());
Inside MyMeshSerializerListener is a routine for processing the found material name:
// called when mesh is being parsed and it comes across a material name
// check that it is defined and, if not, create it and read it in
void MyMeshSerializerListener::processMaterialName(Ogre::Mesh* mesh, Ogre::String* name) {
// get the resource to see if it exists
Ogre::ResourceManager::ResourceCreateOrRetrieveResult result =
Ogre::MaterialManager::getSingleton().createOrRetrieve(*name, OLResourceGroupName);
if ((bool)result.second) {
// the material was created by createOrRetreive() -- try to read it in
// The created material is an empty, blank material so we must fill it
Ogre::MaterialPtr theMaterial = result.first;
// Do the magic to make this material happen
FabricateMaterial(*name, theMaterial);
}
}
// We have been passed a material that needs to be filled in. We try to read it in from
// the file. If the file is not there, we fill it with the default texture and request
// the loading of the real material. When that real material is loaded, someone will call
// RefreshMaterialUsers.
void MyMeshSerializerListener::FabricateMaterial(Ogre::String name, Ogre::MaterialPtr matPtr) {
// Try to get the stream to load the material from.
Ogre::DataStreamPtr stream = Ogre::ResourceGroupManager::getSingleton().openResource(name,
MyResourceGroupName, true, matPtr.getPointer());
if (stream.isNull()) {
// if the underlying material doesn't exist, return the default material
// and request the real material be constructed
MakeMaterialDefault(matPtr);
Ogre::Material* mat = matPtr.getPointer();
RequestResource(name, MyResourceTypeMaterial);
}
else {
// There is a material file under there somewhere, read the thing in
try {
Ogre::MaterialManager::getSingleton().parseScript(stream, Ogre::String(MyResourceGroupName));
Ogre::MaterialPtr matPtr = Ogre::MaterialManager::getSingleton().getByName(name);
if (!matPtr.isNull()) {
// is this necessary to do here? Someday try it without
matPtr->compile();
matPtr->load();
}
}
catch (Ogre::Exception& e) {
Log("MyMeshSerializerListener::FabricateMaterial: error creating material %s: %s",
name.c_str(), e.getDescription().c_str());
}
stream->close();
}
return;
}
So, the above code checks to see if the material exists. If it doesn’t, I do the Ogre::ResourceGroupManager::openResource which does all the searching through the archives. If it’s found there, I have to parse the material script and then see that the material is loaded. If the file does not exist, I request the async thread to create the file (request it from the server) and, in the mean time, I initialize the new material structure with a default material specification.
Now, what happens when the material file is created? I can’t use the re-reading from the file trick because materials don’t unload like a mesh. My routine has to create the needed material (this works for me since I don’t get a .material file from the server but a description of the surface parameters). I serialize it to disk for the next application run. My routine can compile and load the material but that doesn’t update the scene. That is because the materials are integrated into the rendering engine when the mesh is loaded (creating all the different passes and such). Just changing the material does not change any of that. So, what I have to do is go find all the meshes that use a newly loaded material and reload the mesh. This reloads all the submesh info and thus all the materials.
// Routine to cause Ogre to redisplay the material. The problem is that changing a
// material and reloading it does not cause Ogre to reload the entities that use the
// material. Therefore, it looks like you change the material but the thing on the
// screen does not change.
// This routine walks the entity list and reloads all the entities that use the
// named material
// Note that this does not reload the material itself. This presumes you already did
// that and you now just need Ogre to get with the program.
// This must be called at between frames.
void MyMeshSerializerListener::RefreshMaterialUsers(const Ogre::String& matName) {
std::list m_meshesToChange;
// only check the Meshs for use of this material
Ogre::ResourceManager::ResourceMapIterator rmi = Ogre::MeshManager::getSingleton().getResourceIterator();
while (rmi.hasMoreElements()) {
Ogre::MeshPtr oneMesh = rmi.getNext();
Ogre::Mesh::SubMeshIterator smi = oneMesh->getSubMeshIterator();
while (smi.hasMoreElements()) {
Ogre::SubMesh* oneSubMesh = smi.getNext();
if (oneSubMesh->getMaterialName() == matName) {
// this mesh uses the material
// we sometimes get multiple materials for one mesh -- just reload once
std::list::iterator ii = m_meshesToChange.begin();
while (ii != m_meshesToChange.end()) {
if (ii->getPointer()->getName() == oneMesh->getName()) {
break;
}
}
if (ii == m_meshesToChange.end()) {
m_meshesToChange.push_front(oneMesh);
}
break;
}
}
}
if (!m_meshesToChange.empty()) {
for (std::list::iterator ii = m_meshesToChange.begin();
ii != m_meshesToChange.end(); ii++) {
ii->getPointer()->reload();
}
m_meshestoChange.clear();
}
return;
}
Textures
Now I have meshes with materials specified. The materials specify textures so what about them? Well, the texture resource manager is a well behaved manager like the mesh resource manager. When a texture is needed, Ogre::TextureManager is called and it does the searching through the archives for the file. So, for textures, the archive hack works. Yea!!
Update 20090620. The display update of the textures doesn’t happen, though. Sigh. That necessitates a hack similar to the materials but searching deeper into the materials to find the meshes with materials that have the changed texture.
// find all the meshes that use the texture and add it to a list of meshes
// TODO: figure out of just reloading the resource group is faster
void MyMaterialTracker::GetMeshesToRefreshForTexture(std::list* meshes, const Ogre::String& texName) {
// only check the Meshs for use of this material
Ogre::ResourceManager::ResourceMapIterator rmi = Ogre::MeshManager::getSingleton().getResourceIterator();
while (rmi.hasMoreElements()) {
Ogre::MeshPtr oneMesh = rmi.getNext();
Ogre::Mesh::SubMeshIterator smi = oneMesh->getSubMeshIterator();
while (smi.hasMoreElements()) {
Ogre::SubMesh* oneSubMesh = smi.getNext();
Ogre::String subMeshMaterialName = oneSubMesh->getMaterialName();
Ogre::MaterialPtr subMeshMaterial = (Ogre::MaterialPtr)Ogre::MaterialManager::getSingleton().getByName(subMeshMaterialName);
if (!subMeshMaterial.isNull()) {
Ogre::Material::TechniqueIterator techIter = subMeshMaterial->getTechniqueIterator();
while (techIter.hasMoreElements()) {
Ogre::Technique* oneTech = techIter.getNext();
Ogre::Technique::PassIterator passIter = oneTech->getPassIterator();
while (passIter.hasMoreElements()) {
Ogre::Pass* onePass = passIter.getNext();
Ogre::Pass::TextureUnitStateIterator tusIter = onePass->getTextureUnitStateIterator();
while (tusIter.hasMoreElements()) {
Ogre::TextureUnitState* oneTus = tusIter.getNext();
if (oneTus->getTextureName() == texName) {
// this mesh uses the material
// we sometimes get multiple materials for one mesh -- just reload once
std::list::iterator ii = meshes->begin();
while (ii != meshes->end()) {
if (ii->getPointer()->getName() == oneMesh->getName()) {
break;
}
ii++;
}
if (ii == meshes->end()) {
meshes->push_front(oneMesh);
}
break;
}
}
}
}
}
}
}
}
You will notice that this routine and the routine for reloading materials pushes the meshes to update onto a list. This is part of some small optimizations to only reload meshes once (multiple materials can change on a mesh in one frame). I haven’t finished timing this work so I’m not sure if just reloading the resource group would be quicker than this.
If you have special needs for building textures, there is an Ogre::ScriptCompilerListener which is called before the script creates any objects. This would be your chance to create your own texture objects as they are referenced in the material script.
The Other Resource Types
I don’t yet have an answer for this. I haven’t yet added animations and skeletons. Stay tuned.
Additional Notes and Comments
I don’t know why materials are so different than the other resources. I would think that Material.prepare() could do the parsing of the script and thus move that code out of Ogre::ResourceGroupManager. That plus some tainting mechanism to notify the entity containing a material of changes could make material act nearly identical to meshes and textures. I don’t know if a well behaved resource manager for materials would break things. Guess I will have to make up a patch.