[Accepted] Graphics Card Emulation

Threads related to Google Summer of Code
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Looks good. On the texture size, getting this information out of GL is really hard (you can only get it by actually creating and probing), although not impossible.

For info, I added a simple caps reporting program to ogrenew/Tests/CapsReport for another purpose, it justs inits Ogre in both GL and D3D and you pick out the log files, but a similar kind of program will be useful for this obviously.
juozasg
Kobold
Posts: 35
Joined: Sun May 07, 2006 7:13 am

Running out of enum space

Post by juozasg »

To facilitate storing complete render system information in RenderSystemCapabilities I have to add more options to RenderSystemCapabilities::Capabilities enum. Right now the enum is used as a bitfield with each option being a higher bit. This way all options can be ORed in a single enum. Unfortunately, there are not enough bits left in the 32 bit enum.

My questions are: is sizeof(enum) == sizeof(int) always? And if that's the case, what's the best choice for adding more options?

Perhaps someone in the community can suggest a solution. Right now I'm thinking about simply adding more bool member variables to RenderSystemCapabilities and leaving the Capabilities enum alone.
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Post by CABAListic »

My questions are: is sizeof(enum) == sizeof(int) always? And if that's the case, what's the best choice for adding more options?
Not sure about sizeof(enum), but you could always use a 64bit integer instead and define the options as const integers instead of enum values.
I wonder, though. If there are more than 32 options, is ORing them a sensible decision, anyway? ORing 32 different options together would likely make code quite a bit ugly, I'd imagine.
juozasg
Kobold
Posts: 35
Joined: Sun May 07, 2006 7:13 am

Post by juozasg »

It seems that the only choice to is to add another enum to RenderSystemCapabilities. After discussing this with haffax there are few options. Either add a CapabilitiesEx/Capabilities2 type of enum. Or add a RenderSystem-specific Capabilities. Something like DirectX9Capalities (or jus DirectXCapabilities for all future versions) and OpenGLCapabilities.

We also need new hasCapability/setCapability methods. I am expecting for community and Sinbad to agree on one direction, but for now I'll work on OpenGL render system only with all specific enums and methods just for OpenGL specific capabilities.

The option to use all constants is still on the table as well, because it should not break any code.
User avatar
Kencho
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 4011
Joined: Fri Sep 19, 2003 6:28 pm
Location: Burgos, Spain
x 2
Contact:

Post by Kencho »

My vote for adding an extra "SpecificCapabilities" enum or something similar.
Image
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

I've wondered when we'd hit this ;)

A less disruptive option is to categorise the existing enums. That is, instead of using all 32 bits individually (obviously 32 values), allocate 'x' bits at the top to be a category, interpreted like a regular number, and the rest of the bits to be the actual caps in that group. For example if you allocated 4 bits to be the category, that would give you 16 categories of 28 bits, which is plenty (448 caps bits).

You'd have to have 4 32-bit values (top 4 bits set to 0) to store this data, and to test a capability you'd just have to use the top 4 bits to determine the value to test against (easy, just bit-shift down by 28 bits and mask off everything else and it's an array index from 0-3), and then perform a regular bitwise test. This allows the hasCapability / setCapability methods to remain exactly the same. Just remember to mask off the top 4 bits when setting the capabilities (to keep the top 4 bits zeroised in the stored version so the & doesn't return unexpected non-zero results during hasCapability)
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

I see you've had a go at implemementing this, but it's not quite correct. You've done this:

Code: Select all

	// SECOND CAPABILITIES SET (first four bits == 1)
	/// Supports 3d (volume) textures
	RSC_TEXTURE_3D				= 0x04000001,
... but that's not correct. 0x04 sets one of the lowest 4 bits of the top 8 bits, not the top 4 bits. So you're wasting 4 bits there. That's binary 00000100.

Also, the second capabilities should really be '1', not 4. In binary, first = 0000, second = 0001, third = 0010, etc. You can go up to 15 this way.

I recommend defining masks more explicitly, something like this:

Code: Select all

#define CAPS_CATEGORY_SIZE 4
#define CAPS_BITSHIFT (32 - CAPS_CATEGORY_SIZE)
#define CAPS_CATEGORY_MASK (2^CAPS_CATEGORY_SIZE - 1) << CAPS_BITSHIFT
#define CAPS_VALUE(cat, val) (c << CAPS_BITSHIFT) | (1 << val)
That way you can more simply define a capability value by using CAPS_VALUE rather than manually calculating them, and you can just use decimal for simplicity:

Code: Select all

// first param has to be 0-15 (category), second param 0-27 (value bit in category)
CAPS_VALUE(0, 0)
CAPS_VALUE(1, 15)
CAPS_VALUE(1, 16)
CAPS_VALUE(1, 17)
...etc
You can also use the CAPS_BITSHIFT value to perform calculations.
juozasg
Kobold
Posts: 35
Joined: Sun May 07, 2006 7:13 am

Post by juozasg »

I agree that defining the CAPS_VALUE macro is a good idea. And now I understand how I made the mistake of using the wrong bits for categorization - I used a binary calculator with 0b110000....000000 (32 bit string) to pick the top 2 bits (for total of 4 categories) in hex form, but I must have made a mistake and picked the bottom 4 of top 8. It will all become less confusing with some macros
juozasg
Kobold
Posts: 35
Joined: Sun May 07, 2006 7:13 am

Post by juozasg »

I would like to solicit a bit more feedback:

While coding and designing this feature I've really warmed to "custom capabilities" nomenclature (instead of "emulation"). I believe "custom capabilities" is the most accurate description of the feature that is under 3 words in length. If no one has other suggestions and request I would like to design all the interfaces to use this name.

Secondly. I believe RenderSystemCapabilities serialization should be implementing using the Ogre's ResourceManager mechanism. New manager, called RenderSystemCapabilitiesManager, will be created. Whenever some .rendercaps files are included in resource.cfg they will be loaded. This practically eliminates all overhead of custom capabilities. However, sometimes it might be necessary to switch between custom and real capabilities very often. It would be more convenient to a single switch in Ogre::Root than to constantly update resources.cfg. The switch would not stop RenderSystemCapabilitiesManager from being created, but it would prevent it from actually parsing each (or any) .rendercaps file that is included by ResourceGroupManager. The only cost of this is additional complexity.
juozasg
Kobold
Posts: 35
Joined: Sun May 07, 2006 7:13 am

Post by juozasg »

Since RenderSystemCapabilities are resources read from files, we need to make a class called RenderSystemCapabilitiesSerializer. These are my ideas for the implementation. Comments and further ideas would be appreciated.

This serializer is inspired by MaterialSerializer, but it will be leaner and more dynamics. Adding new capabilities to RenderSystemCapabilitiesSerializer will be very easy, but adding new sub-sections and other types of logic will be harder - basically, it's MaterialSerializer rewritten for flatter files.

Here is an example .rendercaps file to illustrate the discussion:

Code: Select all


render_system_capabilities "ATI Mobility Radeon X1600"
{
  gl_driver "8.34.8"
  gl_vendor "ATI Technologies Inc."
  gl_renderer "ATI Mobility Radeon X1600"

  num_world_matrices 10
  num_texture_units 10
  stencil_buffer_bit_depth 8
  non_POW2_textures_limited false

  automipmap            true
  blending                  true
  anisotropy                 true
  dot3                   false

  shader_profile vs_2
  shader_profile ps_1_x
}
There are 5 types of capabilities:
1. string caps set with custom set methods (example: void setGLDriver(String striver)
2. int caps with custom set methods (void setNumTextureUnits(int units)
3. bool caps with custom set methods (void setNonPow2Texture(bool allowNonTextures))
4. bool caps with setCapability(Capabilities cap)
5. string shader profile caps with addShaderProfile(String profile)

For the needs of RenderSystemCapabilitiesSerializer this could be abstracted in the enum CapabilitiesTypes {CUSTOM_STRING, CUSTOM_INT, CUSTOM_BOOL, SET_CAPABILITY, SHADER_STRING}

There would be 5 dispatch tables (each for a different type) that stores all the required information for interpreting each line of .rendercaps file. There would also be a master table that maps each keyword (i.e. first word of every line) to the type table.

An example of a dispatch table entry (for CUSTROM_STRING type):

Code: Select all

typedef void (*CUSTOM_STRING_SETTER)(String value);

typedef map<String, CUSTOM_STRING_SETTER> CustomStringDispatcherMap;

CustomStringDispatcherMap customStringDispatchers;

customStrindDispatchers.insert(CustomStringDispatcherMap::value_type("gl_driver",(*CUSTOM_STRING_SETTER) setGLDriver));
Here is yet another example for SET_CAPABILITY type dispatchers

Code: Select all

typedef map<String, Capabilities> SetCapabilityDispatcherMap;

SetCapabilityDispatcherMap setCapabilityDispatcher;

setCapabilityDispatcher.insert(SetCapabilityDispatcherMap::value_type("automipmap", RSC_AUTOMIPMAP));
Finally, the master table will match each keyword (like "automipmap") with an enum value (like SET_CAPABILITY)

Inside RenderSystemCapabilitiesSerializer various parser and dispatcher methods will use these tables


Everyone's comments are welcome and valued. Is this overengineered? Wasteful duplication of MaterialSerializer? Any suggestions on naming of things would also be useful, because I am not yet totally in sync with Ogre's nomenclature.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Interestingly you could make use of Praetor's Summer of Code project for parsing these files, but that's probably too much of a dependency. A line-by-line simple parser is probably the best approach for a simple format like this.

The gl_ capabilities are interesting. They are occasionally used to avoid GL driver problems but beyond that they have no prupose, they are just there for log information. They certainly won't mean anything to the D3D renderer so I don't think they should really be there in that form. However, a possible other approach would be to unify the detection of the GPU manufacturer, model and driver version by mapping both GL driver strings and D3D vendor caps flags to some unified lookup table which is also exposed in the caps (not as GL strings like that, but generalised versions). That might be quite valuable.

Personally I'm not sure I would have made the capabilities system a ResourceManager - it's definitely a ScriptParser but I'm not sure about a resource which needs loading / unloading. Not a huge deal anyway, it doesn't hurt to leave it like that for now.
juozasg
Kobold
Posts: 35
Joined: Sun May 07, 2006 7:13 am

Post by juozasg »

Thanks for the comments. I'm very unsure that I understand what you mean by "mapping both GL driver strings and D3D vendor caps flags to some unified lookup table which is also exposed in the caps." Could you possibly elaborate on the specifics, because I'm afraid of ending up with the totally incorrect idea.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

What I meant was have a RenderSystem-independent set of information on what the GPU manufacturer, model and driver version are. So, for example (and note this is not very well structured, just an example):

Code: Select all

enum GpuManufacturerID
{
    GM_NVIDIA = 0
    GM_ATI = 1, 
    GM_S3 = 2, 
    ...
    GM_MAX = //(highest val + 1)
    GM_UNRECOGNISED = 0xFF
};
String gpuManufacturerStrings[GM_MAX] = { "NVidia", "ATI", "S3" .. }

struct GpuIdentifier
{
    GpuManufacturerID manufacturer;
    String modelName;
    uint32 deviceID;
    String driverString;
    uint32 driverMajor, driverMinor;
};
In DirectX, each manufacturer has a vendorID and every product a deviceID (we use it for one or two things in the D3D RS). An example database of IDs is here: http://developer.nvidia.com/object/device_ids.html

It would be really nice to be able to have a single way of looking up the vendor id/name, model number/name and driver version of the card being used across rendersystems. It's much easier to do in D3D than GL I think though.

I wouldn't get too distracted on this if you have other things to work on, just a 'would be nice' thing.
juozasg
Kobold
Posts: 35
Joined: Sun May 07, 2006 7:13 am

Post by juozasg »

There are few more changes to custom capabilities interface. Originally I had planned (and implemented to some degree) RenderSystemCapabilities as a Resource. Doing this means that the existing loading/configuration infrastructure (specifically resources.cfg and ogre.cfg) can be reused to request custom capabilities.

There is just one huge problem with doing this. Resource are loaded after first RenderWindow is created - but, custom capabilities are used when creating the first RenderWindows.

The best solution (in my mind) seems to restore RenderSystemCapabilities to its non-Resource status, but keep using RSCManager (no longer inheriting from ResourceManager) and RSCSerializer for parsing. Expect this time RSCManager should store a map of RenderSystemCapabilities. Serializer, when asked to parse a script, would parse it, construct a new RSC, and hand it over to the singleton RSCManager class. That's pretty similar to the resource framework, expect that it must be triggered manually.

The loading of RSCs from .rendercaps scripts should be triggered in Root. My proposal is to add a new String parameter to Root::initialise called "customCapabilitiesConfig" (default set to StringUtil::BLANK). When this parameter is set to a valid fiile, Root will extract the name of custom capabilities and a list of archive scripts from the config file. The only requirement for it is that all the archives be in a single section called "Capabilities Database." Any valid archive can potentially store .rendercaps files.

If this config file is given, Root ask RSCManager to parse every .rendercaps in every archive. If "Custom Capabilities" option is given in the config file, Root will ask RSCManager to get the named RSC (this is just a std::map lookup, because all the RSCs are constructed and stored during parsing). Finally the RenderSystem is told to use the loaded RSC and createWindow is called.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Yeah, I think this is a better approach. As well as the timing issues the caps were never really conceptually resources, the loading aspect was just a convenience.
juozasg
Kobold
Posts: 35
Joined: Sun May 07, 2006 7:13 am

Post by juozasg »

We're getting close to the end of SoC and there are few important parts that are not yet done.

First is driver versioning. I like Sinbads comprehensive GpuIdentifier struct, but I don't believe I'll find time to do it. We need some kind of driver versioning, because we want our database to contain only the caps with latest drivers. This is reasonable, because we can expect users of our programs to update their drivers the latest (as long as the latest have new features, not just big fixes or speed improvements)

My idea is to use a simple DriverVersion struct and 3 ints for major/minor/release. In RSC use few bools to define whether RSC is DX9 or GL (it should be exclusive to Render System, because merging RSCs is not trivial).

The second query I had was about the building the database. I'll write a very simple tool for dumping RSC into .rendercaps file (basically taking the existing integrated test and copying them into a standalone executable). The question I have is where and how should users submit .rendercaps? Perhaps to a new email account as an attachment, or using a simple web form? Anyone has any ideas or preferences?
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Yeah, I didn't really expect you to have time for the unified GpuIdentifier - it would be nice but best to concentrate on the fundamentals of the emulation. Generally caps will stay fixed between driver versions most of the time, it's pretty rare for a driver bump to suddenly expose something that wasn't there before (not unheard of, but fairly rare). Therefore whilst a driver version is useful information (and the 3-int version is generally reasonable) I don't think it need affect how the caps are read necessarily.

As for submitting .rendercaps files I think you can treat that as external to this project too. We'll probably have to provide a web-based system to do it, although perhaps the wiki would be an interesting place to consider, since it's already there, and it's easily browsable.

Out of interest, I'm wondering why the caps definitions can't be used across rendersystems? I realise the way they're extracted and set must be different, but the fact that the caps themselves are deliberately designed to be rendersystem independent leads me to believe that you should in theory be able to 'import' them on either. Maybe that doesn't make a lot of sense, since you're trying to recreate the conditions on that particular render system after all, but for many test setups (e.g. 'no shaders') I'm not understanding why you would need to create 2 different caps configurations. Is it related to things like shader profiles?
juozasg
Kobold
Posts: 35
Joined: Sun May 07, 2006 7:13 am

Post by juozasg »

The reason to not merge them is mostly for that reason: DX9 drivers and OpenGL having slightly different supported profiles. I actually don't know how common this is so I simply made a conservative assumption.

If we're to merge capabilities, (using RSC::appendCapabilities(RSC*) method maybe) I think we could just OR every boolean. We can't AND them because there are few, like D3D9 PERSTAGECONSTANT and ATI_FBO, ARB_FBO and other special cases like this. Perhaps we can AND all the regular caps and OR RS specific caps but then the code will be somewhat more complicated. Shader profiles lists would be concatenated.
User avatar
haffax
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 4823
Joined: Fri Jun 18, 2004 1:40 pm
Location: Berlin, Germany
x 7
Contact:

Post by haffax »

I see two problems with sharing rscaps profiles, but no problems that can't be overcome. First: Drivers can report different capabilities for each Render-API, as Juozas noted. Second: How do we merge reports made with DirectX, DirectX/GL (both Windows) and GL (Linux)? At least between windows drivers and all those kinds of Linux drivers there is a difference to be expected.

But both problems are only relevant for building the database, not for developers using the API. So both RenderSystems should be able to initialise their caps from a single rscaps-profile. That some settings only matter for dx and some only matter for gl should not be too difficult to handle. This is an implementation detail.
team-pantheon programmer
creators of Rastullahs Lockenpracht
juozasg
Kobold
Posts: 35
Joined: Sun May 07, 2006 7:13 am

Post by juozasg »

The end is almost here and although there isn't a database of rendercaps here yet, the functionality for building it is all done and has been unit tested to near death by mad scientists obsessed with asserts and test fixtures.

The last thing to decide is how should we package rcapsdump tool, which produces .rendercaps files to be uploaded into the database.

In my opinion, for windows zip archive with binaries of OgreMain, ogre plugins, plugins.cfg and rcapsdump.ext in a single folder. We should put the link to this .zip file on the front page, tell people to download it and direct them to the web form where they can upload their .rendercaps.

For linux, we should simply ask people to build and run rcapsdump from Ogre CVS.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

On windows a simple installer will be needed, at least for VC8 based binaries, so that the CRT gets installed properly. The Demos or Command-line tools WiX script in CVS would be a good guide.
juozasg
Kobold
Posts: 35
Joined: Sun May 07, 2006 7:13 am

Post by juozasg »

I have tried building CommandLineTools.msi from the existing WIX script with no look. I believe the problem is that the scripts are out of date and will not work with WiX versions 2.5 and 3.0. Updating them is as simple as renaming a single class and changing the build.bat scripts to link with the newer version of UI library.

But I'm not absolutely confident that versioning is the issue and wanted to ask you if you could install the latest WiX and try building the .msi with that.

If it is outdated libraries that causes the problems with new version of WiX, I can fix the script (and add a new one) fairly quickly
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Yeah, I'm still using WiX 2.0, I have no reason to update really. I don't really want to right now since I'll have to change the SDK & XSI exporter scripts too, when the current ones work ok. If it's a 5-minute change that's fine but it looked more than that last time I checked. Not high on my list ;)

Old WiX's are here: http://sourceforge.net/project/showfile ... _id=114109
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

This project has now been merged into CVS HEAD, where work will continue as required. Future posts on this subject should therefore be in the other forums.
Post Reply