[2.2] Load and batch textures from memory
-
- OGRE Expert User
- Posts: 1148
- Joined: Sat Jul 06, 2013 10:59 pm
- Location: Chile
- x 168
[2.2] Load and batch textures from memory
Hello! I am working on encrypting my media files using Botan, its working really well for blend files and sounds (which my engine loads directly), now I have to make it work for textures.
The process is something like this: I previously encrypt all the files, then at runtime I open the encrypted file and decrypt it in to a buffer in memory, I am pretty sure I can make an Ogre::Image2 with that (I am working on that now, its that a good approach?) but I am not sure if its possible to later batch it like createOrRetrieveTexture with Ogre::TextureFlags::AutomaticBatching does
Thanks in advance!
Saludos!
The process is something like this: I previously encrypt all the files, then at runtime I open the encrypted file and decrypt it in to a buffer in memory, I am pretty sure I can make an Ogre::Image2 with that (I am working on that now, its that a good approach?) but I am not sure if its possible to later batch it like createOrRetrieveTexture with Ogre::TextureFlags::AutomaticBatching does
Thanks in advance!
Saludos!
- TaaTT4
- OGRE Contributor
- Posts: 267
- Joined: Wed Apr 23, 2014 3:49 pm
- Location: Bologna, Italy
- x 75
- Contact:
Re: [2.2] Load and batch textures from memory
I'm not sure to have fully understood what you're asking, but if the question is that if TextureFlags::AutomaticBatching works when you load textures from Image2 objects, the answer is yes. Since I procedurally generate (99% of) the textures with Substance, they're being loaded through this snippet of code:
outputPtr->image is an Image2 object that I previously filled with the content of a memory buffer.
OT: I'm interested in encrypting my assets too. At the moment, I'm using this naive approach: https://hbfs.wordpress.com/2011/11/08/mild-obfuscation/. Pardon me if it's a stupid question (I know nothing about cryptography), but do you know if Botan provides something similar? I'm interested in decryption speed over robustness.
Code: Select all
texture = Root::getSingleton().getRenderSystem()->getTextureGpuManager()->createOrRetrieveTexture(textureName, GpuPageOutStrategy::Discard,
TextureFlags::AutomaticBatching, TextureTypes::Type2D);
if (texture->getNextResidencyStatus() == GpuResidency::Resident)
{
texture->scheduleTransitionTo(GpuResidency::OnStorage, nullptr);
}
texture->scheduleTransitionTo(GpuResidency::Resident, outputPtr->image);
OT: I'm interested in encrypting my assets too. At the moment, I'm using this naive approach: https://hbfs.wordpress.com/2011/11/08/mild-obfuscation/. Pardon me if it's a stupid question (I know nothing about cryptography), but do you know if Botan provides something similar? I'm interested in decryption speed over robustness.
Senior programmer at 505 Games; former senior engine programmer at Sandbox Games
Worked on: Racecraft Esport — Racecraft Coin-Op, Victory: The Age of Racing
-
- OGRE Expert User
- Posts: 1148
- Joined: Sat Jul 06, 2013 10:59 pm
- Location: Chile
- x 168
Re: [2.2] Load and batch textures from memory
Thank you so much! I think that's exactly what I need, I didn't noticed that you can pass a image in to the scheduleTransitionTo
still I am getting a crash but must be my fault somewhere else (edit: yes, it was something stupid, its working now)
About encryption, I am just continuing the work of an intern that we had long ago, so I am not 100% certain why he choose this library, but I read that this library is like a standard for encryption. I am noticing that is pretty fast, encrypting like 45 blender files, 80 textures and 10 sounds takes like 4 seconds maybe (considering searching, writing to disk, etc), and decryption adds like 1 second to load times (confirmation pending since its not finished). I am using "AES-256/GCM" which is the fastest, not so sure about security, haven't done much research.
I can share code if you want
thanks again! I'll report when its fully working
still I am getting a crash but must be my fault somewhere else (edit: yes, it was something stupid, its working now)
About encryption, I am just continuing the work of an intern that we had long ago, so I am not 100% certain why he choose this library, but I read that this library is like a standard for encryption. I am noticing that is pretty fast, encrypting like 45 blender files, 80 textures and 10 sounds takes like 4 seconds maybe (considering searching, writing to disk, etc), and decryption adds like 1 second to load times (confirmation pending since its not finished). I am using "AES-256/GCM" which is the fastest, not so sure about security, haven't done much research.
I can share code if you want
thanks again! I'll report when its fully working
- TaaTT4
- OGRE Contributor
- Posts: 267
- Joined: Wed Apr 23, 2014 3:49 pm
- Location: Bologna, Italy
- x 75
- Contact:
Re: [2.2] Load and batch textures from memory
Senior programmer at 505 Games; former senior engine programmer at Sandbox Games
Worked on: Racecraft Esport — Racecraft Coin-Op, Victory: The Age of Racing
-
- OGRE Expert User
- Posts: 1148
- Joined: Sat Jul 06, 2013 10:59 pm
- Location: Chile
- x 168
Re: [2.2] Load and batch textures from memory
this is how I encrypt:
this is how I decrypt:
Code: Select all
int MainWindow::encryptFile( std::string input_file , std::string output_file ){
std::string passphrase = "caasdascfgf";
bool verbose_output = false;
Botan::AutoSeeded_RNG rng;
Botan::SecureVector<Botan::byte> salt = rng.random_vec(16);
Botan::PBKDF *pbkdf = Botan::get_pbkdf("PBKDF2(SHA-256)");
Botan::SymmetricKey key = pbkdf->derive_key(32,
passphrase,
&salt[0],
salt.size(),
10000);
Botan::InitializationVector iv(rng, 16);
if (verbose_output) std::cout << "Encryption key : " << key.as_string() << std::endl;
if (verbose_output) std::cout << "IV : " << iv.as_string() << std::endl;
std::ofstream ofs_file(output_file, std::ios::binary);
ofs_file.write(reinterpret_cast<const char *>(iv.bits_of().data()), 16);
ofs_file.write(reinterpret_cast<const char *>(&salt[0]),
static_cast<std::streamsize>(salt.size()));
if (verbose_output) std::cout << "Encrypting " << input_file << " to " << output_file << std::endl;
Botan::DataSource_Stream dss_file(input_file, true);
Botan::Pipe pipe(get_cipher("AES-256/GCM", key, iv, Botan::ENCRYPTION),
new Botan::DataSink_Stream(ofs_file));
pipe.process_msg(dss_file);
ofs_file.close();
return 0;
}
Code: Select all
int yDecrypt::decryptFile(std::string fileName)
{
std::ifstream fp2(fileName, std::ios::binary|std::ios::ate);
if( !fp2.is_open() ) {
std::cout <<"File not found: "<<fileName<<"\n";
return -1;
}
if(fp2.is_open()){
std::string input_file = fileName;
std::string passphrase = "caasdascfgf";
bool verbose_output = false;
std::ifstream ifs_file(input_file, std::ios::binary);
Botan::SecureVector<Botan::byte> iv_buffer(16);
ifs_file.read(reinterpret_cast<char *>(&iv_buffer[0]), 16);
Botan::InitializationVector iv(iv_buffer);
if (verbose_output) std::cout << "IV : " << iv.as_string() << std::endl;
Botan::SecureVector<Botan::byte> salt_buffer(16);
ifs_file.read(reinterpret_cast<char *>(&salt_buffer[0]), 16);
Botan::SecureVector<Botan::byte> salt(salt_buffer);
Botan::PBKDF *pbkdf = Botan::get_pbkdf("PBKDF2(SHA-256)");
Botan::SymmetricKey key = pbkdf->derive_key(32,
passphrase,
&salt[0],
salt.size(),
10000);
if (verbose_output) std::cout << "Encryption key : " << key.as_string() << std::endl;
Botan::DataSource_Stream dss_file(ifs_file);
Botan::Pipe pipe(get_cipher("AES-256/GCM", key, iv, Botan::DECRYPTION));
pipe.process_msg(dss_file);
ifs_file.close();
std::string cl = pipe.read_all_as_string();
int mFileLen = cl.size();
char* mFileBuffer = reinterpret_cast<char*>(malloc (mFileLen+1));
std::copy(cl.begin(),cl.end(),mFileBuffer);
setFileData(mFileBuffer, mFileLen);
}
return 0;
}
//Getters and Setters
void yDecrypt::setFileData(char* data, int size){
fileBuffer = data;
fileLen = size;
}
//Get fileBuffer
char* yDecrypt::getFileBuffer(){
return fileBuffer;
}
//Get fileLen
int yDecrypt::getFileLen(){
return fileLen;
}
char* fileBuffer;
int fileLen;
- dark_sylinc
- OGRE Team Member
- Posts: 5299
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1279
- Contact:
Re: [2.2] Load and batch textures from memory
Hi!
1. TextureGpuManager by default uses the ResourceGrpupManager. The recommended way to implement this would be to implement your own Archive, DataStream and ArchiveFactory and register the factory at startup (via ArchiveManager::addArchiveFactory) thus all assets can be en/decoded on the fly with no intrusive changes. See OgreZip.cpp and OgreFileSystem.cpp
Your Archive class would be almost identical to FileSystemArchive except it would create a MyEncryptedDataStream instead of a FileStreamDataStream.
And your MyEncryptedDataStream would be very similar to FileStreamDataStream; but with added code so that it de/encrypts on the fly before returning data to the caller.
This would allow you to automatically use encryption (actually this is obfuscation if the malicious agent has access to the client...) almost anywhere where Ogre interacts with I/O.
2. Alternatively for a TextureGpuManager specific solution you can use a ResourceLoadingListener, overload grouplessResourceExists and grouplessResourceLoading and register this listener via ResourceGroupManager::setLoadingListener
Ensure grouplessResourceExists returns true when asked for the encrypted file; and return a MemoryDataStream with the already decrypted data in grouplessResourceLoading
The main purpose for this listener is for procedural generated textures; but it can also be used as an alternative way to load from disk.
Either of these two solutions are preferred.
3. The third solution suggested so far (send an Image2 pointer to scheduleTransitionTo) is discouraged because it will force you to stall the main thread to wait for IO and decryption hence losing the main benefit of background streaming; whereas these other two solutions will perform IO in the background thread.
This is the only reason why I don't recommend it, so if you find this IO blockage to be acceptable, then go ahead.
Cheers
PS: Note that if the malicious agent has access to the code, advanced encryption is just as robust as a silly XOR scramble. All the malicious agent needs to do is to step into your instructions, read the secret passphrase and the algorithm used. This is a minor inconvenience. Perhaps you waste the cracker's one more day of extra work.
Robust encryption is useful when the code is run from a server and the client is just a dumb player (or the code is run from a machine with restricted physical access) or when secret assets need to be sent and stored over an insecure channel, like the internet or a thumbdrive.
A useful case for strong encryption would be if the app requires the user to type the password every time he boots the PC; and the point is to protect the assets from being stolen (e.g. a burglar steals the entire PC). But in such case, perhaps dedicated full-disk encryption services like Bitlocker may be better solutions.
1. TextureGpuManager by default uses the ResourceGrpupManager. The recommended way to implement this would be to implement your own Archive, DataStream and ArchiveFactory and register the factory at startup (via ArchiveManager::addArchiveFactory) thus all assets can be en/decoded on the fly with no intrusive changes. See OgreZip.cpp and OgreFileSystem.cpp
Your Archive class would be almost identical to FileSystemArchive except it would create a MyEncryptedDataStream instead of a FileStreamDataStream.
And your MyEncryptedDataStream would be very similar to FileStreamDataStream; but with added code so that it de/encrypts on the fly before returning data to the caller.
This would allow you to automatically use encryption (actually this is obfuscation if the malicious agent has access to the client...) almost anywhere where Ogre interacts with I/O.
Code: Select all
--------------------------------------------------------------------------
Ensure grouplessResourceExists returns true when asked for the encrypted file; and return a MemoryDataStream with the already decrypted data in grouplessResourceLoading
The main purpose for this listener is for procedural generated textures; but it can also be used as an alternative way to load from disk.
Code: Select all
--------------------------------------------------------------------------
3. The third solution suggested so far (send an Image2 pointer to scheduleTransitionTo) is discouraged because it will force you to stall the main thread to wait for IO and decryption hence losing the main benefit of background streaming; whereas these other two solutions will perform IO in the background thread.
This is the only reason why I don't recommend it, so if you find this IO blockage to be acceptable, then go ahead.
Cheers
PS: Note that if the malicious agent has access to the code, advanced encryption is just as robust as a silly XOR scramble. All the malicious agent needs to do is to step into your instructions, read the secret passphrase and the algorithm used. This is a minor inconvenience. Perhaps you waste the cracker's one more day of extra work.
Robust encryption is useful when the code is run from a server and the client is just a dumb player (or the code is run from a machine with restricted physical access) or when secret assets need to be sent and stored over an insecure channel, like the internet or a thumbdrive.
A useful case for strong encryption would be if the app requires the user to type the password every time he boots the PC; and the point is to protect the assets from being stolen (e.g. a burglar steals the entire PC). But in such case, perhaps dedicated full-disk encryption services like Bitlocker may be better solutions.
-
- OGRE Expert User
- Posts: 1148
- Joined: Sat Jul 06, 2013 10:59 pm
- Location: Chile
- x 168
Re: [2.2] Load and batch textures from memory
thank you so much for the details, I am ok with the last approach since I already had waitForData() because of this: viewtopic.php?p=544492#p544492 but I am really glad that there's options, thanks
Saludos! <3
well at least that might discourage some of the bad guysdark_sylinc wrote: ↑Thu Feb 13, 2020 3:00 am All the malicious agent needs to do is to step into your instructions, read the secret passphrase and the algorithm used. This is a minor inconvenience. Perhaps you waste the cracker's one more day of extra work.
Saludos! <3
- dark_sylinc
- OGRE Team Member
- Posts: 5299
- Joined: Sat Jul 21, 2007 4:55 pm
- Location: Buenos Aires, Argentina
- x 1279
- Contact:
Re: [2.2] Load and batch textures from memory
Note there's still a slight marginal potential improvement (whether it's worth it, that's another matter). For example:xrgo wrote: ↑Thu Feb 13, 2020 4:00 pm thank you so much for the details, I am ok with the last approach since I already had waitForData() because of this: viewtopic.php?p=544492#p544492 but I am really glad that there's options, thanks
Code: Select all
for(...)
{
loadTexture( i ); //Starts loading in bg
createItem( i ); //Loads mesh in main thread
}
doSomethingElse(); // Occupy main thread on something else
waitForData(); // by now, we have probably loaded 'some stuff' in the bg. Now wait until everything's done.
Cheers
Matias