Understanding .skeleton Binary Structure and Conversion to XML

Problems building or running the engine, queries about how to use features etc.
vincentdoan86
Gnoblar
Posts: 10
Joined: Wed May 31, 2023 4:11 am
x 1

Understanding .skeleton Binary Structure and Conversion to XML

Post by vincentdoan86 »

Hi everyone,

I’m currently diving into .skeleton files and trying to understand their binary structure to write a Python script that can convert them into XML. This has become necessary because the OgreXMLConverter tool is unable to handle some of the .skeleton files I’m working with, leaving me stuck.

Here’s what I’ve figured out so far about the structure of these .skeleton files:

The file starts with a header, followed by the bytes 00 00 00.
Then comes the name of a bone (UTF-8 encoded).
After that is the data for the bone with ID 0, followed by another 00 00 00.
This pattern repeats for the next bone (with ID 1), and so on.
There are too many bones in the skeleton files, and each bone seems to have a meaningful name.
While I’ve pieced together some information, I lack a comprehensive understanding of the exact binary structure and how to accurately parse the data.

My Goals:
Fully understand the binary format of .skeleton files.
Write a Python script to convert .skeleton files into XML.
I would greatly appreciate any help the community can provide, whether it’s documentation, sample code, or advice based on your experience. Additionally, if anyone has encountered issues with OgreXMLConverter and found workarounds, I’d love to hear about them.

This project is very important to me, and I believe the collective knowledge of this community can help make it a success. Because I want to import skeleton into Unity but the community does not have a tool to convert to a format that Unity supports, I can only import into blender and export to FBX or GLSL to import into Unity, so I have to find a way to extract skeleton to Xml to be able to import into blender. Thank you in advance for any assistance or insights you can share!

Below is the python code I wrote to convert, but it still cannot read the structure of skeleton file.

Code: Select all

import struct
import xml.etree.ElementTree as ET

# Chunk ID constants
SKELETON_HEADER = 0x1000
SKELETON_BONE = 0x2000
SKELETON_BONE_PARENT = 0x3000
SKELETON_ANIMATION = 0x4000
SKELETON_ANIMATION_TRACK = 0x4100
SKELETON_ANIMATION_TRACK_KEYFRAME = 0x4110

def read_string(data, offset):
    end = data.find(b'\x00', offset)
    if end == -1:
        raise ValueError("Null-terminated string not found.")
    string = data[offset:end].decode('utf-8')
    return string, end + 1

def parse_skeleton(data):
    offset = 0
    skeleton = ET.Element('Skeleton')

while offset < len(data):
    if len(data) - offset < 6:
        break
    chunk_id, length = struct.unpack_from('<HI', data, offset)
    offset += 6  # Move past chunk ID and length

    if chunk_id == SKELETON_HEADER:
        version, offset = read_string(data, offset)
        header = ET.SubElement(skeleton, 'Header')
        header.set('Version', version)

    elif chunk_id == SKELETON_BONE:
        name, offset = read_string(data, offset)
        handle, = struct.unpack_from('<H', data, offset)
        offset += 2
        position = struct.unpack_from('<fff', data, offset)
        offset += 12
        orientation = struct.unpack_from('<ffff', data, offset)
        offset += 16
        scale = struct.unpack_from('<fff', data, offset)
        offset += 12

        bone = ET.SubElement(skeleton, 'Bone')
        bone.set('Name', name)
        bone.set('Handle', str(handle))
        bone.set('Position', f'{position[0]} {position[1]} {position[2]}')
        bone.set('Orientation', f'{orientation[0]} {orientation[1]} {orientation[2]} {orientation[3]}')
        bone.set('Scale', f'{scale[0]} {scale[1]} {scale[2]}')

    elif chunk_id == SKELETON_BONE_PARENT:
        child_handle, parent_handle = struct.unpack_from('<HH', data, offset)
        offset += 4

        parent = ET.SubElement(skeleton, 'BoneParent')
        parent.set('ChildHandle', str(child_handle))
        parent.set('ParentHandle', str(parent_handle))

    elif chunk_id == SKELETON_ANIMATION:
        name, offset = read_string(data, offset)
        length, = struct.unpack_from('<f', data, offset)
        offset += 4

        animation = ET.SubElement(skeleton, 'Animation')
        animation.set('Name', name)
        animation.set('Length', str(length))

    elif chunk_id == SKELETON_ANIMATION_TRACK:
        bone_index, = struct.unpack_from('<H', data, offset)
        offset += 2

        track = ET.SubElement(animation, 'Track')
        track.set('BoneIndex', str(bone_index))

    elif chunk_id == SKELETON_ANIMATION_TRACK_KEYFRAME:
        time, = struct.unpack_from('<f', data, offset)
        offset += 4
        rotation = struct.unpack_from('<ffff', data, offset)
        offset += 16
        translation = struct.unpack_from('<fff', data, offset)
        offset += 12
        scale = struct.unpack_from('<fff', data, offset)
        offset += 12

        keyframe = ET.SubElement(track, 'KeyFrame')
        keyframe.set('Time', str(time))
        keyframe.set('Rotation', f'{rotation[0]} {rotation[1]} {rotation[2]} {rotation[3]}')
        keyframe.set('Translation', f'{translation[0]} {translation[1]} {translation[2]}')
        keyframe.set('Scale', f'{scale[0]} {scale[1]} {scale[2]}')

    else:
        offset += length - 6  # Skip unknown chunks

tree = ET.ElementTree(skeleton)
return tree

def convert_to_xml(input_file, output_file):
    with open(input_file, 'rb') as f:
        data = f.read()
    tree = parse_skeleton(data)
    tree.write(output_file, encoding='utf-8', xml_declaration=True)

# Example usage
input_file = "ninja.skeleton"
output_file = "ninja.skeleton.xml"
convert_to_xml(input_file, output_file)

rpgplayerrobin
Gnoll
Posts: 685
Joined: Wed Mar 18, 2009 3:03 am
x 379

Re: Understanding .skeleton Binary Structure and Conversion to XML

Post by rpgplayerrobin »

Have you debugged why it is unable to handle some of the .skeleton files?

Also, I guess you have checked the importSkeleton function to understand how to reverse engineer it to python?
https://github.com/ehsan/ogre/blob/mast ... alizer.cpp

It may be that the .skeleton file is old, and then you should attempt to get an older version of the source file to be able to use that to reverse engineer it instead.
You can download older source versions of Ogre here:
https://cloudsmith.io/~ogrecave/repos/ogre/packages/
or
https://github.com/OGRECave/ogre/releases

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5436
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1343

Re: Understanding .skeleton Binary Structure and Conversion to XML

Post by dark_sylinc »

The serializer is open source so I suggest you use a debugger and single step through it line by line to understand how it works.

I recommend Visual Studio (Windows), or QtCreator (Linux) or VSCode (Linux) for single-stepping with a debugger through the C++ source.

The brief overview is our file formats are grouped into "chunks" and can be in almost any order.

You can see here it loads the streamID (16-bit), and based on what it is, starts processing it.

Once inside a chunk, those chunks may expect a sub-chunk (possibly nested many sub-chunks), for example readAnimation() starts looking for such subchunks.

Cheers
Matias

PS: If OgreXMLUpgrader cannot read the skeleton file it strongly suggests the file is either corrupted, not a valid Ogre skeleton file, or the source code was customized to contain other data.

paroj
OGRE Team Member
OGRE Team Member
Posts: 2108
Joined: Sun Mar 30, 2014 2:51 pm
x 1134

Re: Understanding .skeleton Binary Structure and Conversion to XML

Post by paroj »

it seems some chinese game modified the .skeleton format. If you figure out what the additions are, we could try to detect that format and add an importer for it. This pops up rather regularly:

vincentdoan86
Gnoblar
Posts: 10
Joined: Wed May 31, 2023 4:11 am
x 1

Re: Understanding .skeleton Binary Structure and Conversion to XML

Post by vincentdoan86 »

After many days of trying to analyze the hex of a new skeleton file from Chinese games, I found some things to note as follows that cause OgreXMLConverter to not be able to decode it.

  1. In the chunk header SKELETON_BONE = 0x2000, in the skeleton file there are many hex codes that match this header, so the converter cannot accurately determine the number of bones.

  2. In the animation section SKELETON_ANIMATION_TRACK_KEYFRAME = 0x4110, this header is replaced with the following structure: the general animation header and the track's chunk id remain the same as 0x4000 and 0x4100, but the key frame's chunk id has been changed to 0x4120 and accompanied by a change in the structure of the key frames as shown in the hex below, 20 42 will be 2 bytes of the chunk, the next 2 bytes are the chunk size, the 2 bytes 00 00 may be padding, the next 2 bytes to 11 00 are the number of key frames in a track, and the last 2 bytes 03 00 are still unclear what attribute they are, followed by the data of each key frame, every 32 bytes will be a key frame with time, translate, and rotation for a track.

Code: Select all

00 40 53 68 00 00 C5 DC B2 BD 0A 0A D7 23 3F
	00 41 30 02 00 00 00 00 
		20 41 28 02 00 00 11 00 03 00 
			00 00 00 00 17 F2 D0 BC C2 80 F9 3D 42 F5 C0 3B 37 01 7E 3F 59 50 D8 BF 80 E5 71 C1 0D 8A 02 BF 
			0A D7 23 3D 42 96 CC BC DE 74 F9 3D A1 92 AA BC 1C F5 7D 3F 4B 3C A2 BF D0 14 36 C1 93 A8 F7 BE 
			0A D7 A3 3D 0D 1D CD BC 29 32 F9 3D 40 7F 4D BD 2C B1 7D 3F 9C 50 58 BF 5F EC E7 C0 21 92 E9 BE 
			8F C2 F5 3D 6C 6C C3 BC 52 D4 F8 3D 69 EB 97 BD BB 51 7D 3F 01 51 D8 BE 02 BC 79 C0 C8 D1 DC BE 
			0A D7 23 3E 5D 9C A0 BC 87 A3 F8 3D 8F EB AD BD FB 1F 7D 3F 00 00 00 00 A1 9D 21 C0 6A 68 D3 BE 
			CD CC 4C 3E 89 88 22 BC F5 D2 F8 3D D8 DE 9E BD 48 50 7D 3F 54 70 00 3F D0 D3 8F C0 15 52 CE BE 
			8F C2 75 3E 36 8A 6B 3B 68 2D F9 3D 35 7F 6A BD 66 AC 7D 3F 09 33 87 3F 68 31 0D C1 B4 56 CC BE 
			29 5C 8F 3E 73 7E 89 3C D2 6E F9 3D CF DC FD BC 19 EF 7D 3F DA A8 C0 3F 30 71 52 C1 BF 29 CC BE 
			0A D7 A3 3E 17 F2 D0 3C D3 80 F9 3D 37 F4 C0 BB 37 01 7E 3F DB 51 D8 3F 80 E5 71 C1 4A 7E CC BE 
			EC 51 B8 3E 34 FE EC 3C 00 6D F9 3D 1F F5 B0 3C 18 ED 7D 3F DA A8 C0 3F 88 65 52 C1 F9 84 CC BE 
			CD CC CC 3E 38 CB EE 3C 4C 20 F9 3D 7E CB 5A 3D F4 9E 7D 3F 09 33 87 3F 60 1F 0D C1 FD 16 CD BE 
			AE 47 E1 3E D5 D4 D8 3C 49 B3 F8 3D C3 E6 A3 3D 00 30 7D 3F 54 70 00 3F 6F B7 8F C0 28 64 CF BE 
			8F C2 F5 3E B0 F4 AD 3C 51 79 F8 3D 39 5F BC 3D 0A F5 7C 3F 00 00 00 00 9F 8F 21 C0 0B 9C D4 BE 
			B8 1E 05 3F 81 BA 45 3C 45 B2 F8 3D F2 3D AB 3D F7 2E 7D 3F 01 51 D8 BE DF 23 84 C0 B9 E3 DD BE 
			29 5C 0F 3F F3 DB 3B BA 83 1F F9 3D 26 E3 79 3D 53 9E 7D 3F 9C 50 58 BF 0F 44 FB C0 69 52 EA BE 
			9A 99 19 3F B7 D1 68 BC C2 6E F9 3D 9A 7D 04 3D E5 EE 7D 3F 4B 3C A2 BF 38 F1 40 C1 10 04 F8 BE 
			0A D7 23 3F 17 F2 D0 BC C2 80 F9 3D 42 F5 C0 3B 37 01 7E 3F 59 50 D8 BF 80 E5 71 C1 0D 8A 02 BF 

I still don't know clearly what the 2 bytes 03 00 represent, can you guys hypothesize what it is?