[Help]How to manage lots of 3rd party libs?

[al2950]

Hi

I have a number of 3rd party libs that I use directly or make small changes to for my system. My system is stored in SVN but the 3rd party libs originate from various different source control systems. Updating just one these libraries in my repository can be a complete nightmare, especially if have added my own customization. So I was wondering what other people do to manage 3rd party libs in your own repository? Are there any tools I could use?

[TheSHEEEP]

Usually you just take one version of a library/tool to include in your project and stick with that version.
Only update when it is really neccessary or would gain you a benefit that is actually worth the hassle.

Updating for the sake of keeping everything up to date rarely makes sense in software projects.

[al2950]

Very true, however even if it is done rarely it must be a common issue. I generally update libraries once a year because I cant ignore some of the new features.

[Kojack]

I use ruby scripts to handle this kind of thing. It's really easy to load visual studio project files (just xml) in ruby and hack them to have the correct settings. For example, I redirect all temp files to a single temp directory, so I can zip up an entire engine and skip all the temp crap just by not selecting one directory.
I also use them to copy source files into my engine.

[Mind Calamity]

Even though I never really keep anything clean as it is, I try to at least keep my libraries in one somewhat clean folder.

I currently have about 67 libraries in "D:\Libraries", I have environment variables for the said directory and for some specific libraries, like NxOgre, Bullet or Qt, and they rely on the $(libdir) variable.

So every time I need to re-format my C:\ drive I can just run a little batch file I made and get my development environment up and running in no-time.

Although, I must say Kojack's idea may be much better than mine, but for now (while I haven't started any serious development on the game engine) this solution works perfectly.

[Kojack]

Mine is a little more complicated than most people would do because I have some extra requirements. Primarily, my projects need to be easily moved from computer to computer with just a file copy or zip, no installers can be run, no environment variables (besides directx), minimum rebuilding after a move, and the destination could be any path in any user account.
Also, since I'm using this for teaching, I want consistent directory layout for all libs.
Unfortunately CMake really complicates moving projects around, most libs have wildly varying directory structures, and a lot of ogre addons need environment variables. So I use ruby scripts to fix all that.

Here's an example snippet from my ogre cloning script:

Code: Select all

require 'fileutils'
include FileUtils
  
ogresource = "c:/work/ogre_mercurial/"
ogrebuild = "c:/work/ogre_mercurial_build_20110222/"
qgf5 = "c:/work/qgf5/"

Dir.mkdir(qgf5+"dev/components/ogre") rescue nil
Dir.mkdir(qgf5+"dev/components/ogre/components") rescue nil
Dir.mkdir(qgf5+"dev/components/ogre/ogremain") rescue nil
Dir.mkdir(qgf5+"dev/components/ogre/plugins") rescue nil
Dir.mkdir(qgf5+"dev/components/ogre/rendersystems") rescue nil
Dir.mkdir(qgf5+"dev/components/ogre/tools") rescue nil
Dir.mkdir(qgf5+"dev/sdk/include/ogre") rescue nil

cp_r Dir.glob(ogresource+"ogremain/src/*"), qgf5+"dev/components/ogre/ogremain/src"
cp_r Dir.glob(ogresource+"ogremain/include/*"), qgf5+"dev/sdk/include/ogre"
cp_r Dir.glob(ogrebuild+"include/*.h"), qgf5+"dev/sdk/include/ogre"
cp_r Dir.glob(ogrebuild+"ogremain/*.vcproj"), qgf5+"dev/components/ogre/ogremain/scripts"

(There's actually 136 lines, that's just an example)
It creates the directories I want in my framework's directory, then copies ogre files over to it.
I have a directory structure like:

Code: Select all

bin
   debug
   release
dev
   components
      ogre
   sdk
      include
         ogre
      lib
media
projects
temp
tools

Instead of ogremain (and every other ogre plugin) having both their cpp and include directories side by side, I have all files used to build all components in dev/components while all built libs and headers go in dev/sdk. This means that after I do a build I can zip up everything but ignore dev/components if I want a binary only copy of the framework (saves space, dev/components is almost 500MB).

For the vcproj modifying ruby script, here's an example:

Code: Select all

require "rexml/document"
include REXML
$doc = nil

def getXML(path, attrib)
  elem = $doc.elements[path]
  if elem
    return $doc.elements[path].attributes[attrib]
  end
  nil
end

def setXML(path, attrib, newvalue)
  elem = $doc.elements[path]
  if elem
      elem.attributes[attrib] = newvalue
  else
    puts "Failed to find path "+path
  end
end

def loadXML(path)
  file = File.new(path,"r")
  if file
    puts "Loaded "+path
    $doc = Document.new file
  else
    puts "Failed to load "+path
    $doc = nil
  end
end

def saveXML(path)
  if $doc
    file = File.new(path,"w")
    if file
      puts "Saving "+path
      file << $doc.to_s().gsub(/'/,"\"")
    else
      puts "Failed to save "+path
    end
  end
end

def deleteFile(filename)
	$doc.each_element("//File") {|k|
		f = k.attributes["RelativePath"]
		if (f[f.length-filename.length...f.length].casecmp filename) == 0 then
			k.parent.delete_element(k)
		end
	}
end

def changePath(ending, find, replace)
	$doc.each_element("//File") {|k|
		f = k.attributes["RelativePath"]
		if (f[f.length-ending.length...f.length].casecmp ending) == 0 then
			if find == nil then
				f[0...f.length-ending.length] = replace
			else
				i = f.index(find)
				if i then
					f[0...i+find.length] = replace
				end
			end
		end
	}
end

convert_ogremain = true

if convert_ogremain then
loadXML "../dev/components/ogre/OgreMain/scripts/ogremain.vcproj"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Debug|Win32']", "OutputDirectory", "$(SolutionDir)"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Debug|Win32']", "IntermediateDirectory", "$(SolutionDir)/temp/$(ConfigurationName)/$(ProjectName)"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Debug|Win32']/Tool[@Name='VCCLCompilerTool']", "AdditionalIncludeDirectories", "$(SolutionDir)/dev/sdk/include/ogre;$(SolutionDir)/dev/sdk/include;$(SolutionDir)/dev/components/ogre/ogremain/src/nedmalloc"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Debug|Win32']/Tool[@Name='VCPostBuildEventTool']", "CommandLine", ""
setXML "VisualStudioProject/Configurations/Configuration[@Name='Debug|Win32']/Tool[@Name='VCLinkerTool']", "OutputFile", "$(SolutionDir)/bin/$(ConfigurationName)/OgreMain_d.dll"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Debug|Win32']/Tool[@Name='VCLinkerTool']", "AdditionalLibraryDirectories", "$(SolutionDir)/dev/sdk/lib/$(ConfigurationName);$(SolutionDir)/dev/sdk/lib/release"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Debug|Win32']/Tool[@Name='VCLinkerTool']", "AdditionalDependencies", "$(NOINHERIT) kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib zlibd.lib zziplibd.lib FreeImaged.lib freetype2311_D.lib"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Debug|Win32']/Tool[@Name='VCLinkerTool']", "ProgramDatabaseFile", "$(SolutionDir)/bin/$(ConfigurationName)/OgreMain_d.pdb"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Debug|Win32']/Tool[@Name='VCLinkerTool']", "ImportLibrary", "$(SolutionDir)/dev/sdk/lib/$(ConfigurationName)/OgreMain_d.lib"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Release|Win32']", "OutputDirectory", "$(SolutionDir)"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Release|Win32']", "IntermediateDirectory", "$(SolutionDir)/temp/$(ConfigurationName)/$(ProjectName)"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Release|Win32']/Tool[@Name='VCCLCompilerTool']", "AdditionalIncludeDirectories", "$(SolutionDir)/dev/sdk/include/ogre;$(SolutionDir)/dev/sdk/include;$(SolutionDir)/dev/components/ogre/ogremain/src/nedmalloc"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Release|Win32']/Tool[@Name='VCPostBuildEventTool']", "CommandLine", ""
setXML "VisualStudioProject/Configurations/Configuration[@Name='Release|Win32']/Tool[@Name='VCLinkerTool']", "OutputFile", "$(SolutionDir)/bin/$(ConfigurationName)/OgreMain.dll"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Release|Win32']/Tool[@Name='VCLinkerTool']", "AdditionalLibraryDirectories", "$(SolutionDir)/dev/sdk/lib/release"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Release|Win32']/Tool[@Name='VCLinkerTool']", "AdditionalDependencies", "$(NOINHERIT) kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib zlib.lib zziplib.lib FreeImage.lib freetype2311.lib"
setXML "VisualStudioProject/Configurations/Configuration[@Name='Release|Win32']/Tool[@Name='VCLinkerTool']", "ProgramDatabaseFile", ""
setXML "VisualStudioProject/Configurations/Configuration[@Name='Release|Win32']/Tool[@Name='VCLinkerTool']", "ImportLibrary", "$(SolutionDir)/dev/sdk/lib/$(ConfigurationName)/OgreMain.lib"
deleteFile("CMakeLists.txt")
deleteFile("ogremain.map")
changePath(".h", "OgreMain\\include", "../../../../sdk/include/ogre")
changePath(".h", "OgreMain\\src", "../src")
changePath(".cpp", "OgreMain\\src", "../src")
changePath("OgreBuildSettings.h", nil, "../../../../sdk/include/ogre\\")
changePath("OgreWin32Resources.rc", nil, "../src/WIN32\\")
saveXML "../dev/components/ogre/OgreMain/scripts/ogremain.vcproj"
end

This one is a LOT bigger too, over 600 lines (it's also got bullet, falcon, every ogre plugin, etc). Took a while to write, but it saves so much time when I upgrade something.

[Klaim]

Kojack, I guess it's too late for you to even try it, but did you consider PreMake? If yes, maybe you played with it and can give us some feedback? Premake seems to match your requirements, that's why I ask.

[Kojack]

Hmm, Premake uses lua? Yay.
Although ruby has better xml and string handling.

Like cmake, premake seems to be intended for multiplatform development, whereas I'm (currently) only working with windows.

[Klaim]

Oh ok then Ruby or Python are enough I guess. I was using Python only for build scripts (with VS solution/projects files) when working only on Windows.