Page 1 of 1

Conan scripts for continuous integration and distribution

Posted: Sun Mar 22, 2020 7:25 pm
by KonradNoTantoo
Hello everyone,

To make it short, over the last months I wrote a set of Conan scripts to help manage build and integration for various libs, among them Ogre 1.12.5 and CEGUI 0.8. Now, while still perfectible, they're at a stage other people could be using them.

I'm posting here in the hope other may find my work useful. Most of what I've done so far is accessible on Github: https://github.com/KonradNoTantoo?tab=repositories

Motivation

I've been working on a small C++ project that uses Ogre on my spare time. Over the years, I had many integration issues, some of them with Ogre, some of them with other libs. In particular, upgrading or changing build chains or OS always feels like a huge pain. Moreover, upgrading library versions can also become time-consuming. It's often not only a problem of updating my own project, but as much of figuring out how various dependencies and their dependencies interact.

Binary incompatibilities always happen with pre-built binaries distributed through the web. Rebuilding everything from source is time-consuming and not always easily reproducible on a slightly different setup.

Conan

Conan offers an agnostic build and dependency management system. For each project, Conan permits easily distributing a "build recipe", that is a relatively short python script that aims to be common accros compiler, OS and CPU architecture. My current recipe for Ogre is around 150 lines of well spaced python. Conan provides painless Debug/Release build management. Last but not least, Conan recipe integrate well with free (to use) continuous integration plqtforms like Travis and Appveyor and binary distribution platforms like Bintray and Artifactory.

How it works

To start, when working with only one system and compiler, it's easier to think of Conan as a binary distribution tool like Debian's apt or Redhat's yum. However, it can do more than that: It can handle binary packages for a whole bunch of OS and compiler versions. And if you need a library for some setup that's not readily available online, the library's Conan recipe can be used to build from source.

Conan recipe for a project contain rules to build from source. Conan integrates well with CMake and a bunch of other build chains. For example, the recipe linked above for Ogre declares a bunch of dependencies, fetches Ogre 1.12.5 sources from Github, injects a few lines into Ogre's CMakeLists.txt then delegates to CMake for configuration, build and install.

Once a build is finished, its resulting packaged binaries can be associated to the original recipe on a remote Conan repository. Anyone with the same build environment can then reuse that same package. When building on a CI platform, the result is automatically pushed to a specified Conan repository.

Those same CI platforms can detect each new push to Github and trigger a build from there. When all goes well, a single git push with the right tag or on the right branch can generate everything needed to release and distribute a library.

For library developers

For developers, it also means that each non-released push triggers a build on a bunch of different platforms, giving some security that the current changes break nothing for a platform not readily available to that developer.

The recipe can also incorporate unit tests (very easily if they're already integrated into CMake's CTest) and usually performs a build test to ensure a produced library links well on all CI platforms.

For library users

Conan is pretty easy to learn for users: go see their docs.

At the beginning, Conan consumes a conanfile.txt the same way CMake consumes a CMakeLists.txt.

Example conanfile.txt:

Code: Select all

[requires]
ogre3d/1.12.5@utopia/testing
ogre-procedural/0.3.200322@utopia/testing
zlib/1.2.11@conan/stable
gtest/1.8.1

[options]
ogre:with_cg=True

[generators]
cmake
To continue with the above example, after setting up Conan and a few remote repositories, an end-user project building with CMake and depending on Ogre (among others) could use binaries generated by CI tools based on my recipes. The project needs only:
  • Include a short text file named conanfile.txt,
  • Pass it to conan install command to generate a conanbuildinfo.cmake file containing CMake code (that enables finding all headers and liking against libs of all those dependency)
  • Include two lines of code at the top of their CMakeLists.txt;

    Code: Select all

     include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
     conan_basic_setup()
Utopia

Utopia is my current repository on Bintray that hosts packages generated by Travis (Linux/gcc) and Appveyor (Windows/Visual Studio).

Once Conan is installed, you can make it aware of my binary repository using the following command line:

Code: Select all

conan remote add utopia https://api.bintray.com/conan/konradnotantoo/utopia
Project status

I don't yet consider those recipes I wrote to be stable. I'm still a beginner with Conan. I'm more than willing to offer support in this thread or through Github issues for anyone who wants to reuse my work, has questions or needs help.

My current recipe only support a subset of options available to build Ogre. In the future, I'd like to be able to add support for Apple and Android builds, if feasible. Any help, of course, would be welcome.

Conclusion

Hopefully, this all will help others who, like me, would like to minimize tedious integration work, liberating some time for other tasks.

Re: Conan scripts for continuous integration and distribution

Posted: Sun Mar 22, 2020 7:59 pm
by paroj
you might want to talk with the rigs of rods guys, which are also using conan to package various ogre stuff:
https://github.com/AnotherFoxGuy/conan-OGRE
https://github.com/RigsOfRods/rigs-of-rods

if you can work out something that works for both of you, we can add a conanfile.py to the main repo and fix this:
https://github.com/OGRECave/ogre/issues/593

for the recommended build options, see this cmake script:
https://github.com/OGRECave/ogre/blob/m ... uild.cmake

it is used to generate the official SDK packages.

Re: Conan scripts for continuous integration and distribution

Posted: Mon Mar 23, 2020 8:28 am
by KonradNoTantoo
Thank you, paroj, for the answer and the links. I've contacted AnotherFoxGuy for now.

Your answer got me thinking a bit.
if you can work out something that works for both of you, we can add a conanfile.py to the main repo and fix this:
https://github.com/OGRECave/ogre/issues/593
I think there's a need for clarification here. What I've been doing until now, is trying to answer the need expressed by issue #593. I believe I already have a pretty strong, yet perfectible answer to that. An out-of-repository, library user oriented, recipe as the one I have, can do the job.

Thinking about it, I realize I've been concentrating on such an out-of-repository, library user oriented, recipe, because it was my own need. An in-repository, developer and CI oriented, recipe may well be written differently.

My current recipe ensures Ogre 1.12.5 is pre-compiled and easily distributed to users. A main repo recipe could ensure rebuild, unit tests and linkage tests at each push + delivery and distribution when a tag or branch with a specific format is pushed.

I hadn't planned to put much effort on a CI recipe, but if there is interest, by Ogre devs, I will give it a try. So, the question I have for you is: Do you think there's a need for current Ogre developers to use Conan for CI and/or setting up dev environment? I have a good idea of my needs as a user, but no vision of what's useful for devs.

With a CI recipe, another advantage I see is delegating currently wrapped dependencies (FreeImage, zziplib) to Conan.

Disadvantages I see are coupling a project with Conan, which entails writing recipes for all current and future dependencies, when they don't already exist, and then maintain them. This adds a one-time cost of entry when integrating and some effort when upgrading a dependency, but also forces at least some devs to learn yet another tool: Argument could be made this is superfluous, not worth the effort, things already work as is, etc. This is a kind of argument I can hear. We all have limited time to consecrate to open source, and it's not like everyone has a moral obligation to do Conan just because it's the new thing.

Whether it's worth the effort really depends on developers day-to-day workflow, pain points and variety of their development and target platforms, which I have no idea about.

tl;dr: Fishing for feedback

Re: Conan scripts for continuous integration and distribution

Posted: Wed Mar 25, 2020 4:04 pm
by paroj
KonradNoTantoo wrote:
Mon Mar 23, 2020 8:28 am
I hadn't planned to put much effort on a CI recipe, but if there is interest, by Ogre devs, I will give it a try. So, the question I have for you is: Do you think there's a need for current Ogre developers to use Conan for CI and/or setting up dev environment? I have a good idea of my needs as a user, but no vision of what's useful for devs.
We already have a CI recipe (CMake based) in place which brings up the dependencies on all platforms and executes the tests.

However, there is obviously user interest in having a Conan recipe as well. Therefore your user centric approach is probably just right. Having it in the main repo would make it more discoverable and give others a verified starting point for adding it to their Conan pipeline.

I am thinking about putting it in the Other/ folder, while explicitly mentioning it in the building guide.
KonradNoTantoo wrote:
Mon Mar 23, 2020 8:28 am
Disadvantages I see are coupling a project with Conan, which entails writing recipes for all current and future dependencies, when they don't already exist, and then maintain them. This adds a one-time cost of entry when integrating and some effort when upgrading a dependency, but also forces at least some devs to learn yet another tool: Argument could be made this is superfluous, not worth the effort, things already work as is, etc. This is a kind of argument I can hear. We all have limited time to consecrate to open source, and it's not like everyone has a moral obligation to do Conan just because it's the new thing.
Yes, exactly that. This is why I would neither make Conan the default path nor switch the CI to Conan (the platform specific solutions must continue to work).

Some people with similar thoughts even written a Wrapper for this purpose, but it does not really solve the inherent problem:
https://vector-of-bool.github.io/2018/10/15/pmm.html