Fixing the big commit of the "Volume Rendering" project

Threads related to Google Summer of Code
Post Reply
User avatar
Mattan Furst
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 260
Joined: Tue Jan 01, 2008 11:28 am
Location: Israel

Fixing the big commit of the "Volume Rendering" project

Post by Mattan Furst » Wed Aug 22, 2012 11:11 pm

This topic started here.

Philip
Don't do anything for now.
Any other change will only increase the size of the terrain

Assaf
I found and succesufully tested a way to do this, but it has repercussion

1. save to the side the changes of "e09c55ac19a0 Volume Rendering: added volume media"
2. rebase: "b36038f3bb7e Volume Rendering: added the volume sample" to "Volume Rendering: added the Volume component"
3.push the change into the repository
4. use bitbucket's website strip functionality on "e09c55ac19a0 Volume Rendering: added volume media"
5. Add a new commit containing the files from "e09c55ac19a0 Volume Rendering: added volume media" without the large file.

The repercussion are that this creates an entire new branch as far as mercurial is concerned from "Volume Rendering: added the Volume component" onwards.
anyone that forked the database will merge to the old branch. and future commits, if we are not carefull, can re-push the "e09c55ac19a0 Volume Rendering: added volume media" changeset.
If this is agreed upon it needs to be done fast. afterwards everyone have to do a get latest and remove any trace of the previous branch.
0 x
it's turtles all the way down

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: Fixing Volume Rendering - big commit

Post by Assaf Raman » Thu Aug 23, 2012 6:22 am

Community - what do you think we should do?
0 x
Watch out for my OGRE related tweets here.

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: Fixing the big commit of the "Volume Rendering" project

Post by CABAListic » Thu Aug 23, 2012 6:53 am

Are you sure your steps are correct? Bitbucket's strip will remove the specified commit and ALL commits after that one, so I think you'd have to do that first, then restore the commits after the media addition.

In any case, the repository size increased from around 125 Mb to 198 Mb. If the terrain file is final and does not need to change, I guess it's tolerable. Particularly if there is no good smaller alternative to sample the volume rendering.

I'd vote leave it, but we should reinvestigate for the upcoming 2.0 release - it might be a good time to try and completely remove media from the code repository and store it externally. This would mean to move to a new repository since we'd have to edit history (if possible), and the 2.0 release would be a good time to do that.
0 x

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: Fixing the big commit of the "Volume Rendering" project

Post by Assaf Raman » Thu Aug 23, 2012 7:19 am

CABAListic wrote:... it might be a good time to try and completely remove media from the code repository and store it externally...
I agree. I think we should move all the samples'\tools - to a different repo - to have two repos - "core" and "samples+tools".
0 x
Watch out for my OGRE related tweets here.

User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
Contact:

Re: Fixing the big commit of the "Volume Rendering" project

Post by tuan kuranes » Thu Aug 23, 2012 8:21 am

Not sure to get what's the problem. That file won't change much, so not many version of a 70mb file.
What's 70mb on a HDD now, even multiple times ?

Imho, Samples and media should even be much bigger with the most possible samples and demos... even including wiki tutorials for example (would help keeping them up to date.), externals tools (like ogremeshy,meshmagick, exporters).
My reasoning being: most users don't read manuals/wiki/forums, and if it's not in the source/download, it's not existing at all for them.

And I would vote for keeping it simple: a single full repo. Otherwise that means a lots of wiki/forum work awaits, like how to set up relative folders each to another, etc.

Though, a new repository base folder could be a good idea for 2.0, ogreCore, Samples, Tools, Exporters, etc... so that splitting/cleaning/erasing of unneeded subparts would be easier.
0 x

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: Fixing the big commit of the "Volume Rendering" project

Post by Assaf Raman » Thu Aug 23, 2012 8:48 am

What is the other side?
Does anyone thinks that we can't live with the added 70MB for now?
If so - why not?
0 x
Watch out for my OGRE related tweets here.

User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4291
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 2
Contact:

Re: Fixing the big commit of the "Volume Rendering" project

Post by spacegaier » Thu Aug 23, 2012 8:54 am

Just an idea: Is there an Atlassian open-source support channel where we could get some insights / tips / opinions from them on the topic? They ought to be the experts...
0 x
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: Fixing the big commit of the "Volume Rendering" project

Post by Assaf Raman » Thu Aug 23, 2012 8:59 am

Can you ask and post a link to that thread?
0 x
Watch out for my OGRE related tweets here.

User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4291
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 2
Contact:

Re: Fixing the big commit of the "Volume Rendering" project

Post by spacegaier » Thu Aug 23, 2012 9:02 am

Assaf Raman wrote:Can you ask and post a link to that thread?
You mean ask Atlassian? Although I got the basic problem here we are talking about I am far off from being an expert, so I actually would prefer it if you or another dev member could handle that.

On this page you can find the respective support eMail address: https://bitbucket.org/support
0 x
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: Fixing the big commit of the "Volume Rendering" project

Post by Assaf Raman » Thu Aug 23, 2012 9:09 am

I will tweet Sinbad.
0 x
Watch out for my OGRE related tweets here.

PhilipLB
Google Summer of Code Student
Google Summer of Code Student
Posts: 550
Joined: Thu Jun 04, 2009 5:07 pm
Location: Berlin

Re: Fixing the big commit of the "Volume Rendering" project

Post by PhilipLB » Thu Aug 23, 2012 9:11 am

Specifically according to the file again: It won't change. Before the merge, I made sure, that it is a nice testcase and also renamed it a last time.

To the media folder: I like the idea of separating the media from the source. But that's hardly possible without editing the history a lot. Either separating to a separate zip or with the LargefilesExtension (preferred, http://mercurial.selenic.com/wiki/LargefilesExtension). Or even separating the samples completely from Ogre + Components.

My opinion to the repo size: Personally, for me it doesn't matter. But I'm at the comfortable position of having 50mb/s downstream at home.
0 x
Google Summer of Code 2012 Student
Topic: "Volume Rendering with LOD aimed at terrain"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: Mattan Furst


Volume GFX, accepting donations.

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: Fixing the big commit of the "Volume Rendering" project

Post by Assaf Raman » Thu Aug 23, 2012 9:11 am

0 x
Watch out for my OGRE related tweets here.

User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19261
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
Contact:

Re: Fixing the big commit of the "Volume Rendering" project

Post by sinbad » Thu Aug 23, 2012 9:21 am

When we moved to hg I stripped the repo down so that it was smaller, the main reason was to keep the initial clone time down. The smaller the clone, the easier it is for everyone, both in terms of download speed, and forking speed.

I agree that it would be a good idea to separate the media out, but if it's in the public repository already then it's too late to do that now. Other people will have pulled the changes already, and stripping at this point will just screw them up. If you were going to tidy this up, it had to be done before the upstream repository was updated.

So there's not much you can do now except live with it, unless you create another 'tidied' repository later on. This is why I did the stripping of the initial OGRE hg repo right at the start, before anyone else used it. DVCS's are great but once someone else sees the commits, you pretty much can't change them without causing major hassle.
0 x

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: Fixing the big commit of the "Volume Rendering" project

Post by Assaf Raman » Thu Aug 23, 2012 9:25 am

Thanks Sinbad.

Well, lets live with it, the pain of fixing it is worse then living with it.
0 x
Watch out for my OGRE related tweets here.

TheSHEEEP
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 972
Joined: Mon Jun 02, 2008 6:52 pm
Location: Berlin

Re: Fixing the big commit of the "Volume Rendering" project

Post by TheSHEEEP » Thu Aug 23, 2012 9:37 am

I agree. We haven't yet reached the point where it becomes a major pain. 200 MB is pretty bearable, IMO, considering it includes samples, etc.
0 x
My site! - Have a look :)
Also on Twitter - extra fluffy

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: Fixing the big commit of the "Volume Rendering" project

Post by CABAListic » Thu Aug 23, 2012 10:38 am

That's what I meant with reconsidering for 2.0 - because it's unavoidable to start over with a new repository if we want to strip media history from it, and 2.0 might be a good moment to do that.

And to emphasize: No one cares particularly about the repository size on the disk, but the download size is another matter, because for me it already takes quite some time to checkout a fresh copy of the Ogre repository (and my internet connection is definitely not bad by German standards). So I cannot agree with tuan that we can simply put high-quality media into the main repository and not care about its size.
0 x

User avatar
Kojack
OGRE Moderator
OGRE Moderator
Posts: 7137
Joined: Sun Jan 25, 2004 7:35 am
Location: Brisbane, Australia
x 6

Re: Fixing the big commit of the "Volume Rendering" project

Post by Kojack » Thu Aug 23, 2012 11:03 am

And I would vote for keeping it simple: a single full repo.
Except we are already on two repos (ogre main and dependency) to build ogre. Or three if you want parts of ogre addons.
or with the LargefilesExtension (preferred, http://mercurial.selenic.com/wiki/LargefilesExtension).
Sadly bitbucket doesn't support the large file extension, so it's not an option (even though it should fix all this).
0 x

PhilipLB
Google Summer of Code Student
Google Summer of Code Student
Posts: 550
Joined: Thu Jun 04, 2009 5:07 pm
Location: Berlin

Re: Fixing the big commit of the "Volume Rendering" project

Post by PhilipLB » Thu Aug 23, 2012 11:07 am

They really doesn't support it? Because it seems to be in the core of HG since 2.0.
0 x
Google Summer of Code 2012 Student
Topic: "Volume Rendering with LOD aimed at terrain"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: Mattan Furst


Volume GFX, accepting donations.

dermont
Orc Shaman
Posts: 761
Joined: Thu Dec 09, 2004 2:51 am
x 7

Re: Fixing the big commit of the "Volume Rendering" project

Post by dermont » Thu Aug 23, 2012 11:15 am

CABAListic wrote:That's what I meant with reconsidering for 2.0 - because it's unavoidable to start over with a new repository if we want to strip media history from it, and 2.0 might be a good moment to do that.

And to emphasize: No one cares particularly about the repository size on the disk, but the download size is another matter, because for me it already takes quite some time to checkout a fresh copy of the Ogre repository (and my internet connection is definitely not bad by German standards). So I cannot agree with tuan that we can simply put high-quality media into the main repository and not care about its size.
And we all appreciate your (and initially Sinbads) efforts in trying to keep the size of the main repository as low as possible and providing easy to follow instructions to retrieve from the Ogre repository.

That very few users have raised issues when checking out/updating from the Ogre repository is a testament your good work.
0 x

User avatar
Zonder
Gargoyle
Posts: 1088
Joined: Mon Aug 04, 2008 7:51 pm
Location: Manchester - England
x 5

Re: Fixing the big commit of the "Volume Rendering" project

Post by Zonder » Thu Aug 23, 2012 11:29 am

Bitbukets issue maybe more watchers will help :)

https://bitbucket.org/site/master/issue ... rt-bb-3903
0 x
There are 10 types of people in the world: Those who understand binary, and those who don't...
My Blog - http://www.This post is suspected as SPAM! If you feel otherwise contact a moderator.

User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19261
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
Contact:

Re: Fixing the big commit of the "Volume Rendering" project

Post by sinbad » Thu Aug 23, 2012 11:30 am

My experience with LargeFiles has been that it's somewhat ropey. Kiln has supported it via their own fork of the LargeFiles extension for ages, and I know that Unity use it in their repo. SourceTree's ability to use custom hg extensions was initially down to their requirement here. Still, it can be a little flaky at times because essentially it treats large files a bit like subrepositories, except ones with stripped history. That loose linking can cause problems sometimes when one updates and the other fails or similar.

But, maybe it's improved. You don't need Bitbucket to support LargeFiles to use it, you can just host the files somewhere else. In theory that could be the ogre3d.org server, but right now that would blow our bandwidth. But the option is on the table regardless of what BB do.
0 x

User avatar
Mind Calamity
Ogre Magi
Posts: 1255
Joined: Sat Dec 25, 2010 2:55 pm
Location: Macedonia

Re: Fixing the big commit of the "Volume Rendering" project

Post by Mind Calamity » Thu Aug 23, 2012 1:12 pm

I too think that the samples should be moved to another repository. As the OGRE core is about ~40-45 MB, which is 1/4 of what the whole repository is right now.

But that would require modification to the CMake files for making it an easier build. I was using a previous commit last night, and I wanted to try out the GSoC samples, so I pulled the new code, and given my slow connection it took about 30-40 minutes.

Maybe the media should be separately-hosted, as it's 120 MB by itself.
0 x
BitBucket username changed to iboshkov (from MindCalamity)
Do you need help? What have you tried?
- xavier
---------------------
HkOgre - a Havok Integration for OGRE | Simple SSAO | My Blog | My YouTube | My DeviantArt

FlorianGeorge
Halfling
Posts: 86
Joined: Tue Sep 01, 2009 7:15 pm
Location: Cologne, Germany

Re: Fixing the big commit of the "Volume Rendering" project

Post by FlorianGeorge » Thu Aug 23, 2012 2:10 pm

If the Media and Samples were split off, that would also allow to let it grow by adding high quality assets, more samples etc as tuan suggested.

An issue with Ogre is the lack of a (non external closed-source) toolchain/productive environment that just works out-of-the box, with all sorts of converters, editors (materials, particles, terrain, ...), etc neatly packaged, so having a seperate repository which is allowed to grow large would help with that.
0 x

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: Fixing the big commit of the "Volume Rendering" project

Post by Assaf Raman » Thu Aug 23, 2012 2:18 pm

I think that the idea is clear, the right time for such a change and who is going to lead it - less clear.
0 x
Watch out for my OGRE related tweets here.

User avatar
Mattan Furst
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 260
Joined: Tue Jan 01, 2008 11:28 am
Location: Israel

Re: Fixing the big commit of the "Volume Rendering" project

Post by Mattan Furst » Thu Aug 23, 2012 3:04 pm

Are you sure your steps are correct? Bitbucket's strip will remove the specified commit and ALL commits after that one, so I think you'd have to do that first, then restore the commits after the media addition.
When you do a rebase on "b36038f3bb7e Volume Rendering: added the volume sample" it doesn't do what rebase usually does (probably because of its history). it copies the tree under that changeset onto the target changeset creating 2 copies of the same branch minus the "e09c55ac19a0 Volume Rendering: added volume media" commit.

The strip is than used to remove the original branch.
0 x
it's turtles all the way down

Post Reply