Announcement

Collapse
No announcement yet.

Git and Binary Files

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Git and Binary Files

    I am a long time user of SVN and while it has worked for me in the past, I always felt like there were features I wasn't using properly. Branching is one of those features. Most of the time when I created branches, I was really creating tags because when I created branches, I never really used them for merging, just more to take a snapshot of the current code. When I did have to make changes to code in one branch and merge them to another, I always just manually merged the files because it was usually 1 or two files and merge with SVN from what I hear is not a straight forward process. I have recently starting using merging at work with SVN and I now really understanding why people try to avoid it in SVN. Another thing I do in SVN that I fell is counter productive is I only commit when I am 100% complete with a feature. It would be great to commit things in chunks instead of one great big change. I have had multiple times where I was like "I wish I could go back just a little bit in time" but of course I have to go back to the initial version which may be a week old or modify the file manually to get back to the state I want it at (which it was I usually do).

    Thankfully I have been introduced to the world of DVCS, specifically git. My work is in the process of porting over to it and as I have been learning it, I am wondering why I am still using SVN. Git fixes the above issues as well as also having a number of cool little features.

    Now while it fixes a lot of things there is one feature that concerns me with trying to using it for game development projects and that is how it handles binary files. In my day job, that is not a huge concern as I do web development and I don't deal with many binary files that need to be versioned. Images are the only ones and they rarely change and are not usually that big. Game development is a different story. Game development in general has many binary files (images, 3d assets, sound files, etc...). Since git generally has a copy of the entire binary file for each change that happens to it and since these assets can change quite often and sometimes be large in size, I have a feeling that it can make the repository of project become huge even with the compression that git does with git pack/git gc.

    Does anyone use git with game development? What is the workflow for binary assets? Do you just store them in git (if so, are they in the main repository or a separate repository using submodules)? If you don't include binaries in the git repository, what tool do you use to manage binary assets and does it work well with git?

    #2
    Until I had access to an unlimited svn account, I used to use git for my Development\Src folders, and subversion for project Content folders. svn is sort of a pig on binary files, but from what I hear, git is far worse. git's scalability for use with large files (far larger than Linus ever uses when dealing with kernel source ) is pretty questionable, from what I'd last read, although there may have been advances made in the years since that was for sure the case.

    I hear that Perforce deals with binary files in a much better fashion, and perforce does have a 2-user free use license, but I've never actually got around to trying it out, as I already use both subversion and git in my workflows for other products, and have not had a need to work with Perforce.

    Comment


      #3
      We use Mercurial, and it seems to handle the binary files OK. If you like SVN, Mercurial is basically distributed SVN.

      The total size of the files in my working dir is 300MB, and the repo itself is 366MB, with 334 revisions so far (including a couple of UDK version upgrades). We decided to not commit the entire UDK to the repo, but only the source files, flash files, config files, and any assets files we intended to add or change from base UDK.

      The main issue with any sort of source control over binary files is handling merges after concurrent changes. I wish there were some form of tool I could use to diff and merge UPKs, but I haven't found one yet.

      So, our workaround is to have Mercurial back up the local changes when choosing to pull in a remote changed copy of a package, then resolve the differences manually in UnrealEd. This process is painful, and I wish it could be better.

      My main gripe with Mercurial is that the GUI tools are rather terrible. We use TortoiseHG, which is improving, but still seems to ignore all basic tenets of UI design.

      Comment


        #4
        Any source control system worth its salt should support some kind of exclusive checkout (the way SourceSafe works).

        I know Perforce and SVN do support it, I don't know about git and Mercurial, but that concept seems to be clashing with the idea of distributed versioning

        Comment


          #5
          Originally posted by Wildebeest View Post
          We decided to not commit the entire UDK to the repo, but only the source files, flash files, config files, and any assets files we intended to add or change from base UDK.
          That's also what my team does. Works out okay.

          One tip I'd add, though: supposing there's some UDK file that you decide to start changing? Probably a good idea to commit it before making changes to it. It's been very annoying when we've decided to make changes to some config file, and then realize it wasn't under version control. Hard to figure out what was changed.

          I hear Perforce integrates well with UDK (and possibly can do versioning at a per-asset level, rather than per-package?), but of course the only free version has a limit of two users.

          Git seems great for source code, but I'm dubious of how well it would handle large binaries. Multiple repos (in Git or otherwise) might be something to try.

          Comment


            #6
            Originally posted by Delehal View Post
            I hear Perforce integrates well with UDK (and possibly can do versioning at a per-asset level, rather than per-package?)
            Nope, that's per package (levels included), as Perforce only deals with files (like the rest), and you only get one file per package.

            Comment


              #7
              We do use SVN and commit everything, including full binaries, for the projects we are working on. It makes build upgrading and various other things much easier, and when you're working with people off-site or team members who aren't technically minded, it makes things much easier.
              I'd like to try Perforce or something else, but often alot of artists struggle with the basics of SVN, which is arguably the simplest of CVS's

              Comment


                #8
                Originally posted by Delehal View Post
                That's also what my team does. Works out okay.

                One tip I'd add, though: supposing there's some UDK file that you decide to start changing? Probably a good idea to commit it before making changes to it. It's been very annoying when we've decided to make changes to some config file, and then realize it wasn't under version control. Hard to figure out what was changed.

                I hear Perforce integrates well with UDK (and possibly can do versioning at a per-asset level, rather than per-package?), but of course the only free version has a limit of two users.

                Git seems great for source code, but I'm dubious of how well it would handle large binaries. Multiple repos (in Git or otherwise) might be something to try.
                That might actually be something worth trying, although I'm not sure how it would be any different from having a single repo. I mean having like multiple repos (one for the Binaries and Engine folder, one for the Development folder, one for the UDKGame folder, perhaps) and having a master repo that has those all listed as externals.

                Reminds me, I do need to learn how to operate SVN with externals, as that will be an easy way to integrate some code to multiple things.

                Comment

                Working...
                X