Fixing Git dependency hell for C/C++ with Bake Bundles

http://eldjazairmag.com/wp-content/uploads/2016/02/

Working with projects that need multiple Git repositories can feel a bit like herding cats. For every breaking API change, chances are multiple repositories need to be updated. If this happens somewhat regularly, mistakes are bound to happen. Even if you are the sole developer of all those repositories, keeping them in a state where somebody else can just clone and build them is a chore.

I have been on the receiving end of this problem multiple times with Flecs, an entity component system for C99, that consists out of a main repository and a handful of “modules”, which live in their separate repositories. Whenever I posted a Flecs example on Reddit, it was all but guaranteed that I would receive an issue a few weeks or months later about its build failing.

Two weeks ago I decided to do something about this, and design a new bake feature called “bundles”, which are collections of repositories and revisions. I had some constraints in mind for the design:

• There should be no need for supporting infrastructure other than git
• Users should be able to define their own dependencies and revisions
• Users should be able to install/run a project with a single command, directly from a git repository
• The installed revisions should be deterministic, and it should be easy to visualize exactly what will be installed
• There should be no (potential for) sharing of data with 3rd parties
• Configuration should be DRY and shareable between projects

So what are bundles? The idea of a bundle is that developers create a JSON file with a set of repositories and revisions that are known to work well together. This can then provide a stable baseline for applications building against a set of repositories. Let’s look at a basic bake project file first:

{
  "id": "apple_pie",
  "value": {
    "use": ["cinnamon"]
  }
}

There is a problem with this file. Bake has no idea where cinnamon can be found, so unless a user has cloned this repository, bake won’t be able to build apple_pie. With bundles, we can now address that by adding a bundle section to our project:

{
  "id": "apple_pie",
  "value": {
    "use": ["cinnamon"]
  },
  "bundle": {
    "repositories": {
      "cinnamon": "https://github.com/SanderMertens/cinnamon"
    }
  }
}

Now we can simply run our project, and bake will automatically clone the cinnamon repository if it wasn’t already installed. However, we still have not specified which revision of cinnamon we want to use. We can do that by adding the refs section to the configuration:

{
  "id": "apple_pie",
  "value": {
   "use": ["cinnamon"]
  },
  "bundle": {
    "repositories": {
      "cinnamon": "https://github.com/SanderMertens/cinnamon"
    },
    "refs": {
      "default": {
        "cinnamon": {
          "tag": "v1.0"
        }
      }
    }
  }
}

This ref instructs bake to checkout tag v1.0 when it clones the repository for cinnamon. We could also specify the branch we want, and if we want to specify a commit SHA instead of a tag we can do that as well.

Such configurations can get unwieldy, especially if there are lots of repositories and revisions. To make sure it does not have to be repeated in every project, bake allows bundles to be defined in their own project, like this:

{
  "id": "baking_bundle",
  "type": "package",
  "value": {
    "language": "none"
  },
  "bundle": {
    "default-host": "https://github.com"
    "repositories": {
      "apple": "SanderMertens/apple",
      "sugar": "SanderMertens/sugar",
      "cinnamon": "SanderMertens/cinnamon"
    },
    "refs": {
      "v1.0": {
        "default-tag": "v1.0"
        "sugar": {
          "commit": "d147bb3"
        }
      },
      "v2.0": {
        "default-tag": "v2.0"
      }
    }
  }
}

Note that where we had a default bundle in our previous project, we now have two bundles, v1.0 and v2.0. For each bundle all repositories will be checked out with tags v1.0 and v2.0 respectively, except for the sugar project which in bundle v1.0 needs to be checked out with a specific SHA.

Back in the apple_pie project, we can now change our project file to this:

{
  "id": "apple_pie",
  "value": {
    "use": ["cinnamon", "sugar", "apple"],
    "use-bundle": ["baking_bundle:v1.0"]
  },
  "bundle": {
    "repositories": {
      "baking_bundle": "https://github.com/baking/baking_bundle"
    }
  }
}

I still added a bundle configuration so that bake will be able to find the baking_bundle project, in case it is not available locally. This way we ensure that our project can always be cloned with the right repositories and revisions for our dependencies, no matter the state of our environment!

Last but not least, you can tell bake to add a bundle to its environment. This will ensure that your environment remains in the state that is specified in the bundle. If a project attempts to clone a repository that is of a different revision than the one specified in the bundle, bake will reject it. To configure bake to use a specific bundle, simply do:

bake use baking_bundle:v1.0

If you would like to see what is contained in this bundle, you can run:

bake list --show-repositories

And the output will look something like this:

In this example, I have loaded the bundle flecs.hub:default which contains the Flecs project and modules. The orange repositories are in the bundle, but not installed locally. To install one, I can simply do:

bake install flecs.systems.physics

I have just started using bundles, and it is already a huge quality of life improvement. Over time I’m sure I’ll come up with more usecases that will more deeply integrate bundles with bake’s features!

If you’d like to know more, check documentation in the bake repository:

https://github.com/SanderMertens/bake

Thanks for reading! If you have ideas or suggestions to bake bundles better, let me know!