Advanced gitops bootstrapping

In this chapter, you are going to read about techniques to structure gitops repositories for infrastructure and application deployments on multiple clusters and environments.

The truth is, you have to make several decisions when you set out to implement gitops.

Decisions related to implementing gitops

No standards emerged yet on how to structure your gitops repository, so you have to weigh tradeoffs as you settle on your structure:

• will you have one central repository or many?
• how do you split repositories?
• how to organize folders inside the repository?
• how to model clusters, environments, teams, namespaces, apps?

In this chapter, you will read about best practices we promote at gimlet.io, but bear in mind that there is no one-size-fits-all solution.

info

In this chapter, you will see bootstrapping commands done with Gimlet CLI, but the concepts are still generally applicable

Model environments and apps rather than clusters and namespaces

First and foremost, let's narrow our problem space. At gimlet.io, we avoid modelling the shape of our clusters. Instead, we work with logical environments that can be mapped to clusters or namespaces within clusters.

This abstraction allows reshaping clusters without having to deal with gitops changes. Flux has a large part in this process, making it possible for a cluster or namespace to assume any logical environment.

Use repos and folders for separation

We use repositories and folders to separate envs and apps.

You typically want to separate one environment from another, staging from production, for example, and you want to separate app configs as well. You can either use separate git repositories or separate folders for this purpose.

Using folders

In simple scenarios, you can model everything in a single repository and use folders to separate staging from production and one application from another.

We use the $env/$app folder structure for these cases.

Pros

• simple to manage

Cons

• no fine-grained access control possibilities

This scales well up to a handful of environments with a couple hundred deployments within each. This approach can work for teams below 10, where responsibilities have not crystallized yet.

Using repos

When we want access control, we split environments into different git repositories and use a naming convention to name the repos like gitops-$env. Inside the repositories, we have folders to separate apps from one another.

Besides separating environments, we sometimes separate teams or departments into different git repositories. But we almost always split infrastructure components from applications we develop, as the devops separation happens much sooner than departmental separation. We use a gitops-$env-infra and gitops-$env-apps convention to name the repos separated this way.

Pros

• access control
• separating different concerns

Cons

• tooling may not support well enough a high number of repos

This scales well up to a handful of environments with a couple hundred deployments within each. This approach can work for teams sized 10-50, where distinct functions within the team are known.

warning

We never use branches to separate environments. On paper, you could open a pull request from your staging branch to the production branch to promote changes, but in reality, the branches will hold environment specific information sooner or later, and the branches start living their own lives. Why not use repos and folders then from the start?

Bootstrapping a single repo for apps and a single repo for infra setup

We usually start out with a single git repository and structure it with folders. But even then, we separate infrastructure components and applications we develop, and use a gitops-infra and gitops-apps convention to name the repos. Just like when we join a company to bootstrap gitops, we have a distinct devops function with greater access than application developers.

In this case, we have two repos, gitops-infra and gitops-apps, and an $env/$app folder structure in each. (In the infra repo, $app means an infrastructure component of course)

Bootstrapping the gitops-infra repo

This git repository will hold infrastructure components like the ingress controller, observability tools, and so on in an $env/$app app folder structure.

Let's bootstrap two environments right away:

$ gimlet gitops bootstrap \
  --env staging \
  --gitops-repo-url git@github.com:mycompany/gitops-infra.git

$ gimlet gitops bootstrap \
  --env production \
  --gitops-repo-url git@github.com:mycompan>/gitops-infra.git

After the previous step, the folder structure will look like this:

.
├── production
│   └── flux
│       ├── deploy-key-production-mycompany-gitops-infra-production.yaml
│       ├── flux.yaml
|       └── gitops-repo-production-mycompany-gitops-infra-production.yaml
└── staging
    └── flux
│       ├── deploy-key-staging-mycompany-gitops-infra-staging.yaml
│       ├── flux.yaml
|       └── gitops-repo-staging-mycompany-gitops-infra-staging.yaml

Now both environments have Flux manifests that sync the staging and production folders respectively.

Bootstrapping the gitops-apps repo

This repository will hold application manifests in an $env/$app structure.

Since Flux is installed on the environment in the gitops-infra repo to sync this repository, we don't need to install Flux anymore, we just need to add a new sync target to Flux. To keep Flux configs close to each other, let's do that in the gitops-infra repo instead. After all, gitops automation is an infrastructure component.

Notice the --no-controller flag when we run the bootstrap process again. This ensures we don't bootstrap Flux again, just the new sync sources.

$ gimlet gitops bootstrap \
  --env staging \
  --no-controller \
  --gitops-repo-url git@github.com:mycompany/gitops-apps.git

$ gimlet gitops bootstrap \
  --env production \
  --no-controller \
  --gitops-repo-url git@github.com:mycompany/gitops-apps.git

.
├── production
│   └── flux
│       ├── deploy-key-production-mycompany-gitops-infra-production.yaml
│       ├── deploy-key-production-mycompany-gitops-apps-production.yaml
│       ├── flux.yaml
|       ├── gitops-repo-production-mycompany-gitops-infra-production.yaml
│       └── gitops-repo-production-mycompany-gitops-apps-production.yaml
└── staging
    └── flux
│       ├── deploy-key-staging-mycompany-gitops-infra-staging.yaml
│       ├── deploy-key-staging-mycompany-gitops-apps-staging.yaml
│       ├── flux.yaml
|       ├── gitops-repo-staging-mycompany-gitops-infra-staging.yaml
│       └── gitops-repo-staging-mycompany-gitops-apps-staging.yaml

At this point, gitops-infra is Flux bootstrapped with both the gitops-infra and the gitops-apps repo and it is synced to both the staging and the production environment.

Bootstrapping a repo per env structure with split infra and app repos

When we use separate repos for envs while also separating the infra and the apps repos, we use a gitops-$env-infra and gitops-$env-apps naming convention. Which means four repos with staging and production environments:

• gitops-staging-infra
• gitops-production-infra
• gitops-staging-apps
• gitops-production-apps

In each repo, we use a folder structure to separate apps from one another, resulting in an $apps folder structure. We can achieve this with the --single-env flag in gimlet gitops bootstrap.

We will stick to bootstrapping gitops for both the infra and the apps repos in the infra repos, therefore, we need to bootstrap the infra repos with:

$ gimlet gitops bootstrap \
  --single-env
  --gitops-repo-url git@github.com:mycompany/gitops-$env-infra.git

and the apps repos with:

$ gimlet gitops bootstrap \
  --single-env
  --no-controller \
  --gitops-repo-url git@github.com:mycompany/gitops-$env-apps.git

This results in a

.
├── flux
│   ├── deploy-key-mycompany-gitops-$env-infra.yaml
│   ├── deploy-key-mycompany-gitops-%env-apps.yaml
│   ├── flux.yaml

structure in each gitops-$env-infra repo.

tip

The bootstrapping process requires a few gimlet CLI commands. Gimlet Dashboard have these presets automated, so you should be able to get started just by choosing the gitops repo strategy.