README.md

# catlader/gitlab-ci

- [App Configuration](APP_CONFIGURATION.md)
- [Values and env vars handling](./VALUES_HANDLING.md)
- [Troubleshoot](./Troubleshoot.md)

## Supported platforms

This repository contains all gitlab-ci include files:

- `node-kubernetes`: include for node, nextjs and nestjs apps
- `meteor-kubernetes`: include for meteor apps
- `rails-kubernetes`: include for rails apps
- `static-js-kubernetes`: include for js project with static export (SSG)
- `panter-kubernetes-base`: use this for anything else that has not yet an include (java, etc.)
- `monorepo`: use this if your app contains multiple apps

## Usage

```
# on top of your gitlab-ci.yml

include:
  - project: catladder/gitlab-ci
    ref: v1
    file: <include to use>

```

E.g for node apps:

```
include:
  - project: catladder/gitlab-ci
    ref: v1
    file: node-kubernetes.yml

```

We recommend to use a certain version as ref. Without a ref, it will always take master (latest) which might break your builds.

You need to provide the following variables in your gitlab-ci.yml:

```
variables:
  CUSTOMER_NAME: pan
  APP_NAME: awesomeapp
  CLUSTER_NAME: production
  COMPONENT_NAME: web # optional, if you want to have multiple services/apps into the same namespace, this needs to be different

  APP_DIR: . # path where your app leaves (package.json) relative to the git-root
```

**Many features can be configured in values**

**[Read this documentation about supported values](APP_CONFIGURATION.MD)**

## General Features

This includes defines the following:

- test, app-build and docker-build
- deploys to kubernetes
- deploys to review environment on every branch (review apps, one per Branch)
- deploys to dev environment on every master commit
- you can trigger a `semantic-release` manually, which will tag the repository and deploys to prod or stage environment.
- you can manually deploy to prod from stage
- supports monorepos (See below)

## Monorepo usage

We support repos containing multiple sub-apps. We call these subapps "components".

Put each of your subapp in a subfolder with the `COMPONENT_NAME` as the folder name.
Each of this subfolders has to contain a .gitlab-ci.yml with any of the other includes as well as values files.

Its best to define variables like CUSTOMER_NAME and APP_NAME in the parent gitlab-ci.yml and omit these in the sub-apps

### Example

Consider a project that has a frontend and an api subfolder. Use this .gitlab-ci.yml for the root folder:

```
include:
  - project: catladder/gitlab-ci
    ref: v1
    file: monorepo.yml

variables:
  CUSTOMER_NAME: pan
  APP_NAME: my-first-monorepo-app

```

Let's assume that both frontend and api are node apps. They both have a .gitlab-ci.yml like this:

```
include:
  - project: catladder/gitlab-ci
    ref: v1
    file: node-kubernetes.yml


```

This will run subpipelines for the `frontend` and `api` folders.

### use variables from other apps

You will most definitly require that one of your apps access the other app.
This is usually achieved by pointing an env var to the `ROOT_URL` of another app.

Let's assume you have a `frontend` and a `api` sub app, here is how you do it:

in `frontend` where you want to define the `API_URL` add this to the values:

```
# values.yml

env:
  fromComponents:
    api: # the name of the other component
      API_URL: ROOT_URL

```

#### caveats:

- your first deployment might currently fail, because the env vars have to exists in kubernetes, before we can include that like this. We try to find a fix for that
- `fromComponents` has no effect locally (using `catladder catenv` in direnv). Use `values-dev-local.yml` to set them.
- `fromComponents` does not work for secrets, but you can share secrets anyway using the same secrets name.

### release handling

You can create a release manually on the repository, which does a `semantic-release`, creates a changelog, bumps the version number and tags the repository. For this to work **_your commits need to follow semantic versionsing_**: `fix(scope): nasty bug` or `feat(scope): new fancy feature`.

After the repository got tagged, a deploy to prod or stage is done (depending on whether staging is enabled, see below).

In the pipeline, click on "create-release" to create a release tag and trigger therefore a new release.

### hotfixing older release

You might come accross the situatio where you have a tagged version that is about to get released,
but its still getting tested by the customer on staging.

Now, a wild bug occures and you have to fix that on production!

With the normal workflow you would need to fix that, merge it into master and trigger a normal release.
But this will release everything that is on stage right now!

To avoid this, you can do this hotfix flow, which follows [this recipe](https://github.com/semantic-release/semantic-release/blob/master/docs/recipes/release-workflow/maintenance-releases.md):

Let's assume your current "good" version on prod is `v2.4.3`
It's important that this version is the latest `patch` version so (v2.4.4 does not exist). This is usually in praxis, as you want to release bugfixes as soon as possible usually.
So we want to have a new patched version 2.4.4 that will be our new prod :

- create a branch for the hotfix: `git checkout -b 2.4.x v2.4.3
- do your fixes and commit with a `fix:` commit: `fix: very important bug to hotfix`
- push this branch to gitlab
- this will now trigger a special pipeline where you have your usuall lint/test/build and review deploy.
- Additionaly this pipeline contains a "create-release"-step. Run this job to create the new hotfix release 2.4.4.
- this will in turn run a pipeline that let's you deploy it to prod and stage. its not done automatically. Chose the right destination for your hotfix.
- pull the branch again locally (if you want to have it in the changelog file) and merge it into master
- this might lead to conflicts now if the hotfix no longer applies, but that's ok! Maybe it was already fixed in some stage feature.

### enable staging

if you set `STAGING_ENABLED: "true"` in your `variables`, it will deploy to stage first when doing a release. You can then do the prod release manually.

### automatically release

if you set AUTO_RELEASE: "true", commits on master automatically creates a release and therefore deploy to prod. This is good for continues deployment.

Note: If you also set STAGING_ENABLED: "true", the automatic release will deploy to stage, so you have to deploy to prod manually.

### automatic cleanup

By default your review environment will selfdestruct after _2 weeks_
You can not override this setting put pin the environment in the environment overview

## Post deploy stages

you can implement `verify-<env>` stages that will run after deployment. You can e.g. run test against the deployed version.

_Cypress example_:

```


.e2ebase:
  image: cypress/browsers:node12.18.3-chrome89-ff86
  extends: .cache-node-modules
  script:
    - yarn install --frozen-lockfile
    #- yarn cypress install
    - CYPRESS_BASE_URL=$CI_ENVIRONMENT_URL yarn cypress:run --browser chrome --headless
  artifacts:
    when: always
    paths:
      - cypress/videos/**/*.mp4
      - cypress/screenshots/**/*.png
    expire_in: 5 day

verify-review:
  extends:
    - .verify-review
    - .e2ebase

verify-dev:
  extends:
    - .verify-dev
    - .e2ebase


```

## Platform specific includes

### kubernetes-node

```

include:
  - project: catladder/gitlab-ci
    ref: v1
    file: kubernetes-node

```

this assumes the following scripts to be available:

- `yarn build`: invoked during build stage. Should produce a build in `./dist` (or in next-app it builds in `./.next`)
- `yarn lint`: lints your app
- `yarn test`: tests your app (usually unit tests)

This include automatically generates a Dockerfile unless you specify one. You can control the node-version by
adding a `.nvmrc` file. It will use the "alpine" version of the specified node version

If you want to add additional instructions to the default Dockerfile, set the variable `DOCKERFILE_ADDITIONS`:

```yaml
variables:
  DOCKERFILE_ADDITIONS: |
    RUN apk add ....
    RUN other command...
```

you can add also commands at the end with `DOCKERFILE_ADDITIONS_END`.

Example project: https://git.panter.ch/catladder/test-projects/ssr-next

### meteor-kubernetes

```yaml
include:
  - project: catladder/gitlab-ci
    ref: v1
    file: meteor-kubernetes.yml

variables:
  # meteors specific:
  MONGODB_ENABLED: "true" # whether to create a mongobd
  WORKER_ENABLED: "false" # whether to launch an additional pod which has WORKER_ENABLED env var. usefull for cronjobs and migrations
  MONGODB_REPLICAS: "3" # how many mongodb-replicas it will launch on production. other envs have always 1
```

This works similarly to `node-kubernetes`, but adds a meteor specific dockerfile

### static-js-kubernetes

```yaml
# .gitlab-ci.yml
include:
  - project: catladder/gitlab-ci
    ref: v1
    file: static-js-kubernetes.yml
```

This include assumes build, lint and test commands in your package.json.
`build` should export the build into `dist` folder. If the staging and/or production build needs a different command, set the variables `STAGE_BUILD_COMMAND` & `PROD_BUILD_COMMAND`:

```yaml
variables:
  STAGE_BUILD_COMMAND: "yarn build-stage"
  PROD_BUILD_COMMAND: "yarn build-prod"
```

It will create a nginx docker image with a default config.
You can replace the default config by adding a `nginx.conf` file into your project. If you need to adjust it,
please raise an issue here, so that we might figure out better defaults (or an easier way to configure)

Example project: https://git.panter.ch/catladder/test-projects/ssg-next/

### rails-kubernetes

This uses [Cloud Native Buildpacks](https://buildpacks.io/) to build the container, similar to how Heroku builds a Rails app. No Dockerfile needed. It also disables the app-build job.

**Important**: the Rails app needs to be stateless. Most importantly it has to:

- Store uploads in Cloud Storage/S3 instead of local disk.
- Store shared state externally.
- Send email with Mailgun instead of the server's SMTP server.
  When it would be able to run in Heroku then it will also work here.

Pipeline configuration:

```yaml
# .gitlab-ci.yml
include:
  - project: catladder/gitlab-ci
    ref: v1
    file: rails-kubernetes.yml

variables:
  # rails specific:
  CNB_ENV_VARS: "MY_ENV_VAR=bla MY_VAR_FROM_ENV" # specify environment variables to be present at container build time, optionally with a value, or else the value is taken from the environment
```

Recommended [values](#values-handling):

```yaml
# values.yml
env:
  public:
    RAILS_ENV: production
    PORT: 8080 # override default port 5000
  secret:
    POSTGRESQL_PASSWORD: app-secrets

application:
  command: ["/cnb/process/web"] # buildpack entrypoint
  livenessProbe:
    httpGet:
      path: /robots.txt # change to your health route
      httpHeaders: # empty
  readinessProbe:
    httpGet:
      path: /robots.txt # change to your health route
      httpHeaders: # empty

cloudsql:
  enabled: true

jobs:
  db-prepare:
    hook: post-install,post-upgrade
    command: "/cnb/lifecycle/launcher bundle exec rake db:prepare"
```

```yaml
# values-review.yml
jobs:
  db-prepare-seed:
    hook: post-install
    command: "/cnb/lifecycle/launcher bundle exec rake db:prepare db:seed"
  db-prepare:
    hook: post-upgrade
    command: "/cnb/lifecycle/launcher bundle exec rake db:prepare"
```

Note: Cloud Native Buildpacks creates these default processes, which are available as `/cnb/process/<process-type>` inside the container:

```
TYPE          SHELL COMMAND
web (default) bash  bin/rails server -p ${PORT:-5000} -e $RAILS_ENV
console       bash  bin/rails console
rake          bash  bundle exec rake
worker        bash  bundle exec rake jobs:work
```

Define additional ones in a `Procfile` like with Heroku.

All other commands have to be prefixed with `/cnb/lifecycle/launcher` to run with the correct environment.

### panter-kubernetes-base

a base image that can be used for anything else.

You can override .lint, .test, .app-build stages to specify mvn commands or similar