README.md 11.9 KB
Newer Older
Marco Wettstein's avatar
Marco Wettstein committed
1
# catlader/gitlab-ci
Marco Wettstein's avatar
Marco Wettstein committed
2

Marco Wettstein's avatar
Marco Wettstein committed
3
- [App Configuration](APP_CONFIGURATION.md)
Marco Wettstein's avatar
Marco Wettstein committed
4
5
6
7
8
- [Values and env vars handling](./VALUES_HANDLING.md)
- [Troubleshoot](./Troubleshoot.md)

## Supported platforms

9
10
11
12
This repository contains all gitlab-ci include files:

- `node-kubernetes`: include for node, nextjs and nestjs apps
- `meteor-kubernetes`: include for meteor apps
Michael Leu's avatar
Michael Leu committed
13
- `rails-kubernetes`: include for rails apps
14
15
- `static-js-kubernetes`: include for js project with static export (SSG)
- `panter-kubernetes-base`: use this for anything else that has not yet an include (java, etc.)
Marco Wettstein's avatar
Marco Wettstein committed
16
- `monorepo`: use this if your app contains multiple apps
17

Marco Wettstein's avatar
Marco Wettstein committed
18
## Usage
Marco Wettstein's avatar
Marco Wettstein committed
19
20
21
22

```
# on top of your gitlab-ci.yml

Marco Wettstein's avatar
Marco Wettstein committed
23
include:
Marco Wettstein's avatar
Marco Wettstein committed
24
25
  - project: catladder/gitlab-ci
    ref: v1
26
    file: <include to use>
Marco Wettstein's avatar
Marco Wettstein committed
27
28
29

```

30
E.g for node apps:
31
32
33

```
include:
Marco Wettstein's avatar
Marco Wettstein committed
34
35
  - project: catladder/gitlab-ci
    ref: v1
36
37
38
39
    file: node-kubernetes.yml

```

Marco Wettstein's avatar
Marco Wettstein committed
40
41
We recommend to use a certain version as ref. Without a ref, it will always take master (latest) which might break your builds.

Marco Wettstein's avatar
Marco Wettstein committed
42
43
44
45
46
47
48
49
You need to provide the following variables in your gitlab-ci.yml:

```
variables:
  CUSTOMER_NAME: pan
  APP_NAME: awesomeapp
  CLUSTER_NAME: production
  COMPONENT_NAME: web # optional, if you want to have multiple services/apps into the same namespace, this needs to be different
Marco Wettstein's avatar
Marco Wettstein committed
50

Marco Wettstein's avatar
Marco Wettstein committed
51
  APP_DIR: . # path where your app leaves (package.json) relative to the git-root
Marco Wettstein's avatar
Marco Wettstein committed
52
53
```

Marco Wettstein's avatar
Marco Wettstein committed
54
**Many features can be configured in values**
Michael Leu's avatar
Michael Leu committed
55

Marco Wettstein's avatar
Marco Wettstein committed
56
**[Read this documentation about supported values](APP_CONFIGURATION.MD)**
Michael Leu's avatar
Michael Leu committed
57

58
## General Features
Marco Wettstein's avatar
Marco Wettstein committed
59

60
This includes defines the following:
Marco Wettstein's avatar
Marco Wettstein committed
61
62

- test, app-build and docker-build
Marco Wettstein's avatar
Marco Wettstein committed
63
64
65
- deploys to kubernetes
- deploys to review environment on every branch (review apps, one per Branch)
- deploys to dev environment on every master commit
66
67
- you can trigger a `semantic-release` manually, which will tag the repository and deploys to prod or stage environment.
- you can manually deploy to prod from stage
Marco Wettstein's avatar
Marco Wettstein committed
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
- supports monorepos (See below)

## Monorepo usage

We support repos containing multiple sub-apps. We call these subapps "components".

Put each of your subapp in a subfolder with the `COMPONENT_NAME` as the folder name.
Each of this subfolders has to contain a .gitlab-ci.yml with any of the other includes as well as values files.

Its best to define variables like CUSTOMER_NAME and APP_NAME in the parent gitlab-ci.yml and omit these in the sub-apps

### Example

Consider a project that has a frontend and an api subfolder. Use this .gitlab-ci.yml for the root folder:

```
include:
  - project: catladder/gitlab-ci
    ref: v1
    file: monorepo.yml

variables:
  CUSTOMER_NAME: pan
  APP_NAME: my-first-monorepo-app

```

Let's assume that both frontend and api are node apps. They both have a .gitlab-ci.yml like this:

```
include:
  - project: catladder/gitlab-ci
    ref: v1
    file: node-kubernetes.yml


```

This will run subpipelines for the `frontend` and `api` folders.

### use variables from other apps

You will most definitly require that one of your apps access the other app.
This is usually achieved by pointing an env var to the `ROOT_URL` of another app.

Let's assume you have a `frontend` and a `api` sub app, here is how you do it:

in `frontend` where you want to define the `API_URL` add this to the values:

```
# values.yml

env:
  fromComponents:
    api: # the name of the other component
      API_URL: ROOT_URL

```

#### caveats:

- your first deployment might currently fail, because the env vars have to exists in kubernetes, before we can include that like this. We try to find a fix for that
- `fromComponents` has no effect locally (using `catladder catenv` in direnv). Use `values-dev-local.yml` to set them.
- `fromComponents` does not work for secrets, but you can share secrets anyway using the same secrets name.
Marco Wettstein's avatar
Marco Wettstein committed
132

133
### release handling
Marco Wettstein's avatar
Marco Wettstein committed
134

135
136
137
138
You can create a release manually on the repository, which does a `semantic-release`, creates a changelog, bumps the version number and tags the repository. For this to work **_your commits need to follow semantic versionsing_**: `fix(scope): nasty bug` or `feat(scope): new fancy feature`.

After the repository got tagged, a deploy to prod or stage is done (depending on whether staging is enabled, see below).

Marco Wettstein's avatar
Marco Wettstein committed
139
140
141
142
143
144
145
146
147
148
149
150
In the pipeline, click on "create-release" to create a release tag and trigger therefore a new release.

### hotfixing older release

You might come accross the situatio where you have a tagged version that is about to get released,
but its still getting tested by the customer on staging.

Now, a wild bug occures and you have to fix that on production!

With the normal workflow you would need to fix that, merge it into master and trigger a normal release.
But this will release everything that is on stage right now!

151
To avoid this, you can do this hotfix flow, which follows [this recipe](https://github.com/semantic-release/semantic-release/blob/master/docs/recipes/release-workflow/maintenance-releases.md):
Marco Wettstein's avatar
Marco Wettstein committed
152
153
154
155
156
157
158
159
160
161
162
163
164
165

Let's assume your current "good" version on prod is `v2.4.3`
It's important that this version is the latest `patch` version so (v2.4.4 does not exist). This is usually in praxis, as you want to release bugfixes as soon as possible usually.
So we want to have a new patched version 2.4.4 that will be our new prod :

- create a branch for the hotfix: `git checkout -b 2.4.x v2.4.3
- do your fixes and commit with a `fix:` commit: `fix: very important bug to hotfix`
- push this branch to gitlab
- this will now trigger a special pipeline where you have your usuall lint/test/build and review deploy.
- Additionaly this pipeline contains a "create-release"-step. Run this job to create the new hotfix release 2.4.4.
- this will in turn run a pipeline that let's you deploy it to prod and stage. its not done automatically. Chose the right destination for your hotfix.
- pull the branch again locally (if you want to have it in the changelog file) and merge it into master
- this might lead to conflicts now if the hotfix no longer applies, but that's ok! Maybe it was already fixed in some stage feature.

166
167
168
169
170
171
172
173
174
### enable staging

if you set `STAGING_ENABLED: "true"` in your `variables`, it will deploy to stage first when doing a release. You can then do the prod release manually.

### automatically release

if you set AUTO_RELEASE: "true", commits on master automatically creates a release and therefore deploy to prod. This is good for continues deployment.

Note: If you also set STAGING_ENABLED: "true", the automatic release will deploy to stage, so you have to deploy to prod manually.
Marco Wettstein's avatar
Marco Wettstein committed
175

176
### automatic cleanup
177
178

By default your review environment will selfdestruct after _2 weeks_
179
180
You can not override this setting put pin the environment in the environment overview

181
182
183
184
## Post deploy stages

you can implement `verify-<env>` stages that will run after deployment. You can e.g. run test against the deployed version.

Marco Wettstein's avatar
Marco Wettstein committed
185
_Cypress example_:
186
187
188
189

```


Marco Wettstein's avatar
Marco Wettstein committed
190
.e2ebase:
191
192
193
194
195
196
197
198
199
200
201
202
203
204
  image: cypress/browsers:node12.18.3-chrome89-ff86
  extends: .cache-node-modules
  script:
    - yarn install --frozen-lockfile
    #- yarn cypress install
    - CYPRESS_BASE_URL=$CI_ENVIRONMENT_URL yarn cypress:run --browser chrome --headless
  artifacts:
    when: always
    paths:
      - cypress/videos/**/*.mp4
      - cypress/screenshots/**/*.png
    expire_in: 5 day

verify-review:
Marco Wettstein's avatar
Marco Wettstein committed
205
  extends:
206
207
208
209
    - .verify-review
    - .e2ebase

verify-dev:
Marco Wettstein's avatar
Marco Wettstein committed
210
  extends:
211
212
213
    - .verify-dev
    - .e2ebase

Marco Wettstein's avatar
Marco Wettstein committed
214
215

```
216
217
218
219
220
221
222
223

## Platform specific includes

### kubernetes-node

```

include:
Marco Wettstein's avatar
Marco Wettstein committed
224
225
  - project: catladder/gitlab-ci
    ref: v1
226
227
228
229
230
231
232
233
234
235
    file: kubernetes-node

```

this assumes the following scripts to be available:

- `yarn build`: invoked during build stage. Should produce a build in `./dist` (or in next-app it builds in `./.next`)
- `yarn lint`: lints your app
- `yarn test`: tests your app (usually unit tests)

Marco Wettstein's avatar
Marco Wettstein committed
236
This include automatically generates a Dockerfile unless you specify one. You can control the node-version by
237
238
adding a `.nvmrc` file. It will use the "alpine" version of the specified node version

239
If you want to add additional instructions to the default Dockerfile, set the variable `DOCKERFILE_ADDITIONS`:
240

241
242
243
244
245
```yaml
variables:
  DOCKERFILE_ADDITIONS: |
    RUN apk add ....
    RUN other command...
246
247
```

248
249
you can add also commands at the end with `DOCKERFILE_ADDITIONS_END`.

250
251
Example project: https://git.panter.ch/catladder/test-projects/ssr-next

252
253
254
### meteor-kubernetes

```yaml
Marco Wettstein's avatar
Marco Wettstein committed
255
include:
Marco Wettstein's avatar
Marco Wettstein committed
256
257
  - project: catladder/gitlab-ci
    ref: v1
258
259
260
261
    file: meteor-kubernetes.yml

variables:
  # meteors specific:
262
263
264
  MONGODB_ENABLED: "true" # whether to create a mongobd
  WORKER_ENABLED: "false" # whether to launch an additional pod which has WORKER_ENABLED env var. usefull for cronjobs and migrations
  MONGODB_REPLICAS: "3" # how many mongodb-replicas it will launch on production. other envs have always 1
265
266
267
268
```

This works similarly to `node-kubernetes`, but adds a meteor specific dockerfile

269
270
271
### static-js-kubernetes

```yaml
272
# .gitlab-ci.yml
273
274
include:
  - project: catladder/gitlab-ci
275
    ref: v1
276
277
278
279
    file: static-js-kubernetes.yml
```

This include assumes build, lint and test commands in your package.json.
280
`build` should export the build into `dist` folder. If the staging and/or production build needs a different command, set the variables `STAGE_BUILD_COMMAND` & `PROD_BUILD_COMMAND`:
281
282
283

```yaml
variables:
284
285
  STAGE_BUILD_COMMAND: "yarn build-stage"
  PROD_BUILD_COMMAND: "yarn build-prod"
286
287
```

288
289
290
291
292
293
It will create a nginx docker image with a default config.
You can replace the default config by adding a `nginx.conf` file into your project. If you need to adjust it,
please raise an issue here, so that we might figure out better defaults (or an easier way to configure)

Example project: https://git.panter.ch/catladder/test-projects/ssg-next/

Michael Leu's avatar
Michael Leu committed
294
295
296
297
298
### rails-kubernetes

This uses [Cloud Native Buildpacks](https://buildpacks.io/) to build the container, similar to how Heroku builds a Rails app. No Dockerfile needed. It also disables the app-build job.

**Important**: the Rails app needs to be stateless. Most importantly it has to:
Marco Wettstein's avatar
Marco Wettstein committed
299

Michael Leu's avatar
Michael Leu committed
300
301
302
- Store uploads in Cloud Storage/S3 instead of local disk.
- Store shared state externally.
- Send email with Mailgun instead of the server's SMTP server.
Marco Wettstein's avatar
Marco Wettstein committed
303
  When it would be able to run in Heroku then it will also work here.
Michael Leu's avatar
Michael Leu committed
304
305

Pipeline configuration:
Marco Wettstein's avatar
Marco Wettstein committed
306

Michael Leu's avatar
Michael Leu committed
307
308
309
310
311
312
313
314
315
316
317
318
319
```yaml
# .gitlab-ci.yml
include:
  - project: catladder/gitlab-ci
    ref: v1
    file: rails-kubernetes.yml

variables:
  # rails specific:
  CNB_ENV_VARS: "MY_ENV_VAR=bla MY_VAR_FROM_ENV" # specify environment variables to be present at container build time, optionally with a value, or else the value is taken from the environment
```

Recommended [values](#values-handling):
Marco Wettstein's avatar
Marco Wettstein committed
320

Michael Leu's avatar
Michael Leu committed
321
322
323
324
325
326
327
328
329
330
```yaml
# values.yml
env:
  public:
    RAILS_ENV: production
    PORT: 8080 # override default port 5000
  secret:
    POSTGRESQL_PASSWORD: app-secrets

application:
Marco Wettstein's avatar
Marco Wettstein committed
331
  command: ["/cnb/process/web"] # buildpack entrypoint
Michael Leu's avatar
Michael Leu committed
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
  livenessProbe:
    httpGet:
      path: /robots.txt # change to your health route
      httpHeaders: # empty
  readinessProbe:
    httpGet:
      path: /robots.txt # change to your health route
      httpHeaders: # empty

cloudsql:
  enabled: true

jobs:
  db-prepare:
    hook: post-install,post-upgrade
Marco Wettstein's avatar
Marco Wettstein committed
347
    command: "/cnb/lifecycle/launcher bundle exec rake db:prepare"
Michael Leu's avatar
Michael Leu committed
348
349
350
351
352
353
354
```

```yaml
# values-review.yml
jobs:
  db-prepare-seed:
    hook: post-install
Marco Wettstein's avatar
Marco Wettstein committed
355
    command: "/cnb/lifecycle/launcher bundle exec rake db:prepare db:seed"
Michael Leu's avatar
Michael Leu committed
356
357
  db-prepare:
    hook: post-upgrade
Marco Wettstein's avatar
Marco Wettstein committed
358
    command: "/cnb/lifecycle/launcher bundle exec rake db:prepare"
Michael Leu's avatar
Michael Leu committed
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
```

Note: Cloud Native Buildpacks creates these default processes, which are available as `/cnb/process/<process-type>` inside the container:

```
TYPE          SHELL COMMAND
web (default) bash  bin/rails server -p ${PORT:-5000} -e $RAILS_ENV
console       bash  bin/rails console
rake          bash  bundle exec rake
worker        bash  bundle exec rake jobs:work
```

Define additional ones in a `Procfile` like with Heroku.

All other commands have to be prefixed with `/cnb/lifecycle/launcher` to run with the correct environment.

375
376
377
378
379
### panter-kubernetes-base

a base image that can be used for anything else.

You can override .lint, .test, .app-build stages to specify mvn commands or similar