Deploying Haskell Apps With Kubernetes

Deni Bertovic

September 5, 2018

Deploying Haskell Apps With Kubernetes

About me

Deni Bertović

https://denibertovic.com

https://github.com/denibertovic

https://twitter.com/denibertovic

Software Engineer

fpco

Overview

Kubernetes intro
Creating a k8s cluster using terraform+kops
Describe various kubernetes terms (Pods, Deployments, Services...)
Build a Haskell docker image
Deploy Haskell Docker image
Helm + CI/CD
Useful Helm charts

What is kubernetes?

"Kubernetes is an open-source system for automating deployment,
scaling, and management of containerized applications."

"It groups containers that make up an application into logical
units for easy management and discovery. Kubernetes builds
upon 15 years of experience of running production workloads
at Google, combined with best-of-breed ideas and
practices from the community."

What is kubernetes?

Abstracts over a cluster of machines
VMs, bare metal, etc
Common "language" for deploying
Resource management (cpu, memory etc)

Kubernetes features

Automatic binpacking
Self-healing
Horizontal scaling
Automated rollouts and rollbacks
Service discovery and load-balancing
Secrets and configuration management
Storage Orchestration

...

Creating a Kubernetes cluster

Terraform

https://github.com/fpco/terraform-aws-foundation

module "vpc" {
  source      = "fpco/foundation/aws//modules/vpc"
  version     = "0.7.5"
  region      = "${var.region}"
  cidr        = "${var.vpc_cidr}"
  name_prefix = "${var.name}"
  extra_tags  = "${merge(var.extra_tags,
                   map("kubernetes.io/cluster/${var.kubernetes_cluster_name}", "shared"))}"
}

module "kube-public-subnets" {
  source      = "fpco/foundation/aws//modules/subnets"
  version     = "0.7.5"
  azs         = "${var.aws_availability_zones}"
  vpc_id      = "${module.vpc.vpc_id}"
  name_prefix = "${var.name}-kube-public"
  cidr_blocks = "${var.kube_public_subnet_cidrs}"
  extra_tags  = "${merge(var.extra_tags,
                   map("kubernetes.io/cluster/${var.kubernetes_cluster_name}", "shared"),
                   map("kubernetes.io/role/elb", "1"))}"
}

Kops 1/3

kops create cluster \
  --authentication=rbac \
  --cloud=aws \
  --kubernetes-version=${KUBERNETES_VERSION} \
  --networking="flannel" \
  --master-size=t2.small \
  --master-zones=us-east-1a,us-east-1b,us-east-1c \
  --network-cidr=${VPC_CIDR} \
  --node-count=${NODE_COUNT} \
  --node-size=${NODE_SIZE} \
  --ssh-public-key=${SSH_PUBLIC_KEY} \
  --zones=us-east-1a,us-east-1b,us-east-1c \
  --vpc=${VPC_ID} \
  --node-volume-size=${NODE_VOLUME_SIZE} \
  --state=s3://${CLUSTER_NAME} \
  --name=${CLUSTER_NAME}

Kops 2/3

kops edit cluster --name ${CLUSTER_NAME} \
  --state s3://${CLUSTER_NAME}

metadata:
  ...
  name: ${CLUSTER_NAME}
spec:
  cloudProvider: aws
  networkCIDR: ....
  networkID: ....
  nonMasqueradeCIDR: 100.64.0.0/10
  subnets:
  - cidr: 172.20.32.0/19
    name: us-east-1b
    type: Public
    zone: us-east-1b
    id: ....

Kops 3/3

kops update cluster --name ${CLUSTER_NAME} \
  --state s3://${CLUSTER_NAME} # Append "--yes" to apply changes

kops validate cluster --name ${CLUSTER_NAME} \
  --state s3://${CLUSTER_NAME}

Kubernetes terminology

A collection of containers

"A pod (as in a pod of whales or pea pod) is a group of
one or more containers (such as Docker containers), with
shared storage/network, and a specification for how to
run the containers."

Kubernetes terminology

Deployment

A collection of pods.

"A Deployment controller provides declarative updates
for Pods and ReplicaSets."

Kubernetes terminology

Service

Exposes a pod to the world (or other pods)

Building a Haskell Docker image

Very simple yesod web application (link).

Yesod

Building a Haskell Docker image

Docker multi stage build

FROM fpco/stack-build:lts-9.9 as build

RUN mkdir /opt/build
COPY . /opt/build

VOLUME /tmp/stackroot

RUN cd /opt/build && stack --stack-root=/tmp/stackroot \
  build --system-ghc

FROM fpco/pid1
RUN mkdir -p /opt/app
WORKDIR /opt/app

RUN apt-get update && apt-get install -y \
  ca-certificates \
  libgmp-dev

COPY entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

COPY --from=build \
  /opt/build/.stack-work/install/x86_64-linux/lts-9.9/8.0.2/bin .
COPY static /opt/app/static
COPY config /opt/app/config

CMD ["/opt/app/myapp"]

Building a Haskell Docker image

Build and push the image

docker build registry.gitlab.fpcomplete.com/fpco-mirrors/haskell-multi-docker-example .

docker push registry.gitlab.fpcomplete.com/fpco-mirrors/haskell-multi-docker-example

Test the image locally

docker run -p 3000:3000 -it -w /opt/app \
  registry.gitlab.fpcomplete.com/fpco-mirrors/haskell-multi-docker-example \
  myapp

Create K8S Specs

Deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    metadata:
      labels:
        app: myapp
    spec:
      imagePullSecrets:
        - name: registry-key
      containers:
        - name: myapp
          image: registry.gitlab.fpcomplete.com/fpco-mirrors/haskell-multi-docker-example:4341
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 3000
          command: ["/opt/app/myapp"]
          workingDir: /opt/app
          readinessProbe:
            httpGet:
              path: /
              port: 3000
          livenessProbe:
            httpGet:
              path: /
              port: 3000

Deployment

kubectl apply -f deployment.yaml

Verifying deployment

kubectl get deployments


NAME                                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
myapp                                        1         1         1            1           10m

kubectl get pods

NAME                                                          READY     STATUS    RESTARTS   AGE
myapp-7b7d556fc9-gzgl7                                        1/1       Running   0          7m

Service

apiVersion: v1
kind: Service
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  ports:
  - name: http
    port: 80
    targetPort: http
  selector:
    app: myapp
  type: LoadBalancer

kubectl apply -f service.yaml

Verify service

kubctl get svc

NAME                                         TYPE           CLUSTER-IP       EXTERNAL-IP        PORT(S)                      AGE
myapp                                        LoadBalancer   100.64.53.144    aacd3aa63aa12...   80:31726/TCP                 8m

Find out more about the Service

kubectl describe svc myapp

LoadBalancer Ingress:     aacd3aa63aa12...us-east-1.elb.amazonaws.com

Problems

No templating
Automation
ELB's are not cheap
No SSL
No custom domain

Delete

kubectl delete svc myapp
kubectl delete deployment myapp

Helm

The package manager for kubernetes

Charts

helm create chart

Helm

Values

replicaCount: 1
image:
  repository: registry.gitlab.fpcomplete.com/fpco-mirrors/haskell-multi-docker-example
  tag: latest
  pullPolicy: IfNotPresent
service:
  name: myapp
  type: ClusterIP
  externalPort: 80
  internalPort: 3000
ingress:
  enabled: false
...

Helm

Deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: {{ template "chart.fullname" . }}
  labels:
    app: {{ template "chart.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    metadata:
      labels:
        app: {{ template "chart.name" . }}
        release: {{ .Release.Name }}
    spec:
      imagePullSecrets:
        - name: registry-key
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - containerPort: {{ .Values.service.internalPort }}
          livenessProbe:
            httpGet:
              path: /
              port: {{ .Values.service.internalPort }}
          readinessProbe:
            httpGet:
              path: /
              port: {{ .Values.service.internalPort }}
          resources:
{{ toYaml .Values.resources | indent 12 }}
    {{- if .Values.nodeSelector }}
      nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
    {{- end }}

Helm

Service

apiVersion: v1
kind: Service
metadata:
  name: {{ template "chart.fullname" . }}
  labels:
    app: {{ template "chart.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.externalPort }}
      targetPort: {{ .Values.service.internalPort }}
      protocol: TCP
      name: {{ .Values.service.name }}
  selector:
    app: {{ template "chart.name" . }}
    release: {{ .Release.Name }}

Deploying with helm

## Deploy helm chart
deploy:
    @helm upgrade \
        --install myapp chart \
        -f chart/values/${CI_ENVIRONMENT_NAME}.yaml \
        --set image.tag="${CI_PIPELINE_ID}"

CI/CD

Gitlab

build-and-push:
  stage: build
  script:
    - make build-ci-image
    - docker login -u gitlab-ci-token -p "${CI_BUILD_TOKEN}" "${CI_REGISTRY}"
    - docker push "${CI_REGISTRY_IMAGE}:${CI_PIPELINE_ID}"

deploy_prod:
  stage: deploy
  script:
    - make deploy
  environment:
    name: production
    url: https://k8s-haskell-webinar.fpcomplete.com
  when: manual
  only:
  - master

Ingress

"Ingress can provide load balancing, SSL termination and name-based virtual hosting."

So...it's Nginx

Ingress

helm upgrade --install \
    fpco-ingress stable/nginx-ingress

Ingress

{{- if .Values.ingress.enabled -}}
{{- $serviceName := include "chart.fullname" . -}}
{{- $servicePort := .Values.service.externalPort -}}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: {{ template "chart.fullname" . }}
  labels:
    app: {{ template "chart.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
  annotations:
    {{- range $key, $value := .Values.ingress.annotations }}
      {{ $key }}: {{ $value | quote }}
    {{- end }}
spec:
  rules:
    {{- range $host := .Values.ingress.hosts }}
    - host: {{ $host }}
      http:
        paths:
          - path: /
            backend:
              serviceName: {{ $serviceName }}
              servicePort: {{ $servicePort }}
    {{- end -}}
  {{- if .Values.ingress.tls }}
  tls:
{{ toYaml .Values.ingress.tls | indent 4 }}
  {{- end -}}
{{- end -}}

Ingress

cat chart/values/production.yaml

ingress:
  enabled: true
  hosts:
    - k8s-haskell-webinar.fpcomplete.com
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  tls:
    - secretName: k8s-haskell-webinar-tls
      hosts:
        - k8s-haskell-webinar.fpcomplete.com

Ingress

fpco

How does SSL work?

Automatic LetsEncrypt certificates

annotations:
  kubernetes.io/ingress.class: nginx
  kubernetes.io/tls-acme: "true"  <---- HERE
tls:
  - secretName: k8s-haskell-webinar-tls
    hosts:
      - k8s-haskell-webinar.fpcomplete.com

kube-lego
https://github.com/jetstack/kube-lego
(official helm chart stable/kube-lego)
After kube 1.8 use cert-manager
https://github.com/jetstack/cert-manager/
(not production ready)

How about DNS records?

Automatic route53 entry (other providers are supported)

ingress:
  ...
  hosts:
    - k8s-haskell-webinar.fpcomplete.com

external-dns
https://github.com/kubernetes-incubator/external-dns

(official helm chart stable/external-dns)

fpco/foundation

https://github.com/fpco/helm-charts

fpco/foundation

Prerequisites:

https://github.com/fpco/terraform-aws-foundation
https://registry.terraform.io/modules/fpco/foundation/aws

module "fpco-dnscontroller" {
  source                 = "fpco/foundation/aws//modules/external-dns-iam"
  version                = "0.7.5"
  name_prefix            = "mycluster"
  kube_cluster_nodes_arn = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/nodes.${var.k8s_cluster_name}"
}

fpco/foundation

## prod/values.yaml

enabled:
  kube-lego: true
  external-dns: true
  heapster: false
  kubernetes-dashboard: false
  kube2iam: true

kube-lego:
  config:
    # set your email
    LEGO_EMAIL: ops@mycompany.com
    LEGO_URL: https://acme-v01.api.letsencrypt.org/directory
  rbac:
    create: true

external-dns:
  image:
    tag: v0.4.5
  podAnnotations:
    "iam.amazonaws.com/role": mycluster-dnscontroller
  rbac:
    create: true

kube2iam:
  host:
    iptables: true
    interface: cni0
  rbac:
    create: true
  extraArgs:
    auto-discover-base-arn: true

fpco/foundation

helm repo add fpco https://s3.amazonaws.com/fpco-charts/stable/
helm update
helm install fpco/foundation --name fpco-foundation --namespace kube-system \
  --values=prod/values.yaml

Thank you

Questions?

fpco