It clicked when I rebuilt the core of pixbin.net in about two hours. Pixbin is a super simple image sharing service I’ve been running since 2012 - drag and drop upload, auto-expiring images, shareable links. The codebase was showing its age and I’ve been putting off a rewrite/upgrade for years. With an agent handling the boilerplate while I focused on architecture and review, the whole thing came together in a single sitting on a hot summers day. Django, Celery for background image processing, S3 storage, Kubernetes deployment configs. Two hours. And then I started adding features!

This changed how I looked at my project backlog and list of ideas.

Templates matter more than prompts

Honstly, the biggest unlock wasn’t better prompting - although precise prompts do help. It was having a solid project template. I built hellok8s-django with all my opinions baked in: Nix and devenv for reproducible environments, Docker builds with uv, Helm charts, SOPS for secrets, GitHub Actions CI/CD. And yes I used Agents to make the template as well. Now, when I start a new project, the agent understands the conventions immediately because they’re encoded in the template. No explaining where things go or how deployments work.

This is important because agents are excellent at following patterns but bad at inventing them. If you give an agent a blank canvas you get mediocre code with inconsistent structure. If you give it a well-organized codebase with clear conventions, it produces code that fits right in.

None of this is vibe coding. I’m reviewing every diff, making the architectural calls, fixing the things it gets wrong - it does get things wrong. But the grunt work - the CRUD views, the serializers, the test scaffolding, the Helm values - that’s largely handled. My throughput went through the roof.

Solving the same problems once

Once I had the template and the workflow down, I started noticing a pattern: every project I build needs the same handful of infrastructure services. Email, image hosting, form handling. I’d been solving these ad-hoc per project, or paying for SaaS tools I didn’t really need. So I started building them as shared services that my future projects can use.

Email

Every web app needs email. Registration, password resets, notifications. I was looking at various SaaS services - but I don’t have real users yet for most of these projects. Paying $20+/month for a service that sends maybe 5 messages a month is absurd. Also some of them doing a rugpull and disabling their free tier without notification didn’t help either.

I remember reading a tweet by Pieter Levels about how the economics of AWS SES were just unmatched, and how most of these SaaS services were wrappers around SES. I don’t know if it was this specific tweet but something to that effect. This rang true to me - at least for my needs.

So I built sesy.email. It’s a Django service that wraps AWS SES behind a REST API. You delegate a subdomain (like notify.yourdomain.com), and SESY handles everything automatically: creates the Route53 hosted zone, provisions the SES identity, writes the DKIM CNAMEs, SPF and DMARC TXT records, configures the MAIL FROM domain. It also handles bounces and complaints via SNS webhooks and maintains per-project suppression lists and limits. It’s a clicky web interface with a “next-next-finish” feel to it. I am thinking about making it agent first but haven’t gotten to it yet.

I built Python, JavaScript, and Django SDKs for it. The Django SDK is a standard EmailBackend, so any Django project can use it with two lines of config. SES costs fractions of a cent at my scale. I have this problem solved forever.

Now … being heavy in infra all my life I’m saddend by the steady degradation of the distributed internet and I briefly thought about building the email backend directly but I wanted to get going faster. I may still do it. I mean, I still self host my own email server so why not.

Image hosting

Pixbin already existed, but I gave it a proper API so all my projects can use it for image uploads and processing. Every web app eventually needs thumbnails, resized versions, maybe format conversion. I don’t want to reimplement this per project. Upload an image, get back URLs. Done.

Forms

I was using one SaaS for contact forms on my static sites. I don’t have any beef with them but I built my own replacement: formsubmit.cloud. It’s a Django app - you create a form, get a POST /f/{uuid}/ endpoint, and submissions get emailed to you via SESY. I added Cloudflare Turnstile for captcha, honeypot fields, some basic spam scoring, and webhook support for forwarding submissions to other services.

Then one evening I wanted to organize a dinner with friends. We usually used Doodle for this kind of thing. Doodle now requires registration and has gotten so bloated that it takes genuinely long to load and it’s a pain to use. I built basic scheduling functionality into formsubmit instead. Took an hour or so.

That’s the theme here. Something annoys me, I build a replacement, and it takes an evening instead of a month. The agent does the mechanical work, I make the design decisions, and the template ensures everything deploys the same way and generally reuses the same patterns.

What I’m working on now

A digital products store - inspired by Basecamp’s ONCE model. Pay once, own forever. No subscriptions. Django backend, Stripe for payments with multi-currency support (USD and EUR, geo-detected via Cloudflare’s CF-IPCountry header), pre-signed S3 URLs for file delivery, and GitHub API integration for granting repository access to buyers. Once it’s ready I’ll list SESY there as a purchasable self-hosted product. Haven’t decided yet if formsubmit will be a product or remain a hosted service.

A social media scheduler - just for myself, just Twitter and LinkedIn. Next.js with PostgreSQL. I want to batch-write posts on Sunday and have them go out during the week. Nothing fancy.

And a few more that I’ll talk about in other posts.

The economics

I run everything on a single bare metal instance on Scaleway. Kubernetes, all my services, all my databases, one box. Total cost is the hosting bill plus whatever pennies SES charges me. No per-service SaaS subscriptions, no usage tiers that suddenly spike when you cross some arbitrary threshold. And yes it runs NixOS. :)

Self-hosting has real costs - I maintain the Kubernetes “cluster”, handle ops, keep things patched. But for a single developer building tools for myself this is manageable. And the marginal cost of deploying one more service is effectively zero. That changes the math on what’s worth building.

If a project takes off I change the credentials in the CICD pipeline and point it to a proper kubernetes cluster where I can scale. Easy peasy.

What changed

I haven’t been this excited about computers since I first learned programming, discovered the internet and deployed my first webapp.

The backlog of projects that lived in my head for years is actually shrinking. Ideas that would have stayed in a notes app are getting deployed - even if I’m the only user, at least I’m trying stuff out. I’m also way less tolerant of small annoynaces in my day to day - like on my laptop - I just tell the agent to fix them.

What used to take a week takes a weekend. What used to take a weekend takes an evening. Not because the agent is doing the thinking for me, but because it’s doing the typing. I still need to know what I want, how to structure it, and what a good solution looks like. But the bottleneck has shifted from implementation speed to decision-making speed, and that’s a genuinely different way to work.

You can just build things.

So I built a small tool called slopbox for running AI agents in isolated Docker containers and a copy of my repo. This way I’m free to merge changes in (or throw them away) without it affecting my main worktree. It was initially inspired by Dagger’s container-use. This is just much more in tune with my workflow and the tools I use - but also I don’t trust an Agent to do something (consistently) just because it says so in a markdown file. I need stronger guarantees.

How it works

Slopbox creates disposable sandboxes using Docker. Each sandbox gets:

• A git worktree clone of my repo (not the actual repo)
• A shared Nix store mounted read-only from a daemon container (yes I don’t want to let it touch my host’s nix store either)
• My devenv environment, so tools “just work”

When I run slop my-feature, it creates a branch called agent/my-feature, clones my repo to ~/.cache/slopbox/worktrees/, and drops me (or the agent) into a container. The agent can commit, install packages, break things…whatever. My real repo and environment stays untouched.

When I’m done, slop diff my-feature shows what’s changed. If I like it, slop apply my-feature merges it back. If not, slop gc deletes everything.

This is an experiment

This is super experimental and exploratory! It may change or break in unexpected ways - or I may abandon it completely. I rarely run unsupervised agent loops right now. Most of my Agent usage is interactive - I’m watching what it does, reviewing diffs, steering it when it goes off course. But I want the option to spin up a sandbox, point an agent at a problem, and check back later.

The use case I keep coming back to is throwaway POCs. “Hey Claude, prototype a CLI that does X” or “sketch out how we’d add Y to this service.” Things where I don’t care about code quality, I just want to see if an idea is feasible. Having the agent work in an isolated clone means I can let it run wild without worrying about the mess.

Future directions

Docker containers provide decent isolation for my use-case. I don’t think I really need much stronger isolation than that - nor am I thinking about restricting it’s network access or some such. That said, I did think about other approaches:

Bubblewrap - The sandboxing tool Flatpak uses. Lighter weight than Docker, gives me fine-grained control over namespaces and bind mounts. Only reason I’m considering it cause it could be faster to spin up and tear down.

Full NixOS VMs - Since I’m already running NixOS on my laptop, I could theoretically replicate my entire workstation inside a VM. VirtualBox, QEMU, whatever - once. The Nix configuration is already there - my system is declarative. This would give stronger isolation than containers (separate kernel, no shared daemon) at the cost of more overhead. Might be worth it for truly untrusted workloads. Realistically I don’t see this happening since it will be even slower than the current approach and that irks me. But we’ll see. Maybe if it’s a long running single VM for all agent workflow and not one per agent.

Neither of these exist yet. Just ideas I’m mulling over.

I also may consider writing a proper CLI in a proper programming language. Right now it’s just a bunch of bash scripts that are distributed via devenv. Honestly, if devenv had this built in that would be kinda cool I guess. But I don’t think the idea has fully crystallized yet and bash is fine for experimentation. Ironically I’m totally fine with bash now that I’m not the one writing them - the Agent is.

Drawbacks

This tool is tightly coupled to my setup and a very specific way I work. I use devenv and Nix for most of my projects. If you’re not already in that ecosystem, the on-boarding cost is high. There are probably simpler solutions for your use case.

Other issues:

• Slow - Even with caching, sandbox startup is slower than running locally. First run for a project is especially painful as it populates the Nix store. The good news is that subsequent runs (even for other projects) are much faster.
• Disk hungry - Each worktree is a full clone. The shared Nix store grows over time. Run slop gc periodically.
• Complexity - Docker daemon, Nix daemon, volumes, git mirrors. More moving parts means more things that can break.
• Manual agent setup - There’s an instructions file at /etc/slop/instructions.txt that explains the environment to agents running inside it, but I have to manually tell them to read it. Haven’t thought about how to automate that yet.
• Multi agent support - Right now I mostly run claude but I’d like to have custom support built it for each major CLI agent. And not have to remember --dangerously-skip-permissions would also be nice.

I’m sharing this mostly for my own documentation, because I promised myself I’d write more this year and because I might gain more insight by writing things down. Maybe it’s useful as a reference for others thinking about similar problems.

Check it out in all it’s bash glory at: github.com/denibertovic/slopbox.

As modern web applications grow in complexity, the gap between development tutorials and production-ready deployments becomes increasingly challenging. Most Next.js guides stop at npm run dev, but production applications require containerization, orchestration, secrets management, CI/CD pipelines, and reproducible development environments. Today, I want to share insights from hellok8s-nextjs, a comprehensive project template that demonstrates how to bridge this gap with battle-tested DevOps practices.

The Challenge: From Tutorial to Production

First, let’s address the elephant in the room: Vercel exists, and it’s excellent. For many Next.js applications, Vercel provides the easiest deployment experience with zero configuration, automatic scaling, and seamless integration with the Next.js ecosystem. If Vercel meets your needs, use it - it’s a fantastic platform that handles most of the complexity I’m about to discuss.

However, real-world enterprise requirements often demand self-hosted solutions for a variety of reasons:

Compliance requirements (HIPAA, SOC 2, PCI DSS) that require data to remain within specific geographic boundaries or private networks.

Cost optimization for high-traffic applications where predictable infrastructure costs matter.

Integration with existing infrastructure and legacy systems that can’t be easily migrated.

Custom security policies that require full control over the deployment environment.

Air-gapped environments or on-premises deployments where external platforms aren’t viable.

Multi-cloud strategies that require vendor independence.

When these constraints apply, scaling from a local development server to a production Kubernetes deployment involves numerous considerations that most tutorials don’t address. How do you ensure consistent environments across team members? How do you manage secrets securely? How do you achieve zero-downtime deployments with proper rollback capabilities?

After implementing these patterns across multiple technology stacks and organizations, I’ve distilled these proven practices into hellok8s-nextjs, a production-ready template that demonstrates the complete DevOps lifecycle for modern Next.js applications. These patterns have proven successful for teams ranging from small startups to enterprise organizations managing hundreds of developers.

What Makes This Different

1. Reproducible Development with Nix and devenv

The project uses Nix and devenv to create completely reproducible development environments. When developers run direnv allow, they get:

• Runtime environment (Node.js 23 with Yarn 4 pre-configured)
• Database services (PostgreSQL and Redis) automatically configured and running
• Kubernetes tooling (kubectl, Helm, SOPS) pre-installed
• AI development tools (Sourcegraph AMP, Claude, Gemini CLI) ready to use
• Git hooks for automated code formatting and linting
• Hot reloading for the Next.js application
• Everything running natively with no container overhead

# One command to rule them all
devenv up

This eliminates the infamous “works on my machine” syndrome and reduces onboarding time from days to minutes. I’ve implemented similar setups for clients managing teams of 100+ developers, resulting in a 90% reduction in environment-related support tickets.

2. Modern Next.js Stack with Type Safety

The application showcases current Next.js best practices using the T3 Stack:

• Next.js 15+ with App Router
• TypeScript for end-to-end type safety
• tRPC for type-safe API routes
• Drizzle ORM for type-safe database operations
• NextAuth.js for authentication with credential-based auth
• Tailwind CSS for utility-first styling
• Vitest for fast testing with transaction rollbacks

3. Optimized Container Builds with Modern Tooling

The Docker build process leverages multi-stage builds and modern Node.js tools for fast, efficient container images:

FROM node:23-slim AS base
ENV YARN_VERSION=4.9.1
RUN yarn set version $YARN_VERSION
...
# dependency installation from lockfile
FROM base AS deps
COPY .yarnrc.yml package.json yarn.lock ./
RUN yarn --immutable
...
# Production build with standalone output
FROM base AS builder
ENV SKIP_ENV_VALIDATION=1
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN yarn run build && yarn run build:scripts
...
# Runtime image
FROM base AS runner
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
...
CMD ["node", "server.js"]

This approach reduces Docker image build times significantly, which is crucial for CI/CD pipelines that build hundreds of images daily. The standalone output feature reduces the final image size by including only necessary files.

4. Enterprise-Grade Secrets Management

Security is built-in from the start using SOPS for encrypted secrets management. Database passwords, API keys, and certificates are stored encrypted in the repository and automatically decrypted during deployment:

# Secrets are encrypted at rest, decrypted at deploy time
database:
  password: ENC[AES256_GCM,data:encrypted_value]
nextauth:
  secret: ENC[AES256_GCM,data:encrypted_value]

The project supports both age encryption and cloud provider KMS (AWS, GCP, Azure), making it suitable for compliance requirements while maintaining developer productivity.

5. Kubernetes-Native with Helm Charts

The template includes a complete Helm chart structure with environment-specific configurations:

• Rolling updates with zero downtime
• Health checks and readiness probes
• Resource limits and requests for optimal scheduling
• Ingress configuration with automatic HTTPS (Let’s Encrypt)
• ConfigMap and Secret management with SOPS integration

# Deploy to any environment with one command
make IMAGE_TAG=sha-123 ENVIRONMENT=prod NAMESPACE=hellok8s deploy

6. Advanced CI/CD with GitHub Actions

The GitHub Actions setup demonstrates modern CI/CD patterns:

• Reusable workflow templates for multiple environments
• Docker layer caching for faster builds
• Automated testing and type checking
• GitOps-style deployments for different environments
• Multi-registry support (Docker Hub, AWS ECR, etc.)

# Example: Reusable deployment workflow
jobs:
  deploy:
    uses: ./.github/workflows/deploy.yml
    with:
      environment: prod
      namespace: hellok8s
      url: https://hellok8s-nextjs.deni.cloud

7. Fast Testing with Real Database

Unlike many projects use a different database (such as in-memory sqlite) for testing - this template uses the same database as production (postgres) while automating transaction rollbacks for lightning-fast tests:

• Speed: Tests run against real PostgreSQL but roll back all changes
• Fast hashing: MD5 for password hashing in tests (bcrypt in production)
• Isolation: Each test runs in a clean state
• Real behavior: Catches actual database constraints and behaviors

// Example: Fast integration test
describe("getAll - listing posts (public)", () => {
it("should return all posts for unauthenticated users", async () => {
  await testDb.runInTransaction(async (db) => {
    const superUser = await createTestUser(db, {
      email: "super@example.com",
      firstName: "Super",
      lastName: "User",
      isSuperuser: true,
    });

    // Create test posts
    await createTestPost(db, superUser.id, {
      title: "First Post",
      slug: "first-post",
    });
    await createTestPost(db, superUser.id, {
      title: "Second Post",
      slug: "second-post",
    });

    const caller = createCaller(createMockContext(db, null));
    const result = await caller.getAll();

    expect(result).toHaveLength(2);
    // Just check that both posts are returned, order may vary in tests
    const titles = result.map((p: any) => p.title).sort();
    expect(titles).toEqual(["First Post", "Second Post"]);
  });
});

Real-World Impact: Measurable Results

These aren’t just theoretical improvements. This template reflects patterns I’ve implemented for organizations ranging from early-stage startups to Fortune 500 companies, delivering measurable improvements:

• Reduced deployment times from hours to minutes (average 15x improvement)
• Eliminated environment drift across development, staging, and production
• Improved security posture with encrypted secrets management
• Decreased onboarding time for new developers by 90%
• Enabled true DevOps practices with infrastructure as code
• Increased deployment frequency from weekly to multiple times per day
• Reduced production incidents by 60% through better testing and staging

How I Can Help Your Organization

This project template reflects years of hands-on experience helping teams overcome production challenges. I offer consulting services to help organizations implement these patterns:

DevOps Transformation & Cloud-Native Architecture

• Kubernetes Strategy: Container optimization, security hardening, and best practices
• CI/CD Pipeline Design: From basic automation to advanced GitOps workflows
• Infrastructure as Code: Terraform, Helm, and cloud provider best practices
• Secrets Management: SOPS, HashiCorp Vault, and cloud-native solutions

Developer Experience & Team Scaling

• Reproducible Environments: Nix, Docker, and devenv implementations
• Onboarding Optimization: Reducing time-to-productivity for new team members
• Quality Processes: Code review standards, automated testing, and quality gates
• Technical Leadership: Architecture reviews and technology selection

Let’s Talk

Whether you’re struggling with Next.js deployment complexity, seeking to modernize your development practices, or planning a cloud migration, the patterns demonstrated in this template can transform your organization’s engineering capabilities.

Common Scenarios Where I Can Help:

• “Our deployments are unreliable and take forever” - Implement automated CI/CD with rollback capabilities
• “New developers take weeks to get productive” - Create reproducible development environments
• “We’re afraid to deploy on Fridays” - Build confidence through automated testing, easy rollbacks and deployable PRs (review apps)
• “Managing secrets is a security nightmare” - Implement encrypted, automated secrets management
• “Our Next.js app doesn’t scale” - Optimize performance and implement proper caching strategies

I’m available for hands-on consulting engagements ranging from focused architecture reviews to full-scale DevOps transformations. Whether you need a quick assessment or comprehensive implementation, let’s discuss how these proven practices can accelerate your team’s delivery and reliability.

Ready to get started? This project serves as a perfect conversation starter for your Next.js infrastructure needs. We can review your current setup, identify pain points, and create a roadmap for improvement that delivers measurable results. Reach out now using this contact form.

The complete source code and documentation for hellok8s-nextjs is available on GitHub. Feel free to use it as a starting point for your own projects, or reach out if you’d like help implementing these patterns in your organization.

Why PID 1 is Special

In Linux, the kernel treats PID 1 differently from all other processes. It’s the “init” process that bootstraps the system and has two critical responsibilities:

Signal handling: The kernel doesn’t deliver certain signals (like SIGTERM) to PID 1 unless it explicitly registers handlers for them.

Process reaping: PID 1 must clean up zombie processes by calling waitpid() on dead children.

When you run a container with:

CMD ["./my-app"]

Your application becomes PID 1, inheriting these kernel expectations whether it’s designed for them or not.

What Goes Wrong

Without proper PID 1 handling, you’ll encounter:

• Unresponsive shutdowns: docker stop sends SIGTERM to PID 1, but if your app doesn’t handle it, the kernel ignores it. Docker waits 10 seconds, then forcibly kills with SIGKILL
• Zombie accumulation: Child processes that die aren’t reaped, leaving zombie entries in the process table
• Orphaned processes: Background processes lose their parent but aren’t properly managed

The Solution

Use a proper init system that handles PID 1 responsibilities:

# Install tini
RUN apk add --no-cache tini

# Use tini as entrypoint
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["./my-app"]

Or enable Docker’s built-in init:

docker run --init my-image

Another excellent option is fpco/pid1, a minimal init system implemented in Haskell:

ENV PID1_VERSION=0.1.3.1
RUN curl -sSL "https://github.com/fpco/pid1/releases/download/v${PID1_VERSION}/pid1" -o /sbin/pid1 && \
    chown root:root /sbin/pid1 && \
    chmod +x /sbin/pid1

ENTRYPOINT ["/sbin/pid1", "-u", "appuser", "-g", "appgroup"]
CMD ["./my-app"]

What makes fpco/pid1 particularly useful are its -u and -g flags, which allow you to execute the child process as a specified user and group. This eliminates the need for separate user switching logic while maintaining proper PID 1 behavior. It is also very useful for development as it makes it possible to switch to a dynamic user that that matches your host UID so no permissons get messed up on munted host volumes.

There’s also the new and improved pid1-rs (rewritten in Rust):

ENV PID1_VERSION=0.1.2
RUN curl -sSL "https://github.com/fpco/pid1-rs/releases/download/v${PID1_VERSION}/pid1-x86_64-unknown-linux-musl" -o /sbin/pid1 && \
    chown root:root /sbin/pid1 && \
    chmod +x /sbin/pid1

ENTRYPOINT ["/sbin/pid1"]
CMD ["./my-app"]

Even better, if you’re writing Rust applications, you can embed PID 1 functionality directly by using the pid1 crate. This means your application can handle PID 1 responsibilities natively without needing an external init process:

use pid1::Pid1Settings;

fn main() {
    let mut settings = Pid1Settings::new();
    settings.enable_signal_handling().enable_reaping();
    settings.launch(|| {
        // Your application code here
        run_my_app();
    }).expect("Failed to launch with pid1");
}

Conclusion

A proper init process:

• Forwards signals to your application
• Reaps zombie processes automatically
• Ensures graceful shutdown by sending SIGTERM to children before exiting
• Provides the “grace period” needed for clean application shutdown

Unless your application is specifically designed to handle PID 1 responsibilities, wrap it with a minimal init system. It’s a small change that prevents hard-to-debug container behavior and ensures predictable shutdown semantics.

Your future self (and your ops team) will thank you.

The Challenge: Bridging the Gap from Tutorial to Production

Most web framework tutorials stop at basic development servers. But production applications require so much more: containerization, orchestration, secrets management, CI/CD pipelines, and reproducible development environments. The gap between “hello world” and production-ready is where most teams struggle and where costly delays, security vulnerabilities, and scaling bottlenecks emerge.

After years of building web applications across multiple technology stacks, I’ve distilled these battle-tested practices into hellok8s-django, a production-ready template that demonstrates the complete DevOps lifecycle, from local development to Kubernetes at scale. These patterns have proven successful with Python, Node.js, Go, Ruby, Haskell and other modern tech stacks.

What Makes This Different

1. Reproducible Development with Nix and devenv

Gone are the days of “works on my machine.” This project uses Nix and devenv to create completely reproducible development environments. When developers run direnv allow, they get:

• Runtime environment (Python 3.13 with uv in this case)
• Database (and other supporting services) automatically configured and running
• kubectl, Helm, and other cloud provider tools pre-installed
• Asset compilation with file watching
• Git hooks for code formatting and linting
• Everything running natively on your local machine with no container overhead

# One command to rule them all
devenv up

This approach eliminates the infamous “works on my machine” syndrome and reduces onboarding time from days to minutes. I’ve implemented similar setups for clients managing teams of 100+ developers across multiple time zones, resulting in a 90% reduction in environment-related support tickets.

2. Lightning-Fast Container Builds with Modern Tooling

While many projects still use legacy dependency management, this template leverages modern, high-performance tooling that greatly simplifies the entire container building process.

# Install the project's dependencies using the lockfile and settings
RUN --mount=type=cache,target=/root/.cache/uv \
    --mount=type=bind,source=uv.lock,target=uv.lock \
    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
    uv sync --locked --no-install-project --no-dev

This approach reduces Docker image build times, crucial for CI/CD pipelines that build hundreds of images daily. For one client, this optimization alone saved 4 hours of build time daily across their development team. The same patterns apply whether you’re using npm, Go modules, Cargo, or other package managers.

3. Enterprise-Grade Secrets Management with SOPS

Security isn’t an afterthought. The project integrates SOPS for encrypted secrets management, supporting both age encryption and cloud native tooling such as AWS KMS. Database passwords, API keys, and certificates are stored encrypted in the repository and automatically decrypted during deployment.

# Secrets are encrypted at rest, decrypted at deploy time
database_password: ENC[AES256_GCM,data:encrypted_value]

This approach satisfies compliance requirements while maintaining developer productivity (secrets are “diffable” in PRs), something I’ve implemented for clients with strict regulatory requirements.

4. Kubernetes-Native with Helm Charts

The template includes a complete Helm chart structure with environment-specific configurations. The deployment process demonstrates:

• Rolling updates with zero downtime
• Health checks and readiness probes
• Resource limits and requests
• Ingress configuration with HTTPS (letsencrypt)
• ConfigMap and Secret management

# Deploy to any environment with one command
make IMAGE_TAG=sha-123 ENVIRONMENT=prod NAMESPACE=hellok8s deploy

5. Advanced CI/CD with Reusable Workflows

The GitHub Actions setup showcases modern CI/CD patterns:

• Reusable workflow templates for multiple environments
• Docker layer caching for faster builds
• Automated testing and security scanning
• GitOps-style deployments with approvals
• Integration with multiple container registries

Real-World Impact: Measurable Results

This isn’t just a demo project. It’s a distillation of patterns I’ve implemented for companies ranging from early-stage startups to Fortune 500 enterprises. The techniques demonstrated here have delivered measurable improvements:

• Reduced deployment times from hours to minutes (average 15x improvement)
• Eliminated environment drift across development, staging, and production
• Improved security posture no more plain text secrets
• Decreased onboarding time for new developers by 80% (from weeks to minutes)
• Enabled true DevOps practices with infrastructure as code and full CI/CD
• Increased deployment frequency from weekly to multiple times daily
• Reduced production incidents by 60% through better testing and staging practices

How I Can Help Your Organization

This project template reflects years of hands-on experience helping teams overcome production challenges. I offer consulting services in several key areas:

DevOps Transformation & Modernization

• Cloud Migration: Lift-and-shift to cloud-native orchestration with zero-downtime strategies
• CI/CD Pipeline Design: From basic automation to advanced GitOps workflows
• Infrastructure as Code: Terraform, Helm, and cloud provider best practices
• Kubernetes Strategy: Container optimization, security hardening, best practices

Application Development Excellence

• Architecture Design: Scalable patterns for high-traffic applications across frameworks
• Performance Optimization: Database tuning, caching strategies, and async processing
• API Development: RESTful APIs with proper authentication and rate limiting
• Modern Tooling: Adoption of cutting-edge development tools and practices for any tech stack

Cloud-Native Architecture

• Microservices Strategy: When, how, and what to decompose from monoliths
• Event-Driven Systems: Message queues, event sourcing, and distributed architectures
• Observability: Comprehensive monitoring, logging, and alerting strategies
• Cost Optimization: Right-sizing resources and implementing FinOps best practices

Team & Process Excellence

• Developer Experience: Reducing friction in daily workflows and improving productivity
• Quality Processes: Code review standards, testing strategies, and quality gates
• Team Scaling: Onboarding processes and knowledge management systems
• Technical Leadership: Architecture reviews, technology selection, and strategic planning

Let’s Talk

Whether you’re struggling with deployment complexity, seeking to modernize your development practices, or planning a cloud migration, the patterns demonstrated in this template can transform your organization’s engineering capabilities. Reach out now using this contact form.

Common Scenarios Where I Can Help:

• “Deployments take hours and often fails” - Implement automated CI/CD with rollback capabilities
• “New developers take weeks to get productive” - Create reproducible development environments
• “Afraid to deploy on Fridays” - Build confidence through automated testing, preview deployments, easy rollbacks and observability tooling.
• “Managing secrets is a nightmare” - Implement secure, automated secrets management (sops, Hashicorp Vault, etc)
• “Cloud costs are spiraling out of control” - Optimize resource usage and implement cost governance

I’m available for hands-on consulting engagements ranging from focused architecture reviews to full-scale DevOps transformations. Whether you need a quick assessment or comprehensive implementation, let’s discuss how these proven practices can accelerate your team’s delivery and reliability.

Ready to get started? This project serves as a perfect conversation starter, regardless of your current technology stack. We can review your current setup, identify pain points, and create a roadmap for improvement that delivers measurable results. Reach out now using this contact form.

The complete source code and documentation for hellok8s-django is available on GitHub. Feel free to use it as a starting point for your own projects, or reach out if you’d like help implementing these patterns in your organization.

It initially started as a project to replace my old 2-bay Synology NAS. I needed an upgrade, storage wise, but I also wanted to switch to ZFS, and since Synology doesn’t support ZFS I knew I’d have to build my own rig. That in turn meant I needed to figure out what I was going to install on the thing. Initially I looked at openmediavault but it didn’t seem to support ZFS at the time (I don’t know if this is still true). Then I looked at FreeNAS as they were going through their rebranding to TrueNAS - but I was weary about switching to FreeBSD at the time. I wanted something that I knew well (Linux) and that I can easily customize. Before you mention TrueNAS Scale (which is Linux based) it wasn’t ready at the time¹ and I was hesitant about all the cruft that comes pre-installed with it (my main issue with Synology). I guess I wanted to install and configure stuff my way when I need it.

So…Since I was going to just install plain ol’ Linux on this thing I figured there’s no reason to restrict it to being just a NAS. That’s when things got a bit out of hand and I went all out. :)

Here’s a list of components I ended up going with:

• Case: Silverstone CS381
• Motherboard: Supermicro X12STH-F
• CPU: Intel Xeon E-2388G
• Noctua NH-L9i (initially Noctua L9x65 but it didn’t fit)
• RAM: 32GB DDR4 ECC (x2)
• Samsung Evo 970 Plus 1TB
• Samsung 870 Evo 4TB (x2)
• WD Red Plus WD60EFZX 6TB (x2)
• Be-Quiet Silent Wings 3 (x2)
• Noctua NF-A9x14 (x2)
• DeLock Kabel Mini SAS HD SFF 8643 x4
• PSU: Be quiet! SFX-L Power 600W 80+ Gold Power Supply
• Mini SAS HD (SFF 8643) to 4X Sata (Reverse Breakout Cable)

With that kind of power I knew I had to utilize this thing beyond just a NAS machine. Since most of my machines these days (including my other homelab server - an Intel NUC) already run NixOS, that’s what I decided to go with. At the time I was doing some heavy duty projects that seemed to prove too big to handle for my slightly outdated X1 Carbon laptop, so I set it up as a remote nix builder among other things. But that’s something for a future blog post. It’s been running for a while now and I’m quite happy with it.

Here are some photos for your viewing pleasure:

I’ve started doing regular backups of my machines after suffering 2 consecutive drive failures with one of my laptops. After something like that (which BTW never happened since XD) you kind of realize that backups are not just a nice thing to have, but a hard requirement. In this blog post I’ll outline how I’ve set up my on-site backup solution.

When two of my hard drives failed I fortunately didn’t loose a lot of data. I used to rsync some of my stuff between 2 machines (but not very often), my work stuff was pushed to remote git repos, I used various cloud storages for some stuff etc. Still, there were some things that I lost with the drives and it was annoying and time consuming to get up and running again. This should not be the case.

After the second incident I decided I would do something about it and set up an on-site backup solution. Why on-site? Well, performance mostly, but I also didn’t trust any of the cloud based solutions (I use tarsnap now but that’s another blog post :)). So - I bought a NAS¹ and went looking for a backup tool.

NOTE: I’ve been sitting on this blog post for 3 years. So bear in mind that some things might have changed since then which I didn’t bother to get up to date on. However, I’m still happily using the method described in this post.

Requirements

While searching the web for potential candidates I put together a couple of requirements:

• The tool needs to be OSS
• Backups needs to be encrypted
• Backups need to be incremental
• Backups need to reside on a local NAS I have running in my homelab
• The backups needs to happen regularly without needing my input from a human

NOTE: Since I only run Linux machines I didn’t care if the tool was cross platform.

Borg backup

I quickly found that borg-backup would be a great fit for my use case. It’s a very widely known (and used) tool that seemed like it checked all of the boxes above (except the last one…but we’ll get to that).

Borg was fairly easy to set up and run. The documentation is excellent. I did fiddle a bit with the ignorelist (so that it would not backup useless files like caches etc) but after that it basically just worked.

Borg is designed as a client/server system. This means that you have to install the server component on your storage device (where the backups are going) so that the client (ie. your laptop) can communicate with it. You can read more about it here. At the time my NAS didn’t have an official/supported way of installing borg² so I opted for the client-only approach where I just mount the backup folder via NFS and let the client do it’s thing. NFS is pretty battle tested and I didn’t see this as a huge downside. Although performance does suffer a bit, it has generally worked well so far.

To run it automatically I wrapped the entire thing in a shell script and set it to run daily at a certain time - and the simplest possible way was to use cron to do that.

This is the lowest possible amount of effort I needed to make to get an “out of sight out of mind” backup system working without me having to babysit it.

Problems

Basically borg was checking all the boxes except the last one. How do I run backups automatically without needing my input or attention.

There were 3 main problems with the above approach:

• What if my laptop was not turned on during the time of day that the backup is scheduled

The simplest possible way to solve this is to try and re-run the backup scripts every so often - between the hours when the laptop is most likely to be running. It was important that the backups didn’t happen in the middle of the day because they can sometimes be a bit CPU intensive if I’m doing X other things (encryption is client side).

I decided to just re-run the script every hour between hours 19h and 23h. I figured this would give the backups enough chance to run (given my laptop usage patterns).

30 19-23 * * * /path/to/my_backups.sh

Side note: check out crontab.guru for a more detailed explanation of the cron syntax.

This solution required that I change my shell script in such a way that it first checks if a backup for the day has already been done. It not, then run the backup, otherwise just exit. Doesn’t seem that big of a deal right?

Well, even small changes in shell scripts can make them more complex and less maintainable.

In retrospect a better solution would most likely be if I had just used anacron which is more suited for systems that don’t run 24/7.

• What if I turn off my laptop in the middle of a backup?

I never really investigated too much what would happen here. I assume if it’s a clean shutdown borg would just cleanly abort and do a backup in the next backup window. Unclean shutdowns are another thing. Usually you end up with a lock file that didn’t get cleaned up and no future backups can run until you manually sort it out.

I chose to ignore this and just make sure that the human (me) is always notified when a backup starts and when it finished. On Linux this is pretty easy with libnotify. I use notify-send to send the start and finish notifications:

notify-send -u normal "backup started..."
notify-send -u normal "backup finished!"

This would catch my attention and I would know not to power off the machine if the backups haven’t finished yet. If I somehow forgot that I dismissed the notification my window manager makes it really easy to cycle through old notifications and verify if the backup was done. It’s not perfect but it works for me.

• What if I’m not connected to my LAN and therefore cannot mount my NFS drive?

This was the biggest issue with the above approach. I could just let the cronjob fail as it would be picked up later anyway - but I didn’t like that.

I briefly thought about extending my shell script wrapper with the ability to check if I’m connected to my home WiFi and only then start the backup. This in turn made the shell script even more complex and hard to maintain. I don’t know about you but when I have a lot of conditional logic in shell scripts things start to fall apart really quickly.

As a friend of my often says “nobody can write even shell one-liners without bugs” - and I tend to agree with him!

I briefly considered rewriting it in Python - and that would have been fine - but I was writing most of my tooling in Haskell at that time.

And so borg-runner was born!³

Now, instead of a brittle shell script, I had a proper binary I can configure with a config file (yes it’s YAML). I still use cron to run it though. I was thinking of making a proper daemon out of it but decided against it. It was too complicated for not much gain.

And there you have it. Backups for folks like me who don’t yet use ZFS! :D I jest but, I’ll likely be migrating my machines to ZFS starting next year so I’m not sure how much I’ll need these tools but we’ll see.

I mentioned at the beginning that I also use Tarsnap for off-site backups (as well as for my servers). Why have both? Redundancy is always a good thing - if my local backups get corrupted I can always turn to my cloud backups or vice versa. Also, you never really know if your backups work if you don’t test restoring from a backup often. I still don’t do that part very often so I figured it’s best to have two systems just in case.

I also talked about the importance of being explicit about which environment you’re currently working on, easily switching between and deactivating an environment so that we don’t run accidental commands in the wrong context.

Continuing this series, in this post I will talk about how to effectively switch between different AWS accounts.

The problem

In my “day to day” I usually interact with multiple AWS accounts tied to various clients. I usually do this from the command line while using tools like terraform, packer, kops, AWS CLI etc.

These AWS accounts are usually set up with some best practices in mind:

• Mandatory MFA

This means that we require each user to sign into the AWS Console and set up a MFA device. This can be a virtual device like Google Authenticator or it can be a physical device like a Yubikey.

• No direct permissions are assigned to users
• Instead users have to assume a role depending on which permissions they need for the task at hand

At work we developed a terraform module that sets all of this up for us.

• Use of short lived STS session credentials

What this means is that we use “short term credentials” from AWS STS which we then use to request (and refresh) role based credentials if we’re assuming a role. For certain endpoints AWS requires MFA when using STS so denv makes sure to enforce it. This means that the users aws_access_key_id and aws_secret_access_key are only used at the beginning to fetch the temporary STS credentials. The STS credentials are then cached and used to assume a role and fetch another set of credentials that have permission to actually do things on AWS (like create EC2 instances, RDS databases and similar). The user is prompted for the MFA token during the initial request and not again during the entire cache period.

Session credentials expire after 36 hours¹ and role credentials expire by default after 1 hour ², after which they need to be refreshed.³

When defining access credentials for AWS accounts you’re going to be dealing with 2 files:

• ~/.aws/config

and

• ~/.aws/credentials

These files can be configured using the AWS CLI but since it’s a straight forward ini format I usually edit them in my editor.

The non-solution

One could argue that you can just prefix each command with the AWS_PROFILE=myprofile environment variable and be done with it. This is true, and most tools do support this way of specifying which AWS profile to use. However, the trouble is it only works for the naive approach, just exporting the aws access key and secret key. It does not work if we’re also using MFA and Role Assuming.

The solution

First things first.

If you have already setup your ~/.aws/config and ~/.aws/credentials files, make sure to delete (or rename) the [default] entry. We don’t want to have a default profile. Instead, we want each profile to have its own unique name that forces us to specify the profile we’re using.

NOTE: Make sure to set the correct permissions: chmod 600 ~/.aws/config && chmod 600 ~/.aws/credentials

Now let’s look at 3 use-cases that you might encounter.

Case 1: MFA and assuming roles

In this case your configuration files will look like this:

#### ~/.aws/config ####

[profile project1]
region=us-east-1

[profile project1-prod-admin]
role_arn=arn:aws:iam::....:role/admin
mfa_serial=arn:aws:iam::....:mfa/deni
source_profile=project2


##### ~/.aws/credentials #####

[project1]
aws_access_key_id=.....
aws_secret_access_key=.....

Alright, let’s break down what happens when we run the following command:

~ denv aws -p project1-prod-admin
Enter MFA code for device "arn:aws:iam::...:mfa/deni": xxxx

Once we enter our MFA code (using Google Authenticator or a similar app):

aws|project1-prod-admin  ~  export | grep AWS
AWS_ACCESS_KEY_ID=xxxx
AWS_DEFAULT_REGION=us-east-1
AWS_REGION=us-east-1
AWS_SECRET_ACCESS_KEY=xxxx
AWS_SECURITY_TOKEN='xxxxx'
AWS_SESSION_TOKEN='xxxxxx'
_DENV_SET_VARS=AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_SESSION_TOKEN,AWS_SECURITY_TOKEN,AWS_DEFAULT_REGION,AWS_REGION,_OLD_DENV_PS1,_DENV_SET_VARS

aws|project1-prod-admin  ~  aws sts get-caller-identity
{
    "UserId": "...",
    "Account": "...",
    "Arn": "arn:aws:sts::...:assumed-role/admin/..."
}

We see that in the newly spawned shell we have all the required environment variables set that all the standard tools will look for. As with the kubernetes example in the previous post, the prompt is annotated with the account name so that we don’t forget which account we’re working on.

NOTE: aws sts get-caller-identity is just an example command. You might be more interested in running something like aws ec2 describe-instances or similar.

Once we’re done, we can run denv deactivate to make sure that all the environment variables are unset from the current shell. This way we make sure that if we run any other AWS CLI command we’re not accessing an account by mistake.

This form of using the denv aws subcommand is called the eval form. Since these credentials expire I recommend using the exec form that looks like this:

~ denv aws -p project1-prod-admin -- aws sts get-caller-identity

The exec form makes sure to execute your command in the required “context” with all the above variables set and once it’s done, none of them get leaked to your currently running shell.

This form is preferable because it will always make sure to fetch fresh credentials if they’re needed. This is not the case with a long running shell prompt because the credentials exposed in that shell might be stale.

Let’s break down what denv aws -p project1-prod-admin does:

1. Parse and fetch the configuration and credentials from the standard files mentioned above
2. Check if we need to assume a role
3. If we do, check if we have valid cached credentials for that role
4. If we don’t have valid cached credentials check if we have valid cached session credentials that we’re using to fetch the above role credentials
5. If we don’t have session credentials request new STS session credentials
6. Request new role credentials with the above session credentials
7. Export the role credentials into the current shell or run the supplied command while making the credentials available.

These series of steps are taken with each denv aws run. The cached credentials are stored in ~/.aws-env/.

Case 2: Assuming a role but without MFA

In this case your configuration would look like this:

#### ~/.aws/config ####

[profile project2]
region=us-west-2

[profile project2-prod-admin]
role_arn=arn:aws:iam::....:role/admin
source_profile=project2

##### ~/.aws/credentials #####

[project2]
aws_access_key_id=.....
aws_secret_access_key=.....

Activate this environment by running: denv -p project2-prod-admin. The main difference between this case and Case 1 is that we’re not going to use intermediate session credentials from STS. Instead, we’re going to use the raw credentials from ~/.aws/credentials to fetch our temporary role credentials. The reason we’re skipping the session credentials is because AWS requires Multi Factor Authentication to access any AWS IAM APIs. ⁴

Case 3: Using raw credentials

#### ~/.aws/config ####

[profile project3]
region=us-west-2

##### ~/.aws/credentials #####

[project3]
aws_access_key_id=.....
aws_secret_access_key=.....

The last case is the “naive case” where denv is only exporting the aws_access_key_id and aws_secret_access_key defined in the above configuration file. It’s not talking to the AWS API and fetching any short term or role credentials of any sort. I find denv is still useful in this case since it explicitly shows me which account I’m working on. Using the eval form for this case is safe and saves us a bunch of keystrokes (which I’m a big fan of).

How it works

We’re using Haskell and some awesome libraries but the most interesting one is amazonka which we use to talk to the AWS APIs and fetch temporary session and role credentials.

The logic I outlined above is encoded like so:

env <- runMaybeT $
  MaybeT (runRIO awsEnv $ maybe nop (getFromRoleCache p) roleArn) <|>
  MaybeT (runRIO awsEnv $ maybe nop (maybe nop1 (getSTSWithRole awsenv region' lgr p sourceProfile) mfaSerial) roleArn) <|>
  MaybeT (runRIO awsEnv $ maybe nop (getFromSessionCache p sourceProfile) mfaSerial) <|>
  MaybeT (runRIO awsEnv $ maybe nop (getSTS awsenv region' lgr p sourceProfile) mfaSerial) <|>
  MaybeT (runRIO awsEnv $ maybe nop (useRawWithRole awsenv region' lgr p sourceProfile) roleArn) <|>
  MaybeT (runRIO awsEnv $ useRaw key' secret')

Fetching STS credentials is fairly straightforward:

mkStsSessionTokenRequest :: MfaSerial -> STS.GetSessionToken -> IO STS.GetSessionToken
mkStsSessionTokenRequest (MfaSerial mfaSerial) st = do
  req <- do
        tokenCode <-
          promptLine $
          "Enter MFA code for device " <> show mfaSerial <> ": "
        return $
          st & STS.gstTokenCode .~ (Just $ T.pack tokenCode) &
          STS.gstSerialNumber .~ (Just mfaSerial) &
          STS.gstDurationSeconds .~ (Just $ fromIntegral defaultSessionDurationSeconds)
  return req

And actually running the request:

execStsRequest :: AWS.AWSRequest a =>
     AWS.Env -> AWS.Logger -> AWS.Region ->  a -> IO (AWS.Rs a)
execStsRequest awsenv lgr region req = do
  ret <-
    runResourceT $
    AWS.runAWS (awsenv & AWS.envLogger .~ lgr) $
    AWS.within region $ AWS.send req
  return ret

Once we have the credentials it’s just a matter of going to the entire process of exporting them into the current shell which I outlined in more detail in the previous post.

Summary

We’ve seen how denv aws helps us switch securely between multiple AWS accounts while using short term credentials. Using short lived credentials is encouraged as they are much easier to replace in case of a leak. They usually expire before it becomes a problem. It’s also good that the raw credentials that the user gets from the AWS Console “travel the wire” much less in this scenario (even though it might be over a secure line).

Haskell helped me add this feature to denv fairly easily and while the initial version had even fewer lines than its bash counterpart⁵ it was much easier to maintain and reason about.

This feature is currently still in beta but I’ve been using it for a few months every day and have not found any bugs so far. I will likely remove the “beta” stamp once I allow for more configuration regarding cache expiry times.

Head over to the repo’s releases page and try it out!

I just don’t think it’s possible to write good and maintainable software with bash, no matter how simple the tool might be. In my opinion the main benefit with bash is that it’s so easy to distribute to end users. There’s no special installation or configuration, you just download a script and run it. With haskell I’m aiming towards distributing statically linked binaries (although in simple cases dynamically linked binaries work just fine as well).¹

This series of posts titled “Haskell Showroom” are my attempt at showcasing what Haskell can be used for. It’s an attempt to answer the question I get asked a lot:

"What is Haskell a good fit for?"

Haskell has a very good reputation when it comes to writing compilers but it’s a general purpose programming language and it really can be used for all sorts of things. I mainly use it for writing web apps and CLI tools.

In the first post in this series I will talk about my tool called denv, which is a tool to help me “manage environments”. While it helps me manage (and switch between) various environments, in this post I will focus on how it helps with switching between multiple kubernetes clusters in a sane and simple way.

The problem

I work with a bunch of kubernetes clusters. Some at work for various clients, and some for personal projects. It’s sometimes hard and confusing to switch between these clusters (and namepsaces within a cluster) while confidently knowing which cluster (and namespace) you’re currently working on.

The “default” way kube does this is by having all the configurations for all the clusters written into one single yaml file: ~/.kube/config. Then you have to use a combination of kubectl config set-context and kubectl config use-context which will in turn mutate the global config file, changing the current-context field. This, to me, feels clumsy and does not address the part where we confidently know which cluster (and namespace) we’re working on.²

There are tools that try to address this, such as: kubectx, kubens and kube-ps1. However from what I can tell they all fall short of a couple of things:

• They still rely on a single global config file that they mutate using kubectl behind the scenes.

There’s something about “mutating global state” that just does not sit well with me. ³ Apart from that it’s a hard requirement for me to be able to separate different projects I’m working on.

• All of these appear to be written in bash

I think the main reason tools like this get written in bash is because it’s easy to inject environment variables in the currently running shell. If you write a custom tool it is going to run as a child process of your shell and so it’s not possible for a child process to change environment variables for the parent process (your shell). Later in the post I’ll show a way how to work around this.

The proposed solution

First things first, let’s get rid of the global config file:

# Don't remove the old file. Back it up just in case you need to extract
# auth info for each cluster and break it up into separate files
mv ~/.kube/config ~/.kube/config.old

touch ~/.kube/config
chmod 400 ~/.kube/config`

This should prevent any tools from the k8s ecosystem from modifying that file. However, since we can’t use the default file we have to set the KUBECONFIG environment variable to something like ~/.kube/customer1-prod.yaml. All of the tooling from the k8s ecosystems (like kubectl, kops, helm etc) respects this environment variable. This is especially useful when creating a cluster with kops, given that it saves a configuration file for you once the cluster is created, and so it will write it to a new file rather than mutate the one single file.

So, it’s crucial to use export KUBECONFIG=~/.kube/customer1-prod.yaml before running any kubectl (or similar) commands. This will make sure that we isolate each cluster in it’s own config file.

There’s a problem though. This seems tedious and does not show us which cluster we’re currently working on in our terminal. Also, we have to prepend --namespace=<NAMESPACE> to each kubectl command.

Using “denv”

Let’s install denv.

Add the following alias to your .bashrc or .zshrc:

alias k='kubectl --namespace=${KUBECTL_NAMESPACE:-default}'

Add this hook at the end:

eval "$(denv hook ZSH)"

or eval "$(denv hook BASH)" for bash.

Run: denv kube -p ~/.kube/customer1-prod.yaml -n kube-system to activate the customer1 cluster within the kube-system namespace. Observe how we are told in the prompt which cluster and namespace we’re working on.

You may be thinking how the same thing can probably be done with an .envrc file and direnv that just auto exports the above variables. While I have a lot of respect for direnv, and I still find inspiration for a lot of stuff from their repo, I have 2 issues with it:

1. I don’t like the fact that changing to a different directory will automatically inject things into my environment and automagically change my configuration for who knows what. I much prefer the semantics of the workon command from Python’s virtualenvwrapper where you have to explicitly activate and deactivate an environment before anything is changed in your running session. And it works independent of which directory you’re currently in. ⁴

2. It requires that I change my current working directory to a specific directory where the .envrc file is located. This does not work well with some of my workflows (it would require duplicating the .envrc file in a couple of places).

What denv kube brings to the table is a way that forces you to activate a certain cluster/namespace before being able to do anything with it. It forces you to deactivate that specific cluster if you’re done working with it. This will make sure to unset the above environment variables so there is no way that you have a long running terminal somewhere that has access to a cluster you’re not aware of.

How it works

Let’s define our cluster (I call it project) and namespace that we will parse from the -p and -n flags respectively:

type KubeProjectName = String
type KubeNamespace = String

We also need a way to define what the list of kube specific environment variables we’re dealing with:

data KubeVariable
  = KubeConfig
  | KubeConfigShort
  | KubectlNamespace
  deriving (Eq)

We will provide a Show typeclass instance so that we can convert these to a String later on.

instance Show KubeVariable where
  show KubeConfig = "KUBECONFIG"
  show KubeConfigShort = "KUBECONFIG_SHORT"
  show KubectlNamespace = "KUBECTL_NAMESPACE"

Apart from kube specific variables we have some special variables that we will track:

data SpecialVariable
  = Prompt
  | OldPrompt
  | DenvSetVars
deriving (Eq)

instance Show SpecialVariable where
  show Prompt = "PS1"
  show OldPrompt = "_OLD_DENV_PS1"
  show DenvSetVars = "_DENV_SET_VARS"

Great, now that we have all these types we need a way to tell our program if we want to Set or Unset a given variable. This is important since we also want to be able to deactivate an environment, that is to say, unset all of the environment variables that we’ve injected into our shell session.

data DenvVariable where
  Set :: (Eq a, Show a, Typeable a) => a -> T.Text -> DenvVariable
  Unset :: (Eq a, Show a, Typeable a) => a -> DenvVariable

Let’s go ahead and create our environment. We’ll use the mkKubeEnv function, which will take the project name, namespace and create the environment:

mkKubeEnv :: KubeProjectName -> Maybe KubeNamespace -> IO ()
mkKubeEnv p n = do
  checkEnv
  exists <- doesFileExist p
  unless exists (die $ "ERROR: Kubeconfig does not exist: " ++ p)
  let p' = takeFileName p
  let n' = fromMaybe "default" n
  let env =
        withVarTracking
          Nothing
          [ Set KubeConfig $ T.pack p
          , Set KubeConfigShort $ T.pack p'
          , Set KubectlNamespace $ T.pack n'
          , Set OldPrompt ps1
          , Set Prompt $
            mkEscapedText "k8s|$KUBECTL_NAMESPACE|$KUBECONFIG_SHORT $PS1"
          ]
  writeRc env

writeRc is a function with the type signature writeRc :: [DenvVariable] -> IO () and it does 2 things:

1. It converts "Set KubeConfig "~/.kube/customer1-prod.yaml" to export KUBECONFIG=~/.kube/customer1-prod.yaml or Unset KubeConfg to unset KUBECONFIG.
2. Writes all of these environment variables to ~/.denv.

Once the ~/.denv file is in place the eval "$(denv hook BASH)" that we put in our .bashrc will check for it and inject the environment variables into the current shell session. This is actually a very ingenious way of injecting variables from a child process into the parent process (denv -> shell). All credit for the idea goes to direnv authors.

If you’re interested in the details please refer to the source code in the github repo.

Summary

In this post I’ve showed you how I go about switching between multiple kubernetes clusters while averting potential disasters and confusions about which cluster I’m currently working on.

I used Haskell to solve this particular problem because Haskell allows me to easily extend this tool with different functionality, furiously refactor without fear and keep me from shooting myself in the foot by not treating everything as a String ⁵.

This code is far from perfect and could definitely use more type safety but hopefully I was able to demonstrate how Haskell can be not scary (no fancy type gymnastics) and very much usable in an “imperative” way if so required to get the job done (and move on).

In the second post in this series I will talk about denv aws and how to switch between multiple AWS accounts, use temporary credentials, multi factor authentication and key rotation.

That’s not to say I just jumped right in. I spent a long time just playing around with the language, exploring various language features, libraries and getting to know the ecosystem in general. Not doing anything serious ¹.

At the time that was a bit more painful than now because now-a-days there’s a lot more resources and organized materials around. The community has really stepped up! ²

Anyhow… Building things. I’ve been using docker since it was released and have grown from resident docker fan-boy to resident expert over time. Also I’ve been involved with the Python API wrapper since the early days so it made sense to try and write a Haskell API wrapper. At the time that was the closest I came to web development (the docker daemon listens on a HTTP API) as I felt comfortable.

The first attempt was awful. But it worked! I was able to launch containers and everything. Of course, the API of the library was horrendous. There were very little type guarantees for anything and the whole thing was just one big giant IO blob.

Having realized this, I went on a crusade to learn all the fancy type machinery and make use of every trick and extension under the sun. That attempt went on and off for a couple of months until I realized that I created a monster and that I didn’t need half the stuff I was using. So, having went from one extreme to the other I decided to delete everything and start fresh. I had some guidance from a couple of friends (thank you!) and I was getting pretty close to an API I was (more) happy with and that should be usable by most people. The library being usable was one criteria, but the other one was that it’s more or less straightforward to contribute to. The reason being that the Docker Engine API is kind of huge at this point and I’m certainly either going to miss something, or implement it wrong, and it should be more or less easy for anyone to go in, and contribute a bugfix or a feature. Naturally, this dragged on since obviously I was doing this for fun and in my spare time. Thankfully, I got help somewhere along the way when James Parker jumped in. He was instrumental in getting the library in better shape so that we can finally release that major refactored version.

As you might have guessed this rewrite can’t even compare to the previous release. Not only did the API change, but even the namespace under which the library lives has changed.

This version is far from stable but I think it’s a step in the right direction. Even though the next major release will bring more API changes we’ve decided to release this version anyway, so that the work gets out there, and to have people using it. Feedback (and help) is always welcome (we’re working on a contribution guide). The other reason for releasing, even though there’s still stuff to do, is for me to get over the “it-must-be-perfect-right-away” mindset and do small iterative changes. I looked at the early releases of some very popular libraries in the haskell ecosystem and they all started small, so why shouldn’t I?!

While this is not my first project with Haskell, it’s certainly the one that was most challenging simply because of the fact that it’s a library. Not a project/product, not an executable that I (or others would) use, but a library. It turns out it’s that much harder to get a library right and make it easy to use in other people’s code.

So let’s see how the library progressed as I learned things, how it looked like at the beginning and how it looks like now.

How it all started

The initial version of the library, while it worked (I could create containers with it) was pretty simplistic. For instance, this is how createContainer looked like:

createContainer :: DockerClientOpts -> CreateContainerOpts -> IO (Maybe Text)

We had to pass the client configuration (api url and the like) to each and every function.. And I figured I’ll make it the first argument so if people get tired of passing it in all the time they can partially apply the functions that they use in their code with the DockerClientOpts of their choosing. I had Text and String all around the place… For instance I was constructing my URLs like this:

printf "/containers/%s/start" containerId
printf "%s%s%s" url apiVersion endpoint

I was using lenses and TemplateHaskell even though I didn’t really need them, or understand any of it for that matter. ³ And of course a lot of non-idiomatic haskell was there, syntax wise and more.

After the initial version I thought to myself, hey, I can make this more type safe, this is haskell dammit. So I deleted everything and went down the rabbit hole. And naturally the first thing that happened was that the number of GHC extensions grew. These are some of the extension that I suddenly discovered that I needed:

{-# LANGUAGE DeriveFunctor     #-}
{-# LANGUAGE FlexibleContexts  #-}
{-# LANGUAGE DataKinds         #-}
{-# LANGUAGE TypeFamilies      #-}
{-# LANGUAGE DeriveFunctor     #-}
{-# LANGUAGE DeriveGeneric     #-}
{-# LANGUAGE GADTs             #-}
{-# LANGUAGE RankNTypes        #-}
{-# LANGUAGE RecordWildCards   #-}

And that’s not all of them… So what was going on?

Among other things, is was using Free monads to construct an interpreter for HTTP requests and I was using singleton types to make sure that I couldn’t accidentally return a list of Image ID’s with the endpoint that’s supposed to return Container ID’s. This was making the library much more complex, and while it provided type safety for the library author, it didn’t provide much benefit for the library user. In fact, the library was harder to use, and so much more harder to contribute to.

The beauty of haskell, to me, is that I can come back to a codebase after a few months and instantly know what’s what, and be able to continue working on as if I didn’t have a pause at all. This, to me, is the single most beneficial feature of haskell. This was very important to this project especially…since I was basically just playing around and would often have breaks like that (couple of months) before coming back to the project again. So the turning point was when I lost this ability…

Now, while this was a wonderful learning exercise, it was a complete overkill for what I was trying to do. Some of the ideas were sound, like using Reader and not have to pass in the DockerClientOpts to every single function, but others were not. On the one hand I was practically dabbling with dependant typing and on the other I was still concatenating strings for URL’s (printf) and returning things like Maybe Text. This just wouldn’t do.

At that point I talked to a friend of mine who’s an experienced Haskell hacker and he said the same thing: get rid of all the extensions and start small. So I did this for the nth time:

commit 2d064140910b69c7b1337f4f1f508dd6c9f3109a
Author: Deni Bertovic
Date:   Wed Mar 2 15:28:59 2016 +0100

    Delete everything. Start from scratch.

Current version

After many more trials and errors I came to an API that I like and now you can do this:

import Docker.Client

runNginxContainer :: IO ContainerID
runNginxContainer = runDockerT (defaultClientOpts, defaultHttpHandler) $ do
    let pb = PortBinding 80 TCP [HostPort "0.0.0.0" 8000]
    let myCreateOpts = addPortBinding pb $ defaultCreateOpts "nginx:latest"
    cid <- createContainer myCreateOpts (Just "myNginxContainer")
    case cid of
        Left err -> error $ show err
        Right i -> do
            _ <- startContainer defaultStartOpts i
            return i

I’m using ReaderT to pass in the defaultClientOpts used for configuring the Client. DockerT is just a wrapper around ReaderT:

newtype DockerT m a = DockerT {
    unDockerT :: Monad m => ReaderT (DockerClientOpts, HttpHandler m) m a
}

And I’m also passing in a default HTTP handler. For most people the defaultHttpHandler will be sufficient, but this leaves room for advanced users to provide their own. The handler’s type is:

type HttpHandler m = Request -> m (Either DockerError Response)

And now the type of createContainer function looks like this:

createContainer :: forall m. Monad m => CreateOpts -> Maybe ContainerName -> DockerT m (Either DockerError ContainerID)

It’s polymorphic in terms of the Monad that’s used and it’s determined by the how the HTTP handler passed in looks like.

runDockerT :: Monad m => (DockerClientOpts, HttpHandler m) -> DockerT m a -> m a
runDockerT (opts, h) r = runReaderT (unDockerT r) (opts, h)

There were some other considerations like how to handle exceptions. I talked to Michael Snoyman about this and while I agree that in the end if you’re ultimately using IO you will have exceptions so there’s no reason to hide those and introduce multiple layers of error handling ⁴. That being said I wanted to make it so that most people don’t even have to know that they are talking to a HTTP API. That’s why we have the Either in the response and don’t just pass the HttpException along to the user. ⁵ People will be trying to use Docker, and they should only have to think in Docker semantics and not worry if the communication is over HTTP or something else. There’s still some work to be done on that front but that’s the general idea.

The fact that we can pass in the http handler should be enough for most people, and should come in handy for writing some interesting tests I think.

I intentionally left the entire git history, with all of these various attempts in the repo, so that I (and anyone else) could refer to them later. It paints a good picture of what it’s like to explore haskell and try different approaches.

I found this design to be somewhere in between simplicity and power. As I already said above, any feedback is appreciated and I’d be very much interested to hear about any advice, tip or critique.

What’s next:

The biggest challenge that’s up next is described in this Github issue. It has to do with streaming. After we figure that out the API should become more stable.

Fin

That’s all from me for now. I’m sharing this in the hopes that my story will help someone else that’s just starting out with haskell by showing that it’s okay to start small, and make a lot of mistakes along the way.

If you’ve liked what you’ve read please share it and comment down below. If you want to learn more about how to manipulate docker containers with Haskell please see the docs here.

P.S. If you want to look at some other Haskell code that I wrote check out these projects:

• A web scraper

I had to move, and was looking for new apartments to rent, but the good ones go so fast so I needed to know when new ads were published as soon as possible. The result is this scraper, for the local ads website, that would notify me about new listings (based on some criteria) via email. I enjoyed using optparse-applicative for the CLI parts.

• Parser for bank statement csv/html files

I wanted to use hledger for my personal accounting but didn’t want to manually enter each statement into the journal file.

• A Trello like REST API

I wanted to learn a few frontend frameworks, and new languages like Elm and Purescript, but wanted to do so building a real(ish) project and not just a Todo app. Something like a subset of Trello seemed to have the right amount of complexity in it so I decided to build a dummy-api that I could use for testing. I built it using the excellent Servant library and learned a lot in the process.

Before we begin let me explain what are Docker Volumes and what they’re used for. The official Docker docs explain this feature as follows:

A data volume is a specially-designated directory within one or more containers
that bypasses the Union File System.

The main use-case for volumes is for persisting data between container runs (seeing as container are ephemeral). This is useful for data directories when running databases (such as PostgreSQL) within containers. Other than persisting databases it’s useful for sharing code folders from your host system to the container when running in your development environment.

However, there are 2 problems we have here:

1. If you write to the volume you won’t be able to access the files that container has written because the process in the container usually runs as root.
2. You shouldn’t run the process inside your containers as root but even if you run as some hard-coded user it still won’t match the user on your laptop/jenkins/staging.

The permissions problem is most annoying in development and testing environments because usually at some point you want to remove files that the process running in the container has created but you can’t because on your laptop you’re running as UID 1000 (on most Linux machines) and the files are owned either by UID 0 (root) or by some other UID that was perhaps hardcoded in the Dockerfile.

RUN  useradd --shell /bin/bash -u 1024 -o -c "" -m myuser
RUN mkdir -p /shared/tmp && chown user. /shared/ -R
USER myuser
CMD /usr/local/bin/myprocess

This solution is inadequate because you hard-code the UID of the user in the build process and even though your process won’t be running as root it’s still running as a user that’s:

1. Not present on your local machine
2. The UID of the user is not 1000 (ie. your UID) and you still won’t be able to cleanup files in the /shared/tmp directory

Docker provides a -u flag with it’s run command to dynamically switch to a specified UID during container start. So we can write something like this:

deni@kanta:~$ docker run -it -u `id -u $USER` debian:jessie /bin/bash
I have no name!@dcb415bad433:/$ id
uid=1000 gid=0(root) groups=0(root)

This approach, while dynamic in the sense that the UID is specified at runtime, has 2 drawbacks:

1. The GID (group id) of the user is still 0 (root)
2. The UID 1000 is not present in the container’s /etc/passwd file.

While no. 1. is definitely problematic for obvious reasons no. 2. is where we hit a wall. Now while the Linux Filesystem doesn’t really care about user names, rather just UID’s, some applications will refuse to start if the user is not present in /etc/passwd.

So what we need is something like -u but that doesn’t just use the UID of our user but actually creates a user with that UID and then starts the process owned by it.

To do that we have to create a base Dockerfile from which all of our other Dockerfiles will inherit. That Dockerfile should look something like this.

FROM debian:jessie

RUN apt-get update && apt-get -y --no-install-recommends install \
    ca-certificates \
    curl

RUN gpg --keyserver ha.pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4
RUN curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/1.4/gosu-$(dpkg --print-architecture)" \
    && curl -o /usr/local/bin/gosu.asc -SL "https://github.com/tianon/gosu/releases/download/1.4/gosu-$(dpkg --print-architecture).asc" \
    && gpg --verify /usr/local/bin/gosu.asc \
    && rm /usr/local/bin/gosu.asc \
    && chmod +x /usr/local/bin/gosu

COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh

ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

In this base Dockerfile we’re installing a tool called gosu and setting an entrypoint. An entrypoint is basically a script that gets executed before any other command that you might pass to your container. So unless we overwrite the entrypoint we are guaranteed to go through this script every time we launch our containers, before we actually run our actual process.

In fact the CMD statement from the Dockefile or from docker CLI gets passed to the entrypoint.sh script as command line arguments. The reason we’re installing gosu is because we will need it to switch to the newly created user.

NOTE: The reason why we don’t use sudo is explained in gosu repo’s README.

Now let’s look at the entrypoint.sh script:

#!/bin/bash

# Add local user
# Either use the LOCAL_USER_ID if passed in at runtime or
# fallback

USER_ID=${LOCAL_USER_ID:-9001}

echo "Starting with UID : $USER_ID"
useradd --shell /bin/bash -u $USER_ID -o -c "" -m user
export HOME=/home/user

exec /usr/local/bin/gosu user "$@"

What we’re doing here is fetching a UID from an environment variable, defaulting to 9001 if it doesn’t exist, and actually creating the user “user” with the familiar useradd command while setting it’s UID explicitly.

And lastly we use gosu to execute our process "$@" as that user. Remember CMD from a Dockerfile or command from docker CLI gets passed to the entrypoint.sh script as command line arguments.

Now to build our base image:

deni@kanta:~$ docker build -t mybase .

And create our new child image:

FROM mybase

CMD ["/bin/bash"]

Build it:

deni@kanta:~$ docker build -t myimage .

Run it:

deni@kanta:~$ docker run -it myimage
Starting with UID : 9001
user@056b9bb45214:/$ id
uid=9001(user) gid=9001(user) groups=9001(user)

Run it with passing in our UID:

deni@kanta:~$ docker run -it -e LOCAL_USER_ID=`id -u $USER` myimage
Starting with UID : 1000
user@fc07b6c32b4f:/$ id
uid=1000(user) gid=1000(user) groups=1000(user)

Done! Now remember, the reason this works is because the Filesystem doesn’t really care if the user is called “user” or “deni” or “jenkins”. It only cares about the UID attached to that user, so the permissions will be preserved and various applications will not complain that there is no user with that UID.

Conclusion

When using docker containers it’s a bad idea to run your processes as root (some applications even refuse to run as root). While running as root or any other hard-coded user it’s hard to work with volume mounts because the files being written from within the container are going to be owned by a different user. That makes working with them or cleaning them up hard and needing to resort to sudo or similar. Which is increasingly annoying in development and CI environments.

In this post I’ve showed you a technique that you can use to build all of your images off of a base image (which you’re probably already doing) that will allow you to start as whatever user you specify making sure to create that user in the process.

If a UID is specified, the container will start as that user, and if no UID is specified it will start as a default user with a random UID that should not collide with any existing users in docker images. (Aaand it’s over 9000!)

So we’re taking care of the permission issue and not allowing the containers to start as root all in one.

If this was helpful please consider following me on twitter.

While working on some projects that used Celery for a task queue I’ve gathered a number of best practices and decided to document them. Nevertheless, this is more a rant about what I think should be the proper way to do things, and about some underused features that the celery ecosystem offers.

No.1: Don’t use the database as your AMQP Broker

Let me explain why I think this is wrong (aside from the limitations pointed out in the celery docs).

A database is not built for doing the things a proper AMQP broker like RabbitMQ is designed for. It will break down at one point, probably in production with not that much traffic/user base.

I guess the most popular reason people decide to use a database is because, well, they already have one for their web app, so why not re-use it. Setting up is a breeze and you don’t need to worry about another component (like RabbitMQ).

Not so hypothetical scenario: Let’s say you have 4 background workers processing the tasks you’ve put in the database. This means that you get 4 processes polling the database for new tasks fairly often, not to mention that each of those 4 workers can have multiple concurrent threads of it’s own. At some point you notice that you are falling behind on your task processing and more tasks are coming in than are being completed, so naturally you increase the number of workers doing the task processing. Suddenly your database starts falling apart due to the huge number of workers polling the database for new tasks, your disk IO goes through the roof and your webapp starts being affected by this slow down because the workers are basically DDOS-ing the database.

This does not happen when you have a proper AMQP like RabbitMQ because, for one thing, the queue resides in memory so you don’t hammer your disk. The consumers (the workers) do not need to resort to polling as the queue has a way of pushing new tasks to the consumers, and if the AMQP does get overwhelmed for some other reason, at least it will not bring down the user facing web app with it.

I would go as far to say that you shouldn’t use a database for a broker even in development, what with things like Docker and a ton of pre-built images that already give you RabbitMQ out of the box.

No.2: Use more Queues (ie. not just the default one)

Celery is fairly simple to set up, and it comes with a default queue in which it puts all the tasks unless you tell it otherwise. The most common thing you’ll see is something like this:

@app.task()
def my_taskA(a, b, c):
    print("doing something here...")

@app.task()
def my_taskB(x, y):
    print("doing something here...")

What happens here is that both tasks will end up in the same Queue (if not specified otherwise in the celeryconfig.py file). I can definitely see the appeal of doing something like this because with just one decorator you’ve got yourself some sweet background tasks. My concern here is that taskA and taskB might be doing totally different things, and perhaps one of them might even be much more important than the other, so why throw them both in the same basket? Even if you’ve got just one worker processing both tasks, suppose that at some point the unimportant taskB gets so massive in numbers that the more important taksA just can’t get enough attention from the worker? At this point increasing the number of workers will probably not solve your problem as all workers still need to process both tasks, and with taskB so great in numbers taskA still can’t get the attention it deserves. Which brings us to the next point.

No.3: Use priority workers

The way to solve the issue above is to have taskA in one queue, and taskB in another and then assign x workers to process Q1 and all the other workers to process the more intensive Q2 as it has more tasks coming in. This way you can still make sure that taskB gets enough workers all the while maintaining a few priority workers that just need to process taskA when one comes in without making it wait to long on processing.

So, define your queues manually:

CELERY_QUEUES = (
    Queue('default', Exchange('default'), routing_key='default'),
    Queue('for_task_A', Exchange('for_task_A'), routing_key='for_task_A'),
    Queue('for_task_B', Exchange('for_task_B'), routing_key='for_task_B'),
)

And your routes that will decide which task goes where:

CELERY_ROUTES = {
    'my_taskA': {'queue': 'for_task_A', 'routing_key': 'for_task_A'},
    'my_taskB': {'queue': 'for_task_B', 'routing_key': 'for_task_B'},
}

Which will allow you to run workers for each task:

celery worker -E -l INFO -n workerA -Q for_task_A
celery worker -E -l INFO -n workerB -Q for_task_B

No.4: Use Celery’s error handling mechanisms

Most tasks I’ve seen in the wild don’t have a notion of error handling at all. If a task fails that’s it, it failed. This might be fine for some use cases, however, most tasks I’ve seen are talking to some kind of 3rd party API and fail because of some sort of network error, or other kind of “resource availability” error. The most simple way we can handle these kinds of errors is to just retry the task, because maybe the 3rd party API just had some server/network issues and it will be back up shortly, why not give it a go?

@app.task(bind=True, default_retry_delay=300, max_retries=5)
def my_task_A():
    try:
        print("doing stuff here...")
    except SomeNetworkException as e:
        print("maybe do some clenup here....")
        self.retry(e)

What I like to do is define per task defaults for how long should a task wait before being retried, and how many retries is enough before finally giving up (the default_retry_delay and max_retries parameters respectively). This is the most basic form of error handling that I can think of and yet I see it used almost never. Of course Celery offers more in terms of error handling but I’ll leave you with the celery docs for that.

No.5: Use Flower

The Flower project is a wonderful tool for monitoring your celery tasks and workers. It’s web based and allows you to do stuff like see task progress, details, worker status, bringing up new workers and so forth. Check out the full list of features in the provided link.

No.6: Keep track of results only if you really need them

A task status is the information about the task exiting with a success or failure. It can be useful for some kind of statistics later on. The big thing to note here is that the exit status is not the result of the job that the task was performing, that information is most likely some sort of side effect that gets written to the database (ie. update a user’s friend list).

Most projects I’ve seen don’t really care about keeping persistent track of a task’s status after it exited yet most of them use either the default sqlite database for saving this information, or even better, they’ve taken the time and use their regular database (postgres or otherwise).

Why hammer your webapp’s database for no reason? Use CELERY_IGNORE_RESULT = True in your celeryconfig.py and discard the results.

No.7: Don’t pass Database/ORM objects to tasks

After giving this talk at a local Python meetup a few people suggested I add this to the list. What’s it all about? You shouldn’t pass Database objects (for instance your User model) to a background task because the serialized object might contain stale data. What you want to do is feed the task the User id and have the task ask the database for a fresh User object.

Actually I was really looking forward to it, but as Linux goes I was expecting it to be a pain, so I was pleasantly surprised that it only took me 10 minutes and wasn’t a hassle at all.

I’ve decided to document the steps I’ve taken in case that someone finds it useful.

First some prerequisites, make sure that you’ve got your system up2date.

sudo apt-get update && sudo apt-get upgrade

Make sure systemd is actually installed (it should be but check anyway):

sudo apt-get install systemd

Before you switch permanently it’s recommended that you try a one-time-boot with systemd just to see if everything’s okay. Reboot your machine and on the grub screen select the kernel you wish to boot and press e. Then at the end of the kernel line add init=/bin/systemd and boot your machine. If everything is fine continue with making systemd a permanent choice.

The recommended way to replace init with systemd is to install the package systemd-sysv which takes care of everything.

sudo apt-get install systemd-sysv

Edit /etc/default/grub and add init=/bin/systemd at the end of the line GRUB_CMDLINE_LINUX_DEFAULT. Save the file and run the following command to update grub:

sudo update-grub2

Reboot and you should be running systemd (use ps aux and check for PID 1).

One last thing with Debian is to set up peristent logging with systemd’s logging component called journal. By default journal will log to /run which is ephemeral meaning the logs will disappear after reboot. The process of making the logs persistent (if you choose to do so) is documented here: /usr/share/doc/systemd/README.Debian Basically you just need to run this:

install -d -g systemd-journal /var/log/journal
setfacl -R -nm g:adm:rx,d:g:adm:rx /var/log/journal

It’s worth noting that you don’t have to use journal if you don’t want to, it’s designed to co-exist with syslog, that’s already running on your system, so you can continue to use that. I personally find journal awesome and would recommend that you at least check it out and see what it brings to the table.

And that’s it folks, now go read more stuff about systemd here.

Thanks to babilen on irc.oftc.net for taking the time and explaining everything patiently :).

This is handy but it has its drawbacks because you don’t get any log rotation and the file size of the collected logs can become an issue as it eats up your host’s disk space. Not to mention the fact that every time you run docker logs container_id you get all the logs of that processes from the beginning.

While there are some interesting things being discussed on the docker-dev mailing list, I wanted to see if I could get docker to play along with proven logging systems out there with the functionality I have now.

First things first, a couple of requirements:

• I don’t want to run multiple processes in a container (syslog + my_process or /sbin/init)
• I don’t want to have to configure the host to keep track of docker logs

If I run the container in a so called “machine mode”, obviously I can leverage the tools a full blown system provides (such as syslog). I didn’t want to do this because it’s not the docker way and I want a separation of concerns meaning every process/service running in it’s own container.

The reason for #2 is because I don’t want to have to do heavy duty host setup when all that’s needed is docker installed so I can run everything in a container. Think of host machines as throw-away cloud instances that you provision, run a few services on them (isolated in containers) and just throw them away when done.

The first solution that came to mind was to use bind mounts and mount the host’s /dev/log inside the container and just have the container log to that. This way I could aggregate the logs on the host and possibly ship them somewhere to a central location. Although a viable solution, I didn’t quite care for it as it still meant needing to properly configure syslog on the host server. To be fair, I could have used a single syslog container and mount it’s volumes to all the containers that needed to log stuff, but I kind of wanted to try out something different than syslog.

I heard great things about Logstash and Elastisearch so I wanted to try this solution out. Especially since I’ve seen the new Kibana web interface for logstash.

To quote their website: “Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.”

Getting Logstash up and running was fairly trivial and I’ve prepared a Dockerfile so you can get started quickly. I’m using the embedded elasticsearch server but you can run you own server on a different machine and just pass the IP and Port to the logstash container.

Now, the logstash service is kind of heavy on the resources so I didn’t want to run a logstash container on every host machine but rather I wanted co collect logs from every container on every host machine and send it to a central logstash server. That’s where logstash-forwarder comes in.

Logstash-forwarder, previously know as lumberjack is used to collect the logs from each and every container on a host machine and send them to a central logstash server (or multiple servers). I’ve prepared a Dockerfile here.

Logstash-forwarder can be configured to watch certain directories and files but I was more interested in the ability to listen on stdin.

So the idea is to run logstash-forwarder on every host machine and expose a named pipe (FIFO) as a volume that other containers on that host can mount and write to.

So first thing’s first, let’s run the logstash container (see github link above for how to build it):

docker run -name logstash -p 9292:9292 -d -t logstash

This will allow access to the Kibana web interface on localhost:9292.

Now let’s run the logstash-forwarder container (again, see github link above for how to build it):

# replace the IP with the actual IP of the logstash container
docker run -name forwarder -d -v /tmp/feeds -e LOGSTASH_SERVER="172.17.0.69:5043" -t forwarder

Now all we would need to do to run a service that would write to the forwarder is:

docker run -volumes-from forwarder -i -t ubuntu /bin/bash -c "echo 'test' >> /tmp/feeds/fifofeed"

If you go to the Kibana web interface you should see that the message got through. We could just as easily build containers using Dockerfiles where we specify a CMD or ENTRYPOINT directives like so:

# My App
#
# VERSION               0.0.1

FROM      ubuntu
MAINTAINER Me "me@email.com"

CMD /usr/local/bin/run_my_app.sh >> /tmp/feed/fifofeed

The downside of this is that we redirect all output to the FIFO pipe and therefore we short circuit the docker logs command as it will get no output anymore. But this is fine as it’s much better to have logs on a central location and not worry about the logfile filling up the disk space on the host machine.

[UPDATE on April 10th 2014]:

As was pointed out to me in the comments (thanks Alan) a named pipe will blow up just the same way as an anonymous pipe would if there is no reader at the reading end. This means that even if my application handles the appropriate signal and reopens the pipe if there is no one on the reading end it would block.

So I did a little change to the architecture explained above: I mount /dev/log from the host system into the app container, that way the process in the container can be set up to log to syslog, which in this case is going to end up on the host’s syslog. After that I just have another container that runs logstash-forwarder (that also mounts /dev/log from the host) and ships the logs off to the logstash server.

The benefit of this approach is, again, that I don’t have to do much setting up on the host seeing as every distro comes with some kind of syslog daemon already set up (mostly rsyslogd these days).

Classes for example are familiar to most people. But code consisting exclusively of classes and OOP concepts isn’t necessarily simple all of the time. The question arises: Is there something simpler?

Well, the first thing that comes to mind, of course, good old functions. Functions are universal, everybody understands them, even new programmers that don’t yet get all the fancy concepts about OOP, get plain old functions.

Keep in mind, I’m not saying that classes don’t have their place in our code, but rather that maybe we don’t need them quite as often as we might think.

For example: Let’s say we want to write a program that needs to do some sort of calculation on some data we provide. We want to have different types of calculations for getting different information from our data. Let’s call them Calculation Type A and Calculation Type B. We need to take into account data normalization before starting the calculation, and let’s make room for future improvements as we might add a Calculation Type C later on that may require a different kind of normalization.

We immediately think to ourselves: Well I’m just going to make an abstract class and just inherit from that. So here we go:

class AbstractCalc(object):
    def normalize(self, data):
        # do complex normalization
        return data

    def calc(self, data):
        raise NotImplementedError

This is nice. We have an abstract class that has the normalize method implemented (which we can override in derived classes if need be) and we have a calc method that needs to be implemented by the derived classes. So let’s implement those:

class ATypeCalc(AbstractCalc):
    def calc(self, data):
        normalizes_data = self.normalize(data)
        print "Doing some complex A Calculations"
        result = ...

        return result

class BTypeCalc(AbstractCalc):
    def calc(self, data):
        normalized_data = self.normalize(data)
        print "Doing some complex B Calculations"
        result = ...

        return result

Awesome. We have our concrete calculation classes and for now we use the parents normalize method but we can just as easily use our own custom one. The usage of this implementation is something like this:

if calc_type == 'a':
    calc = ATypeCalc()
elif calc_type == 'b':
    calc = BTypeCalc()

# do actual calculation
calc.calc(data)

We can just wrap this in a dispatcher function called get_calc and then we get:

def get_calc(calc_type):
    if calc_type == 'a':
        return ATypeCalc()
    elif calc_type == 'b':
        return BTypeCalc()

calc = get_calc(calc_type)
calc.calc(data)

Pretty awesome, because now to use our code we just call the get_calc function which returns an object instantiated from one of the calculation classes and we’re good to go.

This is all fine and dandy but we have just 2 methods in those classes. Do we really need a class for that?

I wonder if we could get away with using just plain old functions? We just need to keep the code modular enough that adding new calculations is easy and preserve the existing API.

Let’s see how that would look. First we define our normalize function, and our a/b calculation functions:

def normalize(data):
    # do complex normalization
    return data

def a_calc(normalize, data):
    normalized_data = normalize(data)
    print "Doing some complex Calculations A"
    result = ...
    return result

def b_calc(normalize, data):
    normalized_data = normalize(data)
    print "Doing some complex Calculations B"
    result = ...
    return result

Seems straight forward enough. We have a shared normalize function which we pass into the a_calc and b_calc functions, and this way if one day we need to change the normalize function for a_calc we just pass in a different function in there. Yes, we can do this in Python because functions are first class citizens so we can pass functions to other functions and have functions return functions. Pretty neat.

What about using this new implementation. We define 2 more functions, one called calc, and one dispatcher function:

def calc(fn, normalize, data):
    fn(normalize, data)

def get_calc(calc_type):
    from functools import partial

    if calc_type == 'a':
        return partial(calc, a_calc, normalize)
    elif calc_type == 'b':
        return partial(calc, b_calc, normalize)

But wait, what’s this partial funny business? Remember how I said simple != familiar, so stay with me.

With partial application we can use our calc function that takes exactly 3 parameters and bind the first 2 parameters (namely the concrete a/b calc and normalize functions) and leave the 3rd parameter (the data) unbound. This effectively returns a new function that accepts only one parameter. You can, of course, partially apply any number of parameters of a function, and get back a new function that takes that much less parameters.

It’s the same solution just implemented solely with functions. Using this method we left room for adding a 3rd calculation type that can use the general normalize function we have implemented or use it’s custom normalization. In the same effect we can change the normalize function in both A and B type calculations.

And now we can use this API in a similar way we did the one before:

calc = get_calc(calc_type)
calc(data)

There you have it. Two different approaches for solving the same problem, with roughly the same amount of lines of code and the same API.

Notice how nowhere in the post did I mention the term FP (Functional programming). This is mostly because Python is not really a functional language despite it having many of the nice functional-like features. Also, when people hear “Functional Programming” most of the time they immediately run away.

Personally I find the functional solution cleaner, more readable, and easier to explain to new programmers. Alas, I leave it to you to decide what the best approach is for your concrete problem. Remember, there’s no silver bullet, always use the best tool at your disposal for the problem at hand.

So, The main criteria was to be able to write posts using Sublime. I spend most of my time in the day using this editor so it’s only logical.

After doing some research I figured that a static site generator would be the best choice. Enter Nikola. The name itself made it a winner and it has really nice features, but most importantly it was really easy to set up.

Customizing the default template (Nikola supports Mako and Jinja2) was the most time consuming but even that was a breeze.

Now I can write new posts using Sublime and have them deployed just by pushing to a git repository. The new posts are rebuilt automatically and Nginx serves the static content.

I’m pretty happy with the end result.

The major component here is using the same database locally, for development, as you use in production. I myself work mostly with Django and, as most of you Django devs know, it’s very easy to get started using a Sqlite database. Now, I’m not bashing on Sqlite, it definitely has it’s purpose but it’s not on the main dev environment. It’s nice to be able to set up the designer’s development environment without too much of a hassle but you should be developing on a database you use in production.

I use Postgres on all of my production servers. I come from a strong Sysadmin background and I set up most of this on my laptop as well. But it’s one thing to setup everything on my laptop and a whole different story setting up Postgres on each workstation of every colleague. It’s exhausting.

Of course you can have a staging server and all that but I really want that everyone has a local Postgres instance even then (you just catch way to many bugs this way even before the code hits the staging server).

Enter docker. Docker is an open-source engine which automates the deployment of applications as highly portable, self-sufficient containers which are independent of hardware, language, framework, packaging system and hosting provider. [3]

Under the hood Docker uses LXC for running isolated containers. Now, I’ve used LXC by itself, and have some production environments that use it, but Docker is a huge deal. Not only is it a higher level wrapper on top of LXC but it really takes so much hassle out of the picture and just enables you to run virtual containers, of your app, or of other processes. Docker is going to be a huge deal in a lot of production environments and the way we do deployments in the coming months, but for now, in this post at least, we are just going to use it for setting up our development environment quickly and easily.

Docker requires a 64 bit Linux distro (kernel 3.8 or higher). Ubuntu has a PPA (link) and it’s fairly easy to install it. For Debian there is a package in the works but you can just download a pre-compiled binary and put it somewhere on your PATH. This goes for other distros as well. For Mac and Windows users, you can run docker inside a Vagrant box (link).

Linux Containers deployed with Docker have some advantages over the full virtual machines like VritualBox. A Linux Container looks just like a real virtual machine form the inside. It has it’s own filesystem, it’s own network interface and so on. In reality it’s just a group of processes totally isolated form the host operating system but running on the same kernel as the host. This in turn means that it’s much easier to emulate I/O devices and hardware, that it has far less CPU overhead and memory consumption and still maintains the portability of a full virtual machine. Last but not least, the most important feature is that it’s lightning fast to boot up. Containers sometime take milliseconds to bring up.

Now, before we continue any further we need to explain the concept of docker images. Docker images are basically snapshots of a system that are used for firing up new containers. So let’s say I have a ‘postgres’ image you could just do:

docker run -i -t postgres /bin/bash

Which would bring up a new container and attach you to it’s bash prompt from which you could run your Postgres process.

Docker images are hosted on a public Docker Index from where you can browse and download all kinds of prepared images, ranging from base images from the docker team (ubuntu, base) to specialized user images like denibertovic/postgres. There’s also an option to run your own local index but that’s another blog post :).

I’ve prepared a Postgres container denibertovic/postgres (versions 9.1 and 9.2) which are uploaded to the docker index and can be used freely by anyone.

Now, I’ve promised you a one-liner solution and I wish to deliver on that. Go ahead and download the Makefile I’ve prepared here. Put it into your project folder or integrate it into your existing Makefile if you already have one.

All it takes now is to just type:

make postgres

This will bring up a new container and run a Postgres 9.2 instance in it. For 9.1 just change the POSTGRES_VERSION variable in the Makefile. If the above command is run for the first time it will first download the “denibertovic/postgres” image from the docker index and then run the container. All subsequent runs will use the already downloaded image.

You can list the images you have locally using the below command:

docker images

Once you have your Postgres instance up and running you can connect to it using:

psql -Upostgres -h localhost

Use this to create your database and user

postgres=# CREATE DATABASE my_database;
postgres=# CREATE USER my_user WITH PASSWORD 'myuserpassword';
postgres=# GRANT ALL ON DATABASE my_database to my_user;

You have now successfully created your database and the user for that database.

Now you can set up you project settings files to point to the given database. For Django projects I like to use dj_database_url and then you end with something looking like this:

DATABASES = {
    'default': dj_database_url.config(
        default='postgres://my_user:myuserpassword@localhost:5432/my_database')
}

Docker containers are ephemeral which means all of the changes done to the container are gone once you stop the container. With our Postgres container this means that once we stop the container our changes to the database, and even the database itself, would disappear. For some cases this might be fine, but most of the time we want out development database to have persistent data.

To accomplish this what we do is tell the containerized Postgres instance to write all the database stuff in a folder called “__data” located in the same folder as the Makefile on the host system. So if at one point you wish to start from scratch and delete all the database data you can do so simply by deleting the whole “__data” folder and everything in it and repeat the procedure above.

Another thing to note is that the containerized Postgres instance is set up to run on the default “5432” port which is in turn NAT-ed to localhost on the same port. This means that if you run one Postgres instance for one project you first need to stop it before you can run another instance on another project (don’t worry the “make postgres” command will warn you about this).

So first grab the container ID from the running container with:

docker ps

ID                  IMAGE                       COMMAND                PORTS
000f0a07e49a        denibertovic/postgres:9.2   /usr/local/bin/start   5432->5432

And then use the ID to stop the container:

docker stop 000f0a07e49a

And that’s it folks. Hope you find this useful and feel free to leave comments/tips/improvements.

Next up will be a post about App deployments.

How to do it?! Here’s an example:

import base64, urllib, urllib2

with open('image.jpeg') as f:
    imagedata = base64.b64encode(f.read())
    data = urllib.urlencode({'image': imagedata, 'caption': 'test'})
    u = urllib2.urlopen('http://pixbin.us/api/post/', data)
    print u.readlines()

The example above is in Python but the same principles apply for your language of choice. Just base64 encode an image and send a POST request with the shown parameters to the given link. The response you get back will be a json like so:

{"link": "http://pixbin.us/abcabc"}

This can be very useful if you want to, let’s say, quickly share a screenshot with somebody. We have a very handy little bash script that let’s you do just that. You can check it out here. Special thanks to Senko for making the script.

Check back next week for more, hopefully useful, stuff. :)

A while ago I stumbled upon a github project that was aiming to solve exactly this kind of issue. The thing is that sometimes people just don’t have enough time on their hands, and with a blog site the content is the thing that matters, not what platform you are using. And there are many of them out there.

I decided to use Syte because it was built specifically with this in mind and was built for web developers/designers. Another thing that was a plus is that it was built using Django :) . Thanks Rodrigo.

Now I have no more excuses so I have to start writing. Mainly I’ll write about all the geeky things that caught my eye and some posts are gonna serve merely as a reference about some technical stuff so I don’t forget later on how I solved a problem. And hey if it helps someone else in the meantime even better.

Over and out. For now.