Table of Contents
By Khimananda Oli | Last reviewed: August 2026
Teams hitting the ceiling of managed CI runners often turn to Buildkite: Scalable CI with Your Own Agents to regain control over performance, security, and cost. Unlike fully SaaS platforms where you rent ephemeral compute, Buildkite separates the orchestration plane from the execution plane, letting you run build agents on your own infrastructure while keeping pipeline logic centralized. This hybrid model is particularly valuable for organizations in Nepal and globally that require air-gapped environments, specific hardware access, or strict data residency compliance without sacrificing modern developer experience.
How does Buildkite: Scalable CI with Your Own Agents differ from pure SaaS?
The fundamental difference lies in the execution boundary. In platforms like GitHub Actions or CircleCI, the default runners are multi-tenant VMs managed by the vendor. You can bring your own runners, but they are often treated as secondary options. With Buildkite, self-hosted agents are the primary abstraction. The Buildkite API serves only as a scheduler and state store; it never touches your source code, artifacts, or secrets during job execution unless you explicitly upload them.
This distinction matters for three practical reasons I encounter frequently when consulting on self-hosted CI runner security:
- Network topology: Agents initiate outbound HTTPS connections to the Buildkite API. No inbound ports need opening, making it viable behind restrictive corporate firewalls or in Nepal-based data centers with limited ingress rules.
- Hardware affinity: You can tag agents with capabilities like
gpu=true,arch=arm64, orfpga=yes. Pipelines target these tags, ensuring ML training jobs land on expensive GPUs while linting runs on cheap spot instances. - Data gravity: For teams with terabyte-scale datasets or proprietary models, moving data to a SaaS runner is cost-prohibitive and slow. Self-hosted agents execute adjacent to your storage, reducing build times from hours to minutes.
A common mistake is treating Buildkite agents as static pets. In production, you should treat them as cattle—ephemeral, auto-scaled, and replaced after every job or on a short cadence to prevent configuration drift and maintain reproducibility.
How do you configure elastic scaling for Buildkite agents on AWS?
Static agent fleets waste money during quiet periods and throttle builds during peaks. The AWS Auto Scaling integration with Buildkite solves this through the Elastic CI Stack, a CloudFormation template maintained by Buildkite that provisions an Auto Scaling Group (ASG) driven by queue depth metrics rather than CPU utilization.
Deploy the Elastic CI Stack
The most reliable method uses the official CloudFormation template. Avoid manual ASG setup unless you have specific networking constraints.
# Clone the elastic-ci-stack-for-aws repository
git clone https://github.com/buildkite/elastic-ci-stack-for-aws.git
cd elastic-ci-stack-for-aws
# Deploy with required parameters
aws cloudformation create-stack \
--stack-name buildkite-agents-prod \
--template-url https://s3.amazonaws.com/buildkite-aws-stack/latest/aws-stack.yml \
--parameters \
ParameterKey=BuildkiteAgentToken,ParameterValue=YOUR_AGENT_TOKEN \
ParameterKey=BuildkiteQueueTag,ParameterValue=default \
ParameterKey=InstanceType,ParameterValue=m7g.large \
ParameterKey=MaxSize,ParameterValue=20 \
ParameterKey=MinSize,ParameterValue=0 \
ParameterKey=VpcId,ParameterValue=vpc-0abc123def456 \
ParameterKey=Subnets,ParameterValue=subnet-aaa,subnet-bbb \
--capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAMTune scaling responsiveness
The default scaling policy reacts conservatively. For fast-moving teams, adjust the CloudWatch alarm thresholds. The key metric is BuildkiteJobQueueDepth. Set the scale-up threshold to trigger when pending jobs exceed available agents for more than 60 seconds, not 300. Scale-down should use a longer cooldown (typically 10–15 minutes) to avoid thrashing during commit bursts.
For Kubernetes-native teams, the Amazon EKS practical guide covers similar patterns using the Buildkite Agent Helm chart with KEDA scalers, which often provides faster scale-up times than EC2 ASGs due to pod scheduling speed versus instance boot time.
How do you secure self-hosted Buildkite agents for compliance?
Running your own agents means owning the security boundary. For SOC 2 or ISO 27001 audits, you must demonstrate that build environments cannot leak secrets, persist malicious state, or access unauthorized resources. I apply these controls consistently across client engagements requiring automated SOC 2 compliance evidence.
Isolate workloads with disposable environments
Never run untrusted code directly on the host OS. Use one of these isolation strategies:
- Docker plugin (recommended): Each step runs in a fresh container. Mount only necessary volumes. Use read-only root filesystems where possible.
- Kubernetes executor: Each job gets a dedicated pod with network policies restricting egress. Ideal for multi-tenant clusters.
- VM reset: For bare-metal or legacy workloads, snapshot the clean state and revert after each job. Slower but provides strongest isolation.
Manage secrets without environment variables
Injecting secrets as environment variables exposes them in process listings and logs. Instead, integrate with HashiCorp Vault or AWS Secrets Manager using the Buildkite secrets plugin. Fetch credentials at runtime inside the isolated container, write them to memory-backed tmpfs, and clean up immediately after use. This pattern satisfies auditors who flag static env vars as findings.
Restrict agent capabilities with tags
Use agent tags to enforce least privilege. Tag agents with env=production or compliance=pci and configure pipeline steps to target only matching agents. Never allow a development-tagged agent to pick up production deployment jobs. This logical separation is critical when sharing infrastructure across teams.
When should you choose Buildkite over GitHub Actions or GitLab CI?
Choosing between CI platforms is rarely about features alone—it's about operational fit. This comparison reflects real-world trade-offs I've navigated when migrating teams from managed to hybrid systems.
| Criteria | Buildkite (Self-Hosted) | GitHub Actions | GitLab CI |
|---|---|---|---|
| Execution Model | Hybrid: Managed orchestration + your infra | SaaS default; self-hosted optional | SaaS or self-managed runners |
| Scaling Speed | Seconds (K8s) to minutes (EC2 ASG) | Instant (SaaS); variable (self-hosted) | Variable; depends on runner config |
| VPC / Private Access | Native; agents live inside your network | Requires tunneling or enterprise plan | Possible with self-managed runners |
| Pipeline Config | YAML + Bash hooks; highly extensible | YAML workflows; reusable actions | YAML .gitlab-ci.yml; includes/extends |
| Cost Model | Per-user fee + your infra costs | Per-minute (SaaS) or free (self-hosted) | Per-user or per-minute; self-hosted free |
| Compliance Fit | Excellent; full infra control + audit logs | Limited on SaaS; better with GHES | Good with self-managed; complex licensing |
| Best For | Complex, secure, hybrid, or specialized hardware | Open source, standard web apps, GitHub-centric | All-in-one DevOps platform preference |
Choose Buildkite when your bottleneck is infrastructure control, not feature parity. If you need GPU clusters, on-premise database access, or deterministic build environments for regulated industries, Buildkite’s agent-first design wins. For standard web app CI where convenience trumps customization, GitHub Actions remains hard to beat. GitLab CI fits teams already invested in the GitLab ecosystem who want integrated MR workflows without stitching tools together.
What are common pitfalls when operating Buildkite agents at scale?
After deploying Buildkite across multiple production environments, these issues surface repeatedly:
- Ignoring agent draining: When an ASG terminates an instance mid-job, builds fail. Always enable the
--disconnect-after-jobflag or use the lifecycle hook script provided by the Elastic CI Stack to gracefully drain agents before shutdown. - Over-provisioning instance types: Using m7g.xlarge for everything wastes money. Profile your builds. Most lint/test jobs fit on m7g.medium. Reserve larger instances for compilation-heavy or integration test steps. Use agent tags to route appropriately.
- Neglecting log retention: Buildkite stores logs in its managed service, but for compliance you may need local copies. Configure the S3 log upload plugin early. Retroactively adding it creates gaps in audit trails.
- Hardcoding agent tokens: Rotate tokens regularly. Store them in Secrets Manager and inject via user-data scripts, not AMI bakes. Baked tokens become stale and create rotation headaches.
Monitoring is non-negotiable. Expose agent metrics via the Prometheus exporter and build dashboards tracking queue wait time, agent utilization, and job duration. Without observability, you're guessing at capacity needs. Refer to Prometheus metrics monitoring fundamentals if you're setting up this telemetry stack for the first time.
Getting started with Buildkite: Scalable CI with Your Own Agents
Start small: deploy a single Elastic CI Stack in a non-production AWS account, migrate one low-risk pipeline, and validate the scaling behavior under load before expanding. Document your agent tagging strategy and secret management approach upfront—retrofitting these later causes migration pain. If you're evaluating whether Buildkite fits your team's compliance requirements or infrastructure constraints, or need help designing a secure agent architecture that passes audit, reach out to discuss your specific CI/CD challenges.
