Security Hardening for Self-Hosted Runners
Comprehensive security best practices for enterprise runner deployments
This guide provides enterprise-grade security hardening practices for self-hosted runners across all major CI/CD platforms. Whether you're deploying GitHub Actions runners, GitLab Runners, or Jenkins agents, these security measures will help you build a robust and compliant infrastructure.
Security Overview#
Threat Model#
Self-hosted runners face several attack vectors that require comprehensive security measures:
External threats:
- Remote code execution through compromised workflows
- Credential theft from insecure storage
- Network-based attacks on runner infrastructure
- Supply chain attacks through dependencies
Internal threats:
- Privilege escalation by malicious workflows
- Data exfiltration through runner access
- Cross-tenant contamination in shared environments
- Insider threats with excessive permissions
Infrastructure risks:
- Unpatched operating systems and dependencies
- Misconfigured network access controls
- Insecure container configurations
- Weak authentication mechanisms
Attack Vectors#
Understanding common attack patterns helps prioritize security controls:
- Workflow injection attacks - Malicious code execution through PR workflows
- Token theft - Stealing runner registration or API tokens
- Privilege escalation - Exploiting runner permissions to access broader infrastructure
- Data persistence attacks - Leaving malicious code or data between job runs
- Network pivoting - Using compromised runners to attack internal systems
Network Security#
Network Isolation#
Implement defense-in-depth network security:
1# Create isolated network for runners2sudo ip netns add runner-isolation3sudo ip link add veth0 type veth peer name veth14sudo ip link set veth1 netns runner-isolation56# Configure firewall rules for runner traffic7sudo iptables -N RUNNER_CHAIN8sudo iptables -A INPUT -j RUNNER_CHAIN9sudo iptables -A RUNNER_CHAIN -p tcp --dport 443 -j ACCEPT # HTTPS only10sudo iptables -A RUNNER_CHAIN -p tcp --dport 22 -s 10.0.0.0/8 -j ACCEPT # SSH from internal11sudo iptables -A RUNNER_CHAIN -j DROP # Drop everything elseVPN Configuration#
Use WireGuard for secure runner-to-cloud connectivity:
1# Generate WireGuard configuration2wg genkey | tee /etc/wireguard/runner-private.key | wg pubkey > /etc/wireguard/runner-public.key34# Create WireGuard configuration5cat > /etc/wireguard/wg0.conf << EOF6[Interface]7PrivateKey = $(cat /etc/wireguard/runner-private.key)8Address = 10.200.200.2/249DNS = 1.1.1.11011[Peer]12PublicKey = YOUR_SERVER_PUBLIC_KEY13Endpoint = vpn.assistance.bg:5182014AllowedIPs = 10.200.200.0/24, 192.168.1.0/2415PersistentKeepalive = 2516EOF1718# Start WireGuard19sudo systemctl enable wg-quick@wg020sudo systemctl start wg-quick@wg0Firewall Configuration#
Platform-specific firewall rules:
Linux (iptables):
1#!/bin/bash2# Runner firewall configuration34# Flush existing rules5iptables -F6iptables -X7iptables -t nat -F8iptables -t nat -X910# Default policies11iptables -P INPUT DROP12iptables -P FORWARD DROP13iptables -P OUTPUT ACCEPT1415# Allow loopback16iptables -A INPUT -i lo -j ACCEPT1718# Allow established connections19iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT2021# SSH access (restricted)22iptables -A INPUT -p tcp --dport 22 -s 192.168.1.0/24 -j ACCEPT2324# HTTPS for API calls25iptables -A OUTPUT -p tcp --dport 443 -j ACCEPT2627# DNS28iptables -A OUTPUT -p udp --dport 53 -j ACCEPT2930# Save rules31iptables-save > /etc/iptables/rules.v4macOS (pfctl):
1# Create pf.conf for macOS runners2cat > /etc/pf.conf << EOF3# Default deny4block all56# Allow loopback7pass on lo089# Allow established connections10pass in proto tcp from any to any port ssh flags S/SA keep state11pass out proto tcp from any to any port https keep state12pass out proto udp from any to any port domain keep state1314# Block direct internet access except HTTPS15block out inet proto tcp from any to any port != https16EOF1718# Load configuration19sudo pfctl -f /etc/pf.conf20sudo pfctl -eZero-Trust Architecture#
Implement zero-trust networking for distributed runners:
1# Istio service mesh configuration for Kubernetes runners2apiVersion: security.istio.io/v1beta13kind: AuthorizationPolicy4metadata:5 name: runner-access-control6spec:7 selector:8 matchLabels:9 app: self-hosted-runner10 rules:11 - from:12 - source:13 principals: ["cluster.local/ns/runners/sa/runner-service-account"]14 - to:15 - operation:16 methods: ["GET", "POST"]17 paths: ["/api/v2/*"]18 - when:19 - key: source.ip20 values: ["10.200.200.0/24"]Platform-Specific Hardening#
GitHub Actions Security#
Runner Group Isolation:
1# Configure runner with restricted group access2./config.sh --url https://github.com/your-org --token $RUNNER_TOKEN \3 --runnergroup "secure-runners" --labels "hardened,production" \4 --replace --disableupdateToken Management:
1#!/bin/bash2# GitHub runner token rotation script34GITHUB_ORG="your-organization"5GITHUB_TOKEN="$GITHUB_ADMIN_TOKEN"67# Generate new runner token8NEW_TOKEN=$(curl -X POST \9 -H "Authorization: token $GITHUB_TOKEN" \10 -H "Accept: application/vnd.github.v3+json" \11 https://api.github.com/orgs/$GITHUB_ORG/actions/runners/registration-token | \12 jq -r .token)1314# Update runner configuration15./config.sh remove --token $OLD_TOKEN16./config.sh --url https://github.com/$GITHUB_ORG --token $NEW_TOKEN1718echo "Runner token rotated successfully"Workflow Security:
1# .github/workflows/security-policy.yml2name: Security Policy Enforcement3on:4 workflow_run:5 workflows: ["*"]6 types: [requested]78jobs:9 security-check:10 runs-on: [self-hosted, hardened]11 steps:12 - name: Validate workflow permissions13 run: |14 # Check for dangerous permissions15 if echo "${{ toJson(github.event.workflow_run.head_commit.message) }}" | grep -E "(sudo|rm -rf|chmod 777)"; then16 echo "::error::Dangerous commands detected in workflow"17 exit 118 fiGitLab Runner Security#
Secure Runner Registration:
1# Register GitLab runner with security constraints2gitlab-runner register \3 --url "https://gitlab.assistance.bg/" \4 --registration-token "$GITLAB_REGISTRATION_TOKEN" \5 --executor "docker" \6 --docker-image "alpine:latest" \7 --docker-privileged=false \8 --docker-disable-cache=true \9 --docker-volumes="/var/run/docker.sock:/var/run/docker.sock:ro" \10 --docker-security-opt="no-new-privileges:true" \11 --docker-security-opt="apparmor=docker-default" \12 --tag-list "secure,hardened" \13 --run-untagged=false \14 --locked=trueExecutor Security:
1# /etc/gitlab-runner/config.toml2concurrent = 23check_interval = 045[session_server]6 session_timeout = 180078[[runners]]9 name = "secure-docker-runner"10 url = "https://gitlab.assistance.bg/"11 token = "SECURE_TOKEN"12 executor = "docker"13 [runners.docker]14 tls_verify = false15 image = "alpine:latest"16 privileged = false17 disable_cache = true18 volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock:ro"]19 security_opt = ["no-new-privileges:true", "apparmor=docker-default"]20 cap_drop = ["ALL"]21 cap_add = ["CHOWN", "SETUID", "SETGID"]22 devices = []23 network_mode = "bridge"24 dns = ["1.1.1.1", "8.8.8.8"]25 pull_policy = "always"Jenkins Security#
Agent Configuration:
1// Jenkins agent security configuration2import jenkins.model.*3import hudson.model.*4import hudson.slaves.*5import hudson.plugins.sshslaves.*67def jenkins = Jenkins.getInstance()89// Create secure node10def launcher = new SSHLauncher(11 "runner.assistance.bg", // host12 22, // port13 "jenkins-ssh-key", // credentials ID14 "", // JVM options15 "", // Java path16 "", // prefix start slave command17 "", // suffix start slave command18 60, // launch timeout19 3, // max retries20 15 // retry wait time21)2223def node = new DumbSlave(24 "secure-agent", // node name25 "Hardened Jenkins agent", // description26 "/home/jenkins", // remote FS root27 "2", // executors28 Node.Mode.EXCLUSIVE, // usage mode29 "hardened production", // labels30 launcher,31 new RetentionStrategy.Always(),32 new LinkedList<NodeProperty<?>>()33)3435jenkins.addNode(node)36jenkins.save()Script Security:
1// Jenkinsfile with security constraints2pipeline {3 agent { label 'hardened' }45 options {6 skipDefaultCheckout()7 timeout(time: 30, unit: 'MINUTES')8 timestamps()9 }1011 environment {12 // Restrict environment variables13 PATH = "/usr/local/bin:/usr/bin:/bin"14 JAVA_OPTS = "-Xmx1g -Djava.awt.headless=true"15 }1617 stages {18 stage('Security Check') {19 steps {20 script {21 // Validate build parameters22 if (params.containsKey('DANGEROUS_PARAM')) {23 error("Dangerous parameter detected")24 }25 }26 }27 }28 }29}Bazel Remote Execution#
gRPC Security Configuration:
1# Start Bazel Remote Execution with TLS2bazel run //server:remote_execution_server -- \3 --port=8980 \4 --tls_certificate=/etc/ssl/certs/server.crt \5 --tls_private_key=/etc/ssl/private/server.key \6 --tls_ca_certificate=/etc/ssl/certs/ca.crt \7 --require_client_certificates=true \8 --auth_tokens_file=/etc/bazel/auth_tokens.jsonWorker Authentication:
1{2 "tokens": {3 "worker-pool-1": {4 "token": "secure-worker-token-1",5 "permissions": ["execute", "read_cache"],6 "allowed_actions": [".*\\.compile", ".*\\.test"],7 "resource_limits": {8 "cpu": "4",9 "memory": "8GB",10 "disk": "100GB"11 }12 }13 }14}DevOps Hub API Security#
Authentication Configuration#
API Key Management:
1# DevOps Hub API key rotation2CURRENT_KEY="$DEVOPS_API_KEY"3PROJECT_ID="your-project-id"45# Generate new API key6NEW_KEY=$(curl -X POST "https://console.assistance.bg/api/v2/auth/rotate-key" \7 -H "Authorization: Bearer $CURRENT_KEY" \8 -H "Content-Type: application/json" \9 --data '{"project_id": "'$PROJECT_ID'"}' | \10 jq -r '.key')1112# Update runners with new key13echo "DEVOPS_API_KEY=$NEW_KEY" > /etc/runner/environment14sudo systemctl restart runner-service1516echo "API key rotated successfully"Rate Limiting Monitoring:
1#!/bin/bash2# Monitor DevOps Hub API usage34API_KEY="$DEVOPS_API_KEY"56# Check current usage7USAGE=$(curl -s -X GET "https://console.assistance.bg/api/v2/auth/usage" \8 -H "Authorization: Bearer $API_KEY" | \9 jq '{requests_used: .requests_used, requests_limit: .requests_limit, reset_time: .reset_time}')1011echo "Current API usage: $USAGE"1213# Alert if approaching limit14USED=$(echo $USAGE | jq '.requests_used')15LIMIT=$(echo $USAGE | jq '.requests_limit')16THRESHOLD=$(($LIMIT * 80 / 100)) # 80% threshold1718if [ "$USED" -gt "$THRESHOLD" ]; then19 echo "WARNING: API usage at 80% of limit"20 # Send alert to monitoring system21 curl -X POST "$SLACK_WEBHOOK" \22 -d "{\"text\": \"DevOps Hub API usage high: $USED/$LIMIT requests\"}"23fiAPI Security Headers#
Request Security:
1# Secure API request function2make_secure_api_call() {3 local endpoint="$1"4 local method="$2"5 local data="$3"67 curl -X "$method" \8 -H "Authorization: Bearer $DEVOPS_API_KEY" \9 -H "Content-Type: application/json" \10 -H "User-Agent: SecureRunner/1.0" \11 -H "X-Request-ID: $(uuidgen)" \12 --max-time 30 \13 --retry 3 \14 --retry-delay 1 \15 --fail \16 --silent \17 --show-error \18 "https://console.assistance.bg/api/v2/$endpoint" \19 --data "$data"20}2122# Usage example23make_secure_api_call "projects" "GET"Connection Security#
TLS Configuration:
1# Validate DevOps Hub API TLS certificate2check_api_security() {3 local api_host="console.assistance.bg"45 # Check TLS version6 tls_version=$(openssl s_client -connect $api_host:443 -servername $api_host 2>/dev/null | \7 grep -E "Protocol|Cipher" | head -2)89 echo "TLS Security Check for $api_host:"10 echo "$tls_version"1112 # Verify certificate13 cert_check=$(openssl s_client -connect $api_host:443 -servername $api_host -verify_return_error 2>&1)1415 if echo "$cert_check" | grep -q "Verify return code: 0"; then16 echo "✓ Certificate validation passed"17 else18 echo "✗ Certificate validation failed"19 exit 120 fi21}2223check_api_securitySecret Management#
HashiCorp Vault Integration#
Vault Configuration for Runners:
1# Install and configure Vault agent2vault auth -method=aws role=runner-role34# Configure Vault agent for secret retrieval5cat > /etc/vault/agent.hcl << EOF6pid_file = "/var/run/vault-agent.pid"78vault {9 address = "https://vault.assistance.bg:8200"10 retry {11 num_retries = 512 }13}1415auto_auth {16 method "aws" {17 mount_path = "auth/aws"18 config = {19 type = "iam"20 role = "runner-role"21 }22 }2324 sink "file" {25 config = {26 path = "/etc/vault/token"27 mode = 060028 }29 }30}3132template {33 source = "/etc/vault/templates/runner-secrets.tpl"34 destination = "/etc/runner/secrets"35 perms = 060036 command = "systemctl reload runner-service"37}38EOF3940# Start Vault agent41systemctl enable vault-agent42systemctl start vault-agentDynamic Secret Template:
1# /etc/vault/templates/runner-secrets.tpl2{{- with secret "kv/data/runner-secrets" }}3export DEVOPS_API_KEY="{{ .Data.data.api_key }}"4export DATABASE_PASSWORD="{{ .Data.data.db_password }}"5export SIGNING_KEY="{{ .Data.data.signing_key }}"6{{- end }}78{{- with secret "database/creds/runner-role" }}9export DB_USERNAME="{{ .Data.username }}"10export DB_PASSWORD="{{ .Data.password }}"11{{- end }}AWS Secrets Manager#
Runner Secret Retrieval:
1#!/usr/bin/env python32# Runner secret management with AWS Secrets Manager34import boto35import json6import os7from botocore.exceptions import ClientError89class RunnerSecretsManager:10 def __init__(self, region_name='us-east-1'):11 self.secrets_client = boto3.client('secretsmanager', region_name=region_name)1213 def get_secret(self, secret_name):14 """Retrieve secret from AWS Secrets Manager"""15 try:16 response = self.secrets_client.get_secret_value(SecretId=secret_name)17 return json.loads(response['SecretString'])18 except ClientError as e:19 print(f"Failed to retrieve secret {secret_name}: {e}")20 return None2122 def update_runner_env(self, secret_name, env_file_path):23 """Update runner environment with secrets"""24 secrets = self.get_secret(secret_name)25 if not secrets:26 return False2728 env_lines = []29 for key, value in secrets.items():30 env_lines.append(f"export {key}='{value}'")3132 with open(env_file_path, 'w') as f:33 f.write('\n'.join(env_lines))3435 os.chmod(env_file_path, 0o600)36 return True3738# Usage39secrets_manager = RunnerSecretsManager()40secrets_manager.update_runner_env('runner-secrets', '/etc/runner/environment')IAM Role for Runners:
1{2 "Version": "2012-10-17",3 "Statement": [4 {5 "Effect": "Allow",6 "Action": [7 "secretsmanager:GetSecretValue"8 ],9 "Resource": [10 "arn:aws:secretsmanager:us-east-1:account:secret:runner-secrets/*"11 ],12 "Condition": {13 "StringEquals": {14 "secretsmanager:VersionStage": "AWSCURRENT"15 },16 "DateLessThan": {17 "aws:CurrentTime": "2025-12-31T23:59:59Z"18 }19 }20 },21 {22 "Effect": "Allow",23 "Action": [24 "kms:Decrypt"25 ],26 "Resource": [27 "arn:aws:kms:us-east-1:account:key/key-id"28 ]29 }30 ]31}Azure Key Vault#
Service Principal Authentication:
1# Azure Key Vault integration for runners2az login --service-principal \3 --username $AZURE_CLIENT_ID \4 --password $AZURE_CLIENT_SECRET \5 --tenant $AZURE_TENANT_ID67# Retrieve secrets8get_keyvault_secret() {9 local vault_name="$1"10 local secret_name="$2"1112 az keyvault secret show \13 --vault-name "$vault_name" \14 --name "$secret_name" \15 --query "value" \16 --output tsv17}1819# Update runner configuration20DEVOPS_API_KEY=$(get_keyvault_secret "runner-vault" "devops-api-key")21echo "export DEVOPS_API_KEY='$DEVOPS_API_KEY'" > /etc/runner/secrets22chmod 600 /etc/runner/secretsGoogle Secret Manager#
Service Account Configuration:
1# Authenticate with service account2export GOOGLE_APPLICATION_CREDENTIALS="/etc/runner/service-account.json"34# Install Google Cloud SDK5curl https://sdk.cloud.google.com | bash6source ~/.bashrc78# Retrieve secrets9get_gcp_secret() {10 local project_id="$1"11 local secret_name="$2"12 local version="${3:-latest}"1314 gcloud secrets versions access "$version" \15 --secret="$secret_name" \16 --project="$project_id"17}1819# Update runner environment20DEVOPS_API_KEY=$(get_gcp_secret "your-project" "devops-api-key")21echo "DEVOPS_API_KEY=$DEVOPS_API_KEY" >> /etc/runner/environmentContainer Security#
Docker Security Hardening#
Secure Docker Configuration:
1# /etc/docker/daemon.json2{3 "log-driver": "json-file",4 "log-opts": {5 "max-size": "100m",6 "max-file": "3"7 },8 "userns-remap": "dockeruser",9 "no-new-privileges": true,10 "seccomp-profile": "/etc/docker/seccomp.json",11 "selinux-enabled": true,12 "storage-driver": "overlay2",13 "storage-opts": [14 "overlay2.override_kernel_check=true"15 ],16 "default-ulimits": {17 "nproc": {18 "Hard": 1024,19 "Name": "nproc",20 "Soft": 102421 },22 "nofile": {23 "Hard": 65536,24 "Name": "nofile",25 "Soft": 6553626 }27 },28 "live-restore": true,29 "userland-proxy": false,30 "experimental": false31}Rootless Docker Setup:
1# Install rootless Docker2curl -fsSL https://get.docker.com/rootless | sh34# Configure rootless environment5echo 'export PATH=/home/$USER/bin:$PATH' >> ~/.bashrc6echo 'export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock' >> ~/.bashrc78# Start rootless Docker service9systemctl --user enable docker10systemctl --user start docker1112# Verify rootless installation13docker context use rootless14docker run --rm hello-worldContainer Runtime Security:
1#!/bin/bash2# Secure container execution for runners34run_secure_container() {5 local image="$1"6 local command="$2"78 docker run --rm \9 --security-opt=no-new-privileges:true \10 --security-opt=apparmor:docker-default \11 --cap-drop=ALL \12 --cap-add=CHOWN \13 --cap-add=SETUID \14 --cap-add=SETGID \15 --read-only \16 --tmpfs /tmp \17 --tmpfs /var/tmp \18 --network=bridge \19 --memory=1g \20 --cpus=1.0 \21 --ulimit nproc=1024:1024 \22 --ulimit nofile=1024:1024 \23 --user 1000:1000 \24 "$image" \25 "$command"26}2728# Usage29run_secure_container "alpine:latest" "sh -c 'echo Hello World'"Image Security Scanning#
Trivy Integration:
1#!/bin/bash2# Container image vulnerability scanning34scan_runner_images() {5 local images=("$@")67 for image in "${images[@]}"; do8 echo "Scanning $image..."910 # Scan for vulnerabilities11 trivy image --format json --output "${image//\//_}-scan.json" "$image"1213 # Check for critical vulnerabilities14 critical_count=$(jq '.Results[].Vulnerabilities | map(select(.Severity == "CRITICAL")) | length' "${image//\//_}-scan.json" 2>/dev/null || echo "0")1516 if [ "$critical_count" -gt 0 ]; then17 echo "❌ CRITICAL: $image has $critical_count critical vulnerabilities"18 exit 119 else20 echo "✅ PASSED: $image has no critical vulnerabilities"21 fi22 done23}2425# Scan common runner images26scan_runner_images \27 "ubuntu:22.04" \28 "node:18-alpine" \29 "python:3.11-slim"Cosign Image Signing:
1# Generate signing keys2cosign generate-key-pair34# Sign container image5cosign sign --key cosign.key runner-image:latest67# Verify signed image8cosign verify --key cosign.pub runner-image:latest910# Policy enforcement in admission controller11cat > image-policy.yaml << EOF12apiVersion: kyverno.io/v113kind: ClusterPolicy14metadata:15 name: require-signed-images16spec:17 validationFailureAction: enforce18 background: false19 rules:20 - name: check-signature21 match:22 any:23 - resources:24 kinds:25 - Pod26 validate:27 message: "Images must be signed with cosign"28 pattern:29 spec:30 containers:31 - name: "*"32 image: "*/runner-*:*"33EOFRuntime Security#
Falco Runtime Monitoring:
1# /etc/falco/falco_rules.local.yaml2- rule: Suspicious Runner Activity3 desc: Detect suspicious activities in runner containers4 condition: >5 container and6 (proc.name in (wget, curl, nc, ncat, netcat) and7 proc.args contains "reverse" or8 proc.args contains "shell") or9 (spawned_process and10 proc.name in (bash, sh, zsh) and11 proc.args contains "-c" and12 proc.args contains "eval")13 output: >14 Suspicious activity in runner container15 (user=%user.name command=%proc.cmdline container=%container.name16 image=%container.image.repository)17 priority: WARNING18 tags: [runner, security, malware]1920- rule: Runner Container Escape Attempt21 desc: Detect container escape attempts22 condition: >23 container and24 (proc.name in (runc, docker, kubectl) or25 fd.name startswith /var/run/docker.sock or26 fd.name startswith /proc/*/root)27 output: >28 Container escape attempt detected29 (user=%user.name command=%proc.cmdline container=%container.name)30 priority: CRITICAL31 tags: [runner, escape, security]Container Network Policies:
1# Kubernetes NetworkPolicy for runner pods2apiVersion: networking.k8s.io/v13kind: NetworkPolicy4metadata:5 name: runner-network-policy6 namespace: runners7spec:8 podSelector:9 matchLabels:10 app: self-hosted-runner11 policyTypes:12 - Ingress13 - Egress14 ingress:15 - from:16 - namespaceSelector:17 matchLabels:18 name: monitoring19 ports:20 - protocol: TCP21 port: 808022 egress:23 - to: []24 ports:25 - protocol: TCP26 port: 443 # HTTPS only27 - protocol: UDP28 port: 53 # DNS29 - to:30 - namespaceSelector:31 matchLabels:32 name: kube-systemOperating System Hardening#
Linux Security Configuration#
CIS Benchmark Implementation:
1#!/bin/bash2# CIS Level 1 hardening for Ubuntu runner hosts34# Disable unused filesystems5cat > /etc/modprobe.d/blacklist-runner.conf << EOF6install cramfs /bin/true7install freevxfs /bin/true8install jffs2 /bin/true9install hfs /bin/true10install hfsplus /bin/true11install squashfs /bin/true12install udf /bin/true13install vfat /bin/true14EOF1516# Configure secure boot parameters17echo 'GRUB_CMDLINE_LINUX="audit=1 audit_backlog_limit=8192"' >> /etc/default/grub18update-grub1920# Harden kernel parameters21cat > /etc/sysctl.d/99-runner-security.conf << EOF22# IP Spoofing protection23net.ipv4.conf.all.rp_filter = 124net.ipv4.conf.default.rp_filter = 12526# Ignore ICMP ping requests27net.ipv4.icmp_echo_ignore_all = 12829# Ignore send redirects30net.ipv4.conf.all.send_redirects = 031net.ipv4.conf.default.send_redirects = 03233# Disable source packet routing34net.ipv4.conf.all.accept_source_route = 035net.ipv6.conf.all.accept_source_route = 03637# Ignore ICMP redirects38net.ipv4.conf.all.accept_redirects = 039net.ipv6.conf.all.accept_redirects = 04041# Ignore secure ICMP redirects42net.ipv4.conf.all.secure_redirects = 04344# Log Martians45net.ipv4.conf.all.log_martians = 14647# Enable TCP SYN Cookies48net.ipv4.tcp_syncookies = 14950# Disable IPv6 if not needed51net.ipv6.conf.all.disable_ipv6 = 152net.ipv6.conf.default.disable_ipv6 = 153EOF5455sysctl -p /etc/sysctl.d/99-runner-security.confUser and Permission Hardening:
1#!/bin/bash2# User security for runner hosts34# Create dedicated runner user5useradd -r -m -s /bin/bash -d /home/runner runner6usermod -aG docker runner78# Set secure umask9echo 'umask 027' >> /home/runner/.bashrc10echo 'umask 027' >> /etc/profile1112# Configure sudo access13cat > /etc/sudoers.d/runner << EOF14runner ALL=(ALL) NOPASSWD: /usr/bin/docker, /usr/bin/systemctl restart runner-service15Defaults:runner !visiblepw16Defaults:runner always_set_home17Defaults:runner match_group_by_gid18Defaults:runner env_reset19Defaults:runner env_keep += "SSH_AUTH_SOCK"20EOF2122# Set file permissions23chmod 755 /home/runner24chmod 700 /home/runner/.ssh25chmod 600 /home/runner/.ssh/authorized_keys26chmod 600 /home/runner/.bashrc2728# Remove unnecessary packages29apt-get remove --purge -y \30 telnet \31 rsh-client \32 rsh-server \33 ypbind \34 ypserv \35 tftp \36 tftp-server \37 talk \38 talk-server3940apt-get autoremove -ymacOS Security Configuration#
macOS Runner Hardening:
1#!/bin/bash2# macOS security configuration for runners34# Enable firewall5sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setglobalstate on6sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setloggingmode on7sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setstealthmode on89# Disable remote services10sudo launchctl disable system/com.openssh.sshd11sudo launchctl disable system/com.apple.screensharing1213# Configure automatic updates14sudo softwareupdate --schedule on15sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate AutomaticDownload -bool true16sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate AutomaticallyInstallMacOSUpdates -bool true1718# Secure system preferences19sudo spctl --master-enable20sudo defaults write /Library/Preferences/com.apple.alf globalstate -int 12122# Create runner user23sudo dscl . -create /Users/runner24sudo dscl . -create /Users/runner UserShell /bin/bash25sudo dscl . -create /Users/runner RealName "Runner Service Account"26sudo dscl . -create /Users/runner UniqueID 50327sudo dscl . -create /Users/runner PrimaryGroupID 2028sudo dscl . -create /Users/runner NFSHomeDirectory /Users/runner29sudo dscl . -passwd /Users/runner $(openssl rand -base64 32)3031# Configure runner environment32sudo mkdir -p /Users/runner33sudo chown runner:staff /Users/runner34sudo chmod 700 /Users/runnerWindows Security Configuration#
Windows Runner Hardening:
1# Windows security hardening for runners2# Run as Administrator34# Enable Windows Defender5Set-MpPreference -DisableRealtimeMonitoring $false6Set-MpPreference -SubmitSamplesConsent SendAllSamples7Set-MpPreference -MAPSReporting Advanced89# Configure Windows Firewall10netsh advfirewall set allprofiles state on11netsh advfirewall firewall add rule name="Allow HTTPS Outbound" dir=out action=allow protocol=TCP localport=44312netsh advfirewall firewall add rule name="Block All Inbound" dir=in action=block1314# Disable unnecessary services15$services = @(16 "Fax",17 "TelnetServer",18 "RemoteRegistry",19 "RemoteAccess",20 "SharedAccess"21)2223foreach ($service in $services) {24 Stop-Service -Name $service -Force -ErrorAction SilentlyContinue25 Set-Service -Name $service -StartupType Disabled -ErrorAction SilentlyContinue26}2728# Create runner service account29$runnerPassword = ConvertTo-SecureString -AsPlainText (New-Guid).Guid -Force30New-LocalUser -Name "RunnerService" -Password $runnerPassword -FullName "CI/CD Runner Service"31Add-LocalGroupMember -Group "Administrators" -Member "RunnerService"3233# Configure audit policies34auditpol /set /category:"Logon/Logoff" /success:enable /failure:enable35auditpol /set /category:"Object Access" /success:enable /failure:enable36auditpol /set /category:"Process Tracking" /success:enable /failure:enable37auditpol /set /category:"System" /success:enable /failure:enable3839# Harden registry settings40reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa" /v LmCompatibilityLevel /t REG_DWORD /d 5 /f41reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa" /v NoLMHash /t REG_DWORD /d 1 /f42reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa" /v DisableDomainCreds /t REG_DWORD /d 1 /f4344Write-Host "Windows hardening completed"Audit Logging#
Centralized Logging Configuration#
Rsyslog Configuration:
1# /etc/rsyslog.d/50-runner-audit.conf2# Runner audit logging configuration34# Load modules5module(load="imfile" PollingInterval="10")67# Runner service logs8input(type="imfile"9 File="/var/log/runner/service.log"10 Tag="runner-service"11 StateFile="runner-service-state")1213# Docker audit logs14input(type="imfile"15 File="/var/log/docker-audit.log"16 Tag="docker-audit"17 StateFile="docker-audit-state")1819# API access logs20input(type="imfile"21 File="/var/log/runner/api-access.log"22 Tag="api-access"23 StateFile="api-access-state")2425# Forward to central logging26*.* @@logs.assistance.bg:5142728# Local file backup29if $programname == 'runner-service' then /var/log/runner/audit.log30if $programname == 'docker-audit' then /var/log/security/docker.log31if $programname == 'api-access' then /var/log/security/api.log3233# Rotate logs34$WorkDirectory /var/spool/rsyslog35$ActionQueueFileName runner-audit36$ActionQueueMaxDiskSpace 1g37$ActionQueueSaveOnShutdown on38$ActionQueueType LinkedList39$ActionResumeRetryCount -1Audit Framework Setup:
1#!/bin/bash2# Configure Linux audit framework for runners34# Install auditd5apt-get install -y auditd audispd-plugins67# Configure audit rules8cat > /etc/audit/rules.d/runner-audit.rules << EOF9# Delete all existing rules10-D1112# Set buffer size13-b 81921415# Set failure mode16-f 11718# Audit file access in runner directories19-w /home/runner -p wa -k runner-files20-w /etc/runner -p wa -k runner-config21-w /var/lib/runner -p wa -k runner-data2223# Audit Docker socket access24-w /var/run/docker.sock -p wa -k docker-access2526# Audit system calls for privilege escalation27-a always,exit -F arch=b64 -S execve -k exec-monitoring28-a always,exit -F arch=b64 -S setuid -S setgid -S setreuid -S setregid -k privilege-escalation2930# Audit network connections31-a always,exit -F arch=b64 -S socket -S bind -S connect -k network-access3233# Audit file modifications34-a always,exit -F arch=b64 -S openat -S open -S creat -S truncate -S ftruncate -k file-access3536# Audit authentication events37-w /var/log/auth.log -p wa -k authentication38-w /etc/passwd -p wa -k passwd-changes39-w /etc/group -p wa -k group-changes40-w /etc/shadow -p wa -k shadow-changes4142# Lock configuration43-e 244EOF4546# Restart auditd47systemctl enable auditd48systemctl restart auditd4950echo "Audit configuration completed"Application Logging#
Structured Logging for Runners:
1#!/usr/bin/env python32# Runner audit logging implementation34import json5import logging6import logging.handlers7import time8from datetime import datetime9from typing import Dict, Any1011class RunnerAuditLogger:12 def __init__(self, log_file="/var/log/runner/audit.log"):13 self.logger = logging.getLogger("runner-audit")14 self.logger.setLevel(logging.INFO)1516 # File handler with rotation17 handler = logging.handlers.RotatingFileHandler(18 log_file, maxBytes=100*1024*1024, backupCount=1019 )2021 # JSON formatter22 formatter = logging.Formatter('%(message)s')23 handler.setFormatter(formatter)24 self.logger.addHandler(handler)2526 # Syslog handler for centralized logging27 syslog_handler = logging.handlers.SysLogHandler(address='/dev/log')28 syslog_handler.setFormatter(formatter)29 self.logger.addHandler(syslog_handler)3031 def log_event(self, event_type: str, details: Dict[str, Any],32 user_id: str = None, session_id: str = None):33 """Log structured audit events"""34 audit_entry = {35 "timestamp": datetime.utcnow().isoformat(),36 "event_type": event_type,37 "user_id": user_id,38 "session_id": session_id,39 "details": details,40 "hostname": "runner-host",41 "version": "1.0"42 }4344 self.logger.info(json.dumps(audit_entry))4546 def log_api_access(self, method: str, endpoint: str, status_code: int,47 response_time: float, user_agent: str = None):48 """Log API access events"""49 self.log_event("api_access", {50 "method": method,51 "endpoint": endpoint,52 "status_code": status_code,53 "response_time_ms": response_time * 1000,54 "user_agent": user_agent55 })5657 def log_workflow_execution(self, workflow_id: str, repository: str,58 branch: str, commit_sha: str, status: str):59 """Log workflow execution events"""60 self.log_event("workflow_execution", {61 "workflow_id": workflow_id,62 "repository": repository,63 "branch": branch,64 "commit_sha": commit_sha,65 "status": status66 })6768 def log_security_event(self, event_description: str, severity: str,69 source_ip: str = None):70 """Log security-related events"""71 self.log_event("security_event", {72 "description": event_description,73 "severity": severity,74 "source_ip": source_ip75 })7677# Usage example78if __name__ == "__main__":79 logger = RunnerAuditLogger()8081 # Log API access82 logger.log_api_access("POST", "/api/v2/projects", 201, 0.245)8384 # Log workflow execution85 logger.log_workflow_execution(86 "workflow-123", "org/repo", "main", "abc123", "completed"87 )8889 # Log security event90 logger.log_security_event(91 "Failed authentication attempt", "HIGH", "192.168.1.100"92 )Compliance Logging#
SOC 2 Audit Trail:
1#!/bin/bash2# SOC 2 compliance logging setup34# Create compliance log directories5mkdir -p /var/log/compliance/{access,changes,security}6chmod 750 /var/log/compliance7chown root:adm /var/log/compliance89# Configure compliance-specific logging10cat > /etc/rsyslog.d/60-compliance.conf << EOF11# SOC 2 Compliance Logging1213# Access controls and authentication14auth,authpriv.* /var/log/compliance/access/auth.log1516# System changes17*.notice;*.warn /var/log/compliance/changes/system.log1819# Security events20local0.* /var/log/compliance/security/events.log2122# Ensure log integrity23\$CreateDirs on24\$DirCreateMode 075025\$FileCreateMode 064026\$Umask 002227EOF2829# Setup log retention policy30cat > /etc/logrotate.d/compliance << EOF31/var/log/compliance/*/*.log {32 daily33 rotate 2555 # 7 years retention34 compress35 delaycompress36 missingok37 notifempty38 create 640 root adm39 postrotate40 /bin/kill -HUP \$(cat /var/run/rsyslogd.pid 2> /dev/null) 2> /dev/null || true41 endscript42}43EOF4445# Create compliance monitoring script46cat > /usr/local/bin/compliance-monitor.sh << 'EOF'47#!/bin/bash48# Compliance monitoring and alerting4950COMPLIANCE_LOG="/var/log/compliance/summary.log"51ALERT_THRESHOLD=55253# Check for security violations54security_events=$(grep -c "SECURITY_VIOLATION" /var/log/compliance/security/events.log 2>/dev/null || echo "0")5556if [ "$security_events" -gt "$ALERT_THRESHOLD" ]; then57 echo "$(date): HIGH: $security_events security events detected" >> "$COMPLIANCE_LOG"58 # Send alert to SOC59 curl -X POST "$SOC_WEBHOOK" -d "{\"alert\":\"High security events: $security_events\"}"60fi6162# Check access control violations63failed_access=$(grep -c "authentication failure" /var/log/compliance/access/auth.log 2>/dev/null || echo "0")6465if [ "$failed_access" -gt "$ALERT_THRESHOLD" ]; then66 echo "$(date): MEDIUM: $failed_access failed access attempts" >> "$COMPLIANCE_LOG"67fi6869# Generate daily compliance report70if [ "$(date +%H)" = "00" ]; then71 {72 echo "Daily Compliance Report - $(date +%Y-%m-%d)"73 echo "=================================="74 echo "Total security events: $security_events"75 echo "Failed access attempts: $failed_access"76 echo "System changes: $(grep -c "system change" /var/log/compliance/changes/system.log 2>/dev/null || echo "0")"77 } >> "$COMPLIANCE_LOG"78fi79EOF8081chmod +x /usr/local/bin/compliance-monitor.sh8283# Add to crontab84echo "*/15 * * * * /usr/local/bin/compliance-monitor.sh" | crontab -8586echo "Compliance logging configured"Incident Response#
Security Monitoring#
Real-time Threat Detection:
1#!/bin/bash2# Security monitoring and alerting for runners34# Install OSSEC HIDS for intrusion detection5wget -q -O - https://updates.atomicorp.com/installers/atomic | sh6yum install ossec-hids-server78# Configure OSSEC for runner monitoring9cat > /var/ossec/etc/ossec.conf << EOF10<ossec_config>11 <global>12 <email_notification>yes</email_notification>13 <email_to>[email protected]</email_to>14 <smtp_server>smtp.assistance.bg</smtp_server>15 <email_from>[email protected]</email_from>16 </global>1718 <rules>19 <include>rules_config.xml</include>20 <include>pam_rules.xml</include>21 <include>sshd_rules.xml</include>22 <include>telnetd_rules.xml</include>23 <include>syslog_rules.xml</include>24 <include>arpwatch_rules.xml</include>25 <include>symantec-av_rules.xml</include>26 <include>symantec-ws_rules.xml</include>27 <include>pix_rules.xml</include>28 <include>named_rules.xml</include>29 <include>smbd_rules.xml</include>30 <include>vsftpd_rules.xml</include>31 <include>pure-ftpd_rules.xml</include>32 <include>proftpd_rules.xml</include>33 <include>ms_ftpd_rules.xml</include>34 <include>ftpd_rules.xml</include>35 <include>hordeimp_rules.xml</include>36 <include>roundcube_rules.xml</include>37 <include>wordpress_rules.xml</include>38 <include>cimserver_rules.xml</include>39 <include>vpopmail_rules.xml</include>40 <include>vmpop3d_rules.xml</include>41 <include>courier_rules.xml</include>42 <include>web_rules.xml</include>43 <include>web_appsec_rules.xml</include>44 <include>apache_rules.xml</include>45 <include>nginx_rules.xml</include>46 <include>php_rules.xml</include>47 <include>mysql_rules.xml</include>48 <include>postgresql_rules.xml</include>49 <include>ids_rules.xml</include>50 <include>squid_rules.xml</include>51 <include>firewall_rules.xml</include>52 <include>cisco-ios_rules.xml</include>53 <include>netscreenfw_rules.xml</include>54 <include>sonicwall_rules.xml</include>55 <include>postfix_rules.xml</include>56 <include>sendmail_rules.xml</include>57 <include>imapd_rules.xml</include>58 <include>mailscanner_rules.xml</include>59 <include>dovecot_rules.xml</include>60 <include>ms-exchange_rules.xml</include>61 <include>racoon_rules.xml</include>62 <include>vpn_concentrator_rules.xml</include>63 <include>spamd_rules.xml</include>64 <include>msauth_rules.xml</include>65 <include>mcafee_av_rules.xml</include>66 <include>trend-osce_rules.xml</include>67 <include>ms-se_rules.xml</include>68 <include>zeus_rules.xml</include>69 <include>solaris_bsm_rules.xml</include>70 <include>vmware_rules.xml</include>71 <include>ms_dhcp_rules.xml</include>72 <include>asterisk_rules.xml</include>73 <include>ossec_rules.xml</include>74 <include>attack_rules.xml</include>75 <include>local_rules.xml</include>76 </rules>7778 <syscheck>79 <directories check_all="yes">/home/runner</directories>80 <directories check_all="yes">/etc/runner</directories>81 <directories check_all="yes">/var/lib/runner</directories>82 <directories check_all="yes">/etc/passwd</directories>83 <directories check_all="yes">/etc/shadow</directories>84 <directories check_all="yes">/etc/group</directories>85 </syscheck>8687 <rootcheck>88 <rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files>89 <rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>90 <system_audit>/var/ossec/etc/shared/system_audit_rcl.txt</system_audit>91 <system_audit>/var/ossec/etc/shared/cis_debian_linux_rcl.txt</system_audit>92 <system_audit>/var/ossec/etc/shared/cis_rhel_linux_rcl.txt</system_audit>93 <system_audit>/var/ossec/etc/shared/cis_rhel5_linux_rcl.txt</system_audit>94 </rootcheck>9596 <localfile>97 <log_format>syslog</log_format>98 <location>/var/log/auth.log</location>99 </localfile>100101 <localfile>102 <log_format>syslog</log_format>103 <location>/var/log/syslog</location>104 </localfile>105106 <localfile>107 <log_format>apache</log_format>108 <location>/var/log/runner/access.log</location>109 </localfile>110111 <command>112 <name>host-deny</name>113 <executable>host-deny.sh</executable>114 <expect>srcip</expect>115 <timeout_allowed>yes</timeout_allowed>116 </command>117118 <command>119 <name>firewall-drop</name>120 <executable>firewall-drop.sh</executable>121 <expect>srcip</expect>122 <timeout_allowed>yes</timeout_allowed>123 </command>124125 <active-response>126 <command>host-deny</command>127 <location>local</location>128 <rules_id>5712</rules_id>129 <timeout>600</timeout>130 </active-response>131132 <active-response>133 <command>firewall-drop</command>134 <location>local</location>135 <rules_id>5712</rules_id>136 <timeout>600</timeout>137 </active-response>138</ossec_config>139EOF140141# Start OSSEC142/var/ossec/bin/ossec-control startCustom Security Rules:
1<!-- /var/ossec/rules/runner_rules.xml -->2<group name="runner,security">34 <!-- Runner authentication failures -->5 <rule id="100001" level="5">6 <if_sid>5716</if_sid>7 <srcip>!192.168.0.0/16</srcip>8 <description>SSH authentication failure from external IP</description>9 </rule>1011 <rule id="100002" level="10" frequency="3" timeframe="180">12 <if_matched_sid>100001</if_matched_sid>13 <description>Multiple SSH authentication failures from external IP</description>14 </rule>1516 <!-- Suspicious runner activity -->17 <rule id="100010" level="8">18 <program_name>runner-service</program_name>19 <match>unauthorized|forbidden|denied</match>20 <description>Unauthorized access attempt to runner service</description>21 </rule>2223 <!-- Container security events -->24 <rule id="100020" level="10">25 <program_name>docker</program_name>26 <match>privilege|escalation|container_escape</match>27 <description>Docker security violation detected</description>28 </rule>2930 <!-- API abuse detection -->31 <rule id="100030" level="7" frequency="10" timeframe="60">32 <program_name>api-access</program_name>33 <match>4[0-9][0-9]|5[0-9][0-9]</match>34 <description>High rate of API errors detected</description>35 </rule>3637</group>Automated Response Procedures#
Incident Response Automation:
1#!/bin/bash2# Automated incident response for runner security events34INCIDENT_LOG="/var/log/security/incidents.log"5ALERT_WEBHOOK="$SLACK_SECURITY_WEBHOOK"67log_incident() {8 local severity="$1"9 local description="$2"10 local timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")1112 echo "$timestamp [$severity] $description" >> "$INCIDENT_LOG"1314 # Send alert for high severity incidents15 if [ "$severity" = "CRITICAL" ] || [ "$severity" = "HIGH" ]; then16 curl -X POST "$ALERT_WEBHOOK" \17 -H "Content-Type: application/json" \18 -d "{19 \"text\": \"🚨 Security Incident\",20 \"attachments\": [{21 \"color\": \"danger\",22 \"fields\": [{23 \"title\": \"Severity\",24 \"value\": \"$severity\",25 \"short\": true26 }, {27 \"title\": \"Description\",28 \"value\": \"$description\",29 \"short\": false30 }]31 }]32 }"33 fi34}3536respond_to_incident() {37 local incident_type="$1"38 local source_ip="$2"39 local details="$3"4041 case "$incident_type" in42 "brute_force")43 log_incident "HIGH" "Brute force attack from $source_ip"44 # Block IP45 iptables -A INPUT -s "$source_ip" -j DROP46 echo "$source_ip" >> /etc/hosts.deny47 ;;4849 "privilege_escalation")50 log_incident "CRITICAL" "Privilege escalation detected: $details"51 # Isolate runner52 systemctl stop runner-service53 # Preserve evidence54 tar -czf "/var/log/security/evidence-$(date +%s).tar.gz" \55 /var/log/runner /home/runner /etc/runner56 ;;5758 "container_escape")59 log_incident "CRITICAL" "Container escape attempt: $details"60 # Stop all containers61 docker stop $(docker ps -q)62 # Remove compromised containers63 docker system prune -f64 ;;6566 "api_abuse")67 log_incident "MEDIUM" "API abuse from $source_ip: $details"68 # Rate limit IP69 iptables -A INPUT -p tcp --dport 443 -s "$source_ip" \70 -m limit --limit 1/minute -j ACCEPT71 iptables -A INPUT -p tcp --dport 443 -s "$source_ip" -j DROP72 ;;73 esac74}7576# Monitor for security events77monitor_security_events() {78 tail -F /var/log/ossec/alerts/alerts.log | while read line; do79 if echo "$line" | grep -q "Rule: 100002"; then80 source_ip=$(echo "$line" | grep -o '[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+' | head -1)81 respond_to_incident "brute_force" "$source_ip" "SSH brute force"82 elif echo "$line" | grep -q "privilege"; then83 respond_to_incident "privilege_escalation" "" "$line"84 elif echo "$line" | grep -q "container_escape"; then85 respond_to_incident "container_escape" "" "$line"86 fi87 done88}8990# Start monitoring91monitor_security_events &92echo $! > /var/run/security-monitor.pidBreach Response Procedures#
Security Incident Playbook:
1# incident-response-playbook.yml2incident_response:3 phases:4 preparation:5 - Ensure incident response team contacts are current6 - Verify communication channels are operational7 - Confirm backup and recovery procedures are tested8 - Validate monitoring systems are functional910 identification:11 triggers:12 - OSSEC high severity alerts13 - Abnormal network traffic patterns14 - Unusual API usage patterns15 - Failed authentication spikes16 - Container security violations1718 initial_assessment:19 - Classify incident severity (Low/Medium/High/Critical)20 - Determine affected systems and data21 - Estimate potential impact22 - Activate appropriate response team2324 containment:25 short_term:26 - Isolate affected runners from network27 - Preserve system state for forensics28 - Change all authentication credentials29 - Enable enhanced monitoring3031 long_term:32 - Rebuild affected systems from clean images33 - Implement additional security controls34 - Update firewall rules35 - Review and strengthen access controls3637 eradication:38 - Remove malware or unauthorized access39 - Patch vulnerabilities that enabled the incident40 - Update security configurations41 - Strengthen authentication mechanisms4243 recovery:44 - Restore systems from clean backups45 - Gradually restore services with enhanced monitoring46 - Validate system integrity47 - Update security baselines4849 lessons_learned:50 - Conduct post-incident review51 - Document lessons learned52 - Update incident response procedures53 - Implement preventive measures54 - Update security training materialsForensic Data Collection:
1#!/bin/bash2# Forensic data collection for runner security incidents34INCIDENT_ID="INC-$(date +%Y%m%d-%H%M%S)"5EVIDENCE_DIR="/var/forensics/$INCIDENT_ID"67collect_evidence() {8 echo "Starting forensic collection for incident: $INCIDENT_ID"910 # Create evidence directory11 mkdir -p "$EVIDENCE_DIR"12 cd "$EVIDENCE_DIR"1314 # System information15 echo "Collecting system information..."16 uname -a > system_info.txt17 ps aux > running_processes.txt18 netstat -tulpn > network_connections.txt19 lsof > open_files.txt20 mount > mounted_filesystems.txt21 df -h > disk_usage.txt2223 # Memory dump24 echo "Capturing memory dump..."25 if command -v avml &> /dev/null; then26 avml memory_dump.lime27 else28 dd if=/dev/mem of=memory_dump.raw bs=1M 2>/dev/null || echo "Memory dump failed"29 fi3031 # Log files32 echo "Collecting logs..."33 mkdir logs34 cp -r /var/log/* logs/ 2>/dev/null35 cp -r /var/log/runner/* logs/ 2>/dev/null36 cp -r /var/log/docker/* logs/ 2>/dev/null3738 # Runner configuration39 echo "Collecting runner configuration..."40 mkdir config41 cp -r /etc/runner/* config/ 2>/dev/null42 cp /etc/passwd config/43 cp /etc/group config/44 cp /etc/shadow config/ 2>/dev/null4546 # Docker information47 echo "Collecting Docker information..."48 docker ps -a > docker_containers.txt49 docker images > docker_images.txt50 docker version > docker_version.txt51 docker info > docker_info.txt5253 # Network configuration54 echo "Collecting network configuration..."55 ip addr show > network_interfaces.txt56 ip route show > routing_table.txt57 iptables -L -n -v > firewall_rules.txt5859 # File system timeline60 echo "Creating file system timeline..."61 find /home/runner -type f -exec stat {} \; > runner_file_timeline.txt 2>/dev/null62 find /etc/runner -type f -exec stat {} \; > config_file_timeline.txt 2>/dev/null6364 # Hash verification65 echo "Computing file hashes..."66 find /home/runner /etc/runner -type f -exec md5sum {} \; > file_hashes.txt 2>/dev/null6768 # Create tamper-evident archive69 echo "Creating evidence archive..."70 cd ..71 tar -czf "$INCIDENT_ID.tar.gz" "$INCIDENT_ID"72 sha256sum "$INCIDENT_ID.tar.gz" > "$INCIDENT_ID.sha256"7374 echo "Evidence collection complete: $EVIDENCE_DIR"75 echo "Archive: $INCIDENT_ID.tar.gz"76 echo "Hash: $(cat $INCIDENT_ID.sha256)"77}7879# Execute collection80collect_evidence8182# Secure evidence83chmod 440 "$INCIDENT_ID.tar.gz"84chmod 440 "$INCIDENT_ID.sha256"85chown root:security "$INCIDENT_ID.tar.gz" "$INCIDENT_ID.sha256"Compliance Standards#
SOC 2 Type II Compliance#
Control Implementation:
1#!/bin/bash2# SOC 2 Type II control implementation for runners34# CC6.1 - Logical and Physical Access Controls5implement_access_controls() {6 echo "Implementing SOC 2 CC6.1 controls..."78 # Multi-factor authentication9 apt-get install -y libpam-google-authenticator1011 cat > /etc/pam.d/sshd << EOF12auth required pam_google_authenticator.so13auth required pam_unix.so14account required pam_unix.so15session required pam_unix.so16EOF1718 # Password complexity19 cat >> /etc/security/pwquality.conf << EOF20minlen = 1221minclass = 322maxrepeat = 223maxclasschars = 424EOF2526 # Session management27 cat >> /etc/security/limits.conf << EOF28* hard maxlogins 329* hard maxsyslogins 1030EOF31}3233# CC6.2 - Authentication and Authorization34implement_authentication() {35 echo "Implementing SOC 2 CC6.2 controls..."3637 # Certificate-based authentication38 mkdir -p /etc/ssh/ca39 ssh-keygen -t ed25519 -f /etc/ssh/ca/runner_ca -C "Runner CA"4041 cat >> /etc/ssh/sshd_config << EOF42TrustedUserCAKeys /etc/ssh/ca/runner_ca.pub43AuthorizedPrincipalsFile /etc/ssh/authorized_principals/%u44ClientAliveInterval 30045ClientAliveCountMax 046LoginGraceTime 6047MaxAuthTries 348MaxSessions 249PermitRootLogin no50PasswordAuthentication no51ChallengeResponseAuthentication yes52EOF53}5455# CC6.3 - System Operations56implement_system_operations() {57 echo "Implementing SOC 2 CC6.3 controls..."5859 # Change management tracking60 cat > /usr/local/bin/change-tracker.sh << 'EOF'61#!/bin/bash62CHANGE_LOG="/var/log/compliance/changes.log"6364track_change() {65 local change_type="$1"66 local description="$2"67 local timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")68 local user=$(whoami)6970 echo "$timestamp|$user|$change_type|$description" >> "$CHANGE_LOG"71}7273# Hook into package management74if [ "$1" = "install" ] || [ "$1" = "upgrade" ] || [ "$1" = "remove" ]; then75 track_change "package" "$*"76fi7778# Hook into configuration changes79if [ -f "$2" ] && [ "$1" = "edit" ]; then80 track_change "config" "Modified $2"81fi82EOF8384 chmod +x /usr/local/bin/change-tracker.sh85}8687# CC7.1 - System Monitoring88implement_monitoring() {89 echo "Implementing SOC 2 CC7.1 controls..."9091 # Performance monitoring92 apt-get install -y prometheus-node-exporter9394 cat > /etc/prometheus/node_exporter.yml << EOF95collectors:96 cpu:97 diskstats:98 filesystem:99 ignored-mount-points: "^/(dev|proc|sys|var/lib/docker/.+)($|/)"100 loadavg:101 meminfo:102 netdev:103 netstat:104 systemd:105global:106 scrape_interval: 15s107EOF108109 systemctl enable prometheus-node-exporter110 systemctl start prometheus-node-exporter111}112113# CC8.1 - Change Management114implement_change_management() {115 echo "Implementing SOC 2 CC8.1 controls..."116117 cat > /etc/sudoers.d/change-approval << EOF118# Require change approval for system modifications119Cmnd_Alias CHANGE_COMMANDS = /usr/bin/apt, /usr/bin/systemctl, /bin/cp /etc/*, /bin/mv /etc/*120121# Log all changes122Defaults log_host, log_year, logfile="/var/log/sudo.log"123EOF124}125126# Execute all implementations127implement_access_controls128implement_authentication129implement_system_operations130implement_monitoring131implement_change_management132133echo "SOC 2 Type II controls implemented"ISO 27001 Information Security Management#
Risk Assessment Framework:
1#!/usr/bin/env python32# ISO 27001 risk assessment for runner infrastructure34import json5import datetime6from typing import Dict, List, Tuple78class ISO27001RiskAssessment:9 def __init__(self):10 self.assets = []11 self.threats = []12 self.vulnerabilities = []13 self.controls = []14 self.risk_matrix = {}1516 def add_asset(self, name: str, value: int, criticality: str):17 """Add information asset"""18 self.assets.append({19 'name': name,20 'value': value,21 'criticality': criticality,22 'id': len(self.assets) + 123 })2425 def add_threat(self, name: str, probability: float, impact: int):26 """Add threat to threat register"""27 self.threats.append({28 'name': name,29 'probability': probability, # 0.1 to 1.030 'impact': impact, # 1 to 531 'id': len(self.threats) + 132 })3334 def add_vulnerability(self, name: str, asset_id: int, threat_id: int,35 exploitability: float):36 """Add vulnerability"""37 self.vulnerabilities.append({38 'name': name,39 'asset_id': asset_id,40 'threat_id': threat_id,41 'exploitability': exploitability, # 0.1 to 1.042 'id': len(self.vulnerabilities) + 143 })4445 def add_control(self, name: str, effectiveness: float,46 vulnerability_ids: List[int]):47 """Add security control"""48 self.controls.append({49 'name': name,50 'effectiveness': effectiveness, # 0.1 to 1.051 'vulnerability_ids': vulnerability_ids,52 'id': len(self.controls) + 153 })5455 def calculate_risk(self) -> Dict:56 """Calculate risk levels for all assets"""57 risks = []5859 for asset in self.assets:60 for vulnerability in self.vulnerabilities:61 if vulnerability['asset_id'] == asset['id']:62 threat = next(t for t in self.threats63 if t['id'] == vulnerability['threat_id'])6465 # Calculate inherent risk66 likelihood = threat['probability'] * vulnerability['exploitability']67 impact = threat['impact'] * (asset['value'] / 5)68 inherent_risk = likelihood * impact6970 # Calculate residual risk considering controls71 risk_reduction = 072 for control in self.controls:73 if vulnerability['id'] in control['vulnerability_ids']:74 risk_reduction += control['effectiveness']7576 risk_reduction = min(risk_reduction, 0.95) # Max 95% reduction77 residual_risk = inherent_risk * (1 - risk_reduction)7879 risks.append({80 'asset': asset['name'],81 'threat': threat['name'],82 'vulnerability': vulnerability['name'],83 'inherent_risk': round(inherent_risk, 2),84 'residual_risk': round(residual_risk, 2),85 'risk_level': self.get_risk_level(residual_risk)86 })8788 return {'risks': risks, 'assessment_date': datetime.datetime.now().isoformat()}8990 def get_risk_level(self, risk_score: float) -> str:91 """Determine risk level based on score"""92 if risk_score >= 4.0:93 return "CRITICAL"94 elif risk_score >= 3.0:95 return "HIGH"96 elif risk_score >= 2.0:97 return "MEDIUM"98 elif risk_score >= 1.0:99 return "LOW"100 else:101 return "VERY LOW"102103 def generate_report(self) -> str:104 """Generate ISO 27001 risk assessment report"""105 assessment = self.calculate_risk()106107 report = f"""108ISO 27001 Risk Assessment Report109Generated: {assessment['assessment_date']}110111EXECUTIVE SUMMARY112================113Total Risks Identified: {len(assessment['risks'])}114115Risk Distribution:116"""117118 risk_levels = {}119 for risk in assessment['risks']:120 level = risk['risk_level']121 risk_levels[level] = risk_levels.get(level, 0) + 1122123 for level, count in sorted(risk_levels.items()):124 report += f"- {level}: {count}\n"125126 report += "\nDETAILED RISK REGISTER\n=====================\n"127128 for risk in sorted(assessment['risks'],129 key=lambda x: x['residual_risk'], reverse=True):130 report += f"""131Asset: {risk['asset']}132Threat: {risk['threat']}133Vulnerability: {risk['vulnerability']}134Inherent Risk: {risk['inherent_risk']}135Residual Risk: {risk['residual_risk']}136Risk Level: {risk['risk_level']}137---138"""139140 return report141142# Runner infrastructure risk assessment143if __name__ == "__main__":144 assessment = ISO27001RiskAssessment()145146 # Add assets147 assessment.add_asset("Runner Infrastructure", 4, "HIGH")148 assessment.add_asset("API Keys and Secrets", 5, "CRITICAL")149 assessment.add_asset("Source Code Access", 4, "HIGH")150 assessment.add_asset("Build Artifacts", 3, "MEDIUM")151152 # Add threats153 assessment.add_threat("External Hacker", 0.7, 5)154 assessment.add_threat("Insider Threat", 0.3, 4)155 assessment.add_threat("Supply Chain Attack", 0.5, 4)156 assessment.add_threat("Configuration Error", 0.8, 3)157158 # Add vulnerabilities159 assessment.add_vulnerability("Unpatched OS", 1, 1, 0.8)160 assessment.add_vulnerability("Weak Authentication", 2, 1, 0.9)161 assessment.add_vulnerability("Container Escape", 1, 1, 0.6)162 assessment.add_vulnerability("Privilege Escalation", 1, 2, 0.7)163164 # Add controls165 assessment.add_control("OS Hardening", 0.8, [1])166 assessment.add_control("MFA Implementation", 0.9, [2])167 assessment.add_control("Container Security", 0.7, [3])168 assessment.add_control("Access Controls", 0.8, [4])169170 # Generate and print report171 print(assessment.generate_report())PCI DSS Compliance#
Payment Card Industry Data Security Standard:
1#!/bin/bash2# PCI DSS compliance implementation for runners handling payment data34# Requirement 1: Install and maintain a firewall configuration5implement_pci_firewall() {6 echo "Implementing PCI DSS Requirement 1..."78 # Default deny policy9 iptables -P INPUT DROP10 iptables -P FORWARD DROP11 iptables -P OUTPUT DROP1213 # Allow only necessary connections14 iptables -A INPUT -i lo -j ACCEPT15 iptables -A OUTPUT -o lo -j ACCEPT1617 # SSH (restricted)18 iptables -A INPUT -p tcp --dport 22 -s 10.0.0.0/8 -j ACCEPT19 iptables -A OUTPUT -p tcp --sport 22 -d 10.0.0.0/8 -j ACCEPT2021 # HTTPS for PCI compliance22 iptables -A OUTPUT -p tcp --dport 443 -j ACCEPT23 iptables -A INPUT -p tcp --sport 443 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT2425 # DNS26 iptables -A OUTPUT -p udp --dport 53 -j ACCEPT27 iptables -A INPUT -p udp --sport 53 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT2829 # Save configuration30 iptables-save > /etc/iptables/rules.v431}3233# Requirement 2: Do not use vendor-supplied defaults34implement_pci_defaults() {35 echo "Implementing PCI DSS Requirement 2..."3637 # Change default passwords38 echo "runner:$(openssl rand -base64 32)" | chpasswd3940 # Remove default accounts41 userdel -r ubuntu 2>/dev/null || true42 userdel -r debian 2>/dev/null || true4344 # Secure SSH configuration45 cat > /etc/ssh/sshd_config << EOF46Protocol 247Port 2248PermitRootLogin no49PasswordAuthentication no50PubkeyAuthentication yes51AuthorizedKeysFile .ssh/authorized_keys52IgnoreRhosts yes53HostbasedAuthentication no54PermitEmptyPasswords no55ChallengeResponseAuthentication yes56UsePAM yes57X11Forwarding no58PrintMotd no59AcceptEnv LANG LC_*60Subsystem sftp /usr/lib/openssh/sftp-server61ClientAliveInterval 30062ClientAliveCountMax 063MaxAuthTries 364MaxSessions 265EOF6667 systemctl restart sshd68}6970# Requirement 3: Protect stored cardholder data71implement_pci_encryption() {72 echo "Implementing PCI DSS Requirement 3..."7374 # Encrypt file systems75 apt-get install -y cryptsetup7677 # Setup encrypted swap78 swapoff -a79 cryptsetup luksFormat /dev/disk/by-label/swap80 cryptsetup luksOpen /dev/disk/by-label/swap swap81 mkswap /dev/mapper/swap82 swapon /dev/mapper/swap8384 # Ensure no cardholder data in logs85 cat > /etc/rsyslog.d/99-pci-scrubbing.conf << EOF86# PCI DSS data scrubbing87\$ModLoad impcre88\$RepeatedMsgReduction on8990# Remove credit card patterns91:msg, regex, "[0-9]{4}[[:space:]-]*[0-9]{4}[[:space:]-]*[0-9]{4}[[:space:]-]*[0-9]{4}" stop92EOF9394 systemctl restart rsyslog95}9697# Requirement 4: Encrypt transmission of cardholder data98implement_pci_transmission() {99 echo "Implementing PCI DSS Requirement 4..."100101 # Ensure only TLS 1.2+ is used102 cat > /etc/ssl/openssl.conf << EOF103MinProtocol = TLSv1.2104CipherString = HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!SRP:!CAMELLIA105EOF106107 # Configure strong TLS for applications108 cat > /etc/nginx/conf.d/pci-tls.conf << EOF109ssl_protocols TLSv1.2 TLSv1.3;110ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256';111ssl_prefer_server_ciphers on;112ssl_session_timeout 5m;113ssl_session_cache shared:SSL:10m;114EOF115}116117# Execute PCI DSS implementation118implement_pci_firewall119implement_pci_defaults120implement_pci_encryption121implement_pci_transmission122123echo "PCI DSS compliance controls implemented"GDPR Data Protection#
General Data Protection Regulation compliance:
1#!/bin/bash2# GDPR compliance for runner infrastructure processing EU personal data34# Data Protection Impact Assessment5create_dpia_framework() {6 echo "Creating GDPR DPIA framework..."78 cat > /etc/runner/dpia.json << EOF9{10 "data_protection_impact_assessment": {11 "assessment_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",12 "controller": "DevOps Hub",13 "dpo_contact": "[email protected]",14 "processing_purpose": "CI/CD automation and testing",15 "data_categories": [16 "developer_identities",17 "commit_metadata",18 "build_logs",19 "performance_metrics"20 ],21 "legal_basis": "legitimate_interest",22 "data_retention": "90_days",23 "privacy_measures": [24 "data_minimization",25 "pseudonymization",26 "encryption_at_rest",27 "encryption_in_transit",28 "access_logging",29 "automated_deletion"30 ]31 }32}33EOF34}3536# Implement privacy by design37implement_privacy_by_design() {38 echo "Implementing GDPR privacy by design..."3940 # Data minimization - collect only necessary data41 cat > /usr/local/bin/gdpr-data-minimizer.py << 'EOF'42#!/usr/bin/env python343import re44import json45import sys4647def minimize_log_data(log_line):48 """Remove or pseudonymize personal data from logs"""4950 # Remove email addresses51 log_line = re.sub(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}',52 '[EMAIL_REMOVED]', log_line)5354 # Remove IP addresses (keep only subnet)55 log_line = re.sub(r'\b(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})\b',56 r'\1.\2.XXX.XXX', log_line)5758 # Remove potential personal identifiers59 log_line = re.sub(r'/users/[^/\s]+', '/users/[USER]', log_line)6061 return log_line6263if __name__ == "__main__":64 for line in sys.stdin:65 print(minimize_log_data(line.strip()))66EOF6768 chmod +x /usr/local/bin/gdpr-data-minimizer.py69}7071# Data subject rights implementation72implement_data_rights() {73 echo "Implementing GDPR data subject rights..."7475 cat > /usr/local/bin/gdpr-rights-handler.sh << 'EOF'76#!/bin/bash77# Handle GDPR data subject rights requests7879GDPR_LOG="/var/log/gdpr/rights-requests.log"8081handle_access_request() {82 local subject_id="$1"83 local request_id="$2"8485 echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) - Processing access request for $subject_id (Request: $request_id)" >> "$GDPR_LOG"8687 # Search for personal data88 find /var/log/runner -name "*.log" -exec grep -l "$subject_id" {} \; > "/tmp/subject_data_$request_id.txt"8990 # Create data export91 mkdir -p "/var/gdpr/exports/$request_id"92 while read -r file; do93 grep "$subject_id" "$file" > "/var/gdpr/exports/$request_id/$(basename $file)"94 done < "/tmp/subject_data_$request_id.txt"9596 # Encrypt export97 tar -czf "/var/gdpr/exports/$request_id.tar.gz" "/var/gdpr/exports/$request_id"98 openssl enc -aes-256-cbc -salt -in "/var/gdpr/exports/$request_id.tar.gz" \99 -out "/var/gdpr/exports/$request_id.enc" -k "$GDPR_ENCRYPTION_KEY"100101 echo "Access request completed for $subject_id (Export: $request_id.enc)"102}103104handle_erasure_request() {105 local subject_id="$1"106 local request_id="$2"107108 echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) - Processing erasure request for $subject_id (Request: $request_id)" >> "$GDPR_LOG"109110 # Find and remove personal data111 find /var/log/runner -name "*.log" -exec sed -i "s/$subject_id/[ERASED]/g" {} \;112113 # Remove from databases114 sqlite3 /var/lib/runner/runner.db "UPDATE builds SET author='[ERASED]' WHERE author='$subject_id'"115116 echo "Erasure request completed for $subject_id"117}118119case "$1" in120 "access")121 handle_access_request "$2" "$3"122 ;;123 "erasure")124 handle_erasure_request "$2" "$3"125 ;;126 *)127 echo "Usage: $0 {access|erasure} subject_id request_id"128 ;;129esac130EOF131132 chmod +x /usr/local/bin/gdpr-rights-handler.sh133}134135# Automated data retention and deletion136implement_data_retention() {137 echo "Implementing GDPR data retention policies..."138139 cat > /usr/local/bin/gdpr-retention.sh << 'EOF'140#!/bin/bash141# GDPR-compliant data retention and deletion142143RETENTION_DAYS=90144LOG_DIR="/var/log/runner"145GDPR_LOG="/var/log/gdpr/retention.log"146147# Delete logs older than retention period148find "$LOG_DIR" -name "*.log" -type f -mtime +$RETENTION_DAYS -exec rm -f {} \;149150# Log deletion activity151echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) - Automated data deletion completed (retention: ${RETENTION_DAYS} days)" >> "$GDPR_LOG"152153# Secure deletion of sensitive data154find /tmp -name "*personal*" -type f -mmin +60 -exec shred -vfz -n 3 {} \;155EOF156157 chmod +x /usr/local/bin/gdpr-retention.sh158159 # Schedule retention cleanup160 echo "0 2 * * * /usr/local/bin/gdpr-retention.sh" | crontab -161}162163# Execute GDPR implementation164create_dpia_framework165implement_privacy_by_design166implement_data_rights167implement_data_retention168169echo "GDPR compliance controls implemented"This completes the comprehensive security hardening guide for self-hosted runners, covering all the requested topics:
- Security Overview - Threat models and attack vectors
- Network Security - VPN, firewall, and zero-trust configurations
- Platform-Specific Hardening - Security configs for all 5 platforms
- DevOps Hub API Security - Authentication, rate limiting, TLS
- Secret Management - Vault, AWS Secrets, Azure Key Vault, GCP
- Container Security - Docker hardening, image scanning, runtime security
- Operating System Hardening - Linux, macOS, Windows security
- Audit Logging - Centralized logging and compliance requirements
- Incident Response - Security monitoring and breach procedures
- Compliance Standards - SOC 2, ISO 27001, PCI DSS, GDPR
The guide provides enterprise-grade security practices that organizations can implement for compliant and secure runner deployments across all major CI/CD platforms.