GitHub Actions: How to Use Self-Hosted Runners for Custom Workflows

In today’s fast-paced development world, automation is key to maintaining efficiency. GitHub Actions offers a robust platform for automating workflows, from CI/CD pipelines to other development tasks. While GitHub provides hosted runners for convenience, self-hosted runners offer unparalleled customization and control. In this post, we’ll explore how to set up and utilize GitHub self-hosted runners to tailor your workflows to your exact requirements.

What Are Self-Hosted Runners?

Self-hosted runners are virtual machines (VMs) or physical machines that you set up and manage yourself. These runners execute your GitHub Actions workflows on your own hardware, offering greater flexibility than GitHub’s hosted runners. They’re particularly valuable when your workflows require specialized tools, configurations, or resources that are not available on GitHub-hosted runners.

Benefits of Self-Hosted Runners

Custom Environments:
Self-hosted runners give you full control over the environment. You can install specific tools, dependencies, and configurations that align with your project’s needs.
Faster Workflows:
If you use high-performance hardware with optimized configurations, self-hosted runners can significantly accelerate build, test, and deployment processes.
Cost Efficiency:
Running workflows on your infrastructure can reduce costs, especially for resource-intensive jobs, compared to relying on GitHub-hosted runners.
Security and Compliance:
You can manage sensitive data within your own network, providing better isolation and control over your workflows.

Setting Up a Self-Hosted Runner

Setting up a self-hosted runner involves these steps:

Prepare Your Machine

Select a machine that meets the requirements for GitHub self-hosted runners. Ensure the operating system and hardware specifications align with your workflow needs. Install necessary dependencies like Docker if your workflows use containerized builds.

Download the Runner Application:

Navigate to your repository, organization, or enterprise in GitHub.

Go to Settings > Actions > Runners, and click Add Runner to download the runner application for your operating system.

Configure the Runner

Extract the downloaded package and follow the instructions to register the runner. During registration, you’ll need an authentication token, which is generated in the GitHub UI when you add a runner. Assign the runner to a specific repository, organization, or enterprise and set any necessary labels to help identify the runner for specific jobs.

Start the Runner

After registration, start the runner application. Verify its status by checking the GitHub Actions interface. Run a test workflow to confirm the self-hosted runner is functioning as expected.

Automating Self-Hosted Runner Deployment

To streamline the deployment of self-hosted runners, you can automate the process using Infrastructure as Code (IaC). This is especially useful for managing runners at scale or provisioning VMs dynamically.

Bicep Deployment

Using IaC tools like Terraform or Bicep, you can deploy and configure the VM to host the runner automatically. Here’s an example of deploying a virtual machine with Azure CLI for a GitHub self-hosted runner:

Please ensure Microsoft.AzureCLI and Microsoft.Bicep are installed

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
targetScope = 'subscription'

// Imported Values

@description('The location of the resources')
param location string

@description('The short code of the location')
param locationShortCode string

@description('The environment type')
@allowed([
  'dev'
  'acc'
  'prod'
])
param environmentType string

@description('The Public IP Address')
param publicIp string

@description('The name of the virtual machine')
param vmHostName string = 'github-agent-${environmentType}-${locationShortCode}'

@description('The Local User Account Name')
param vmUserName string

@description('The Local User Account Password')
@secure()
param vmUserPassword string

@description('The Resource Group Name')
param resourceGroupName string = 'rg-github-agent-${environmentType}-${locationShortCode}'

@description('The Network Security Group Name')
param networkSecurityGroupName string = 'nsg-github-agent-${locationShortCode}'

@description('The Virtual Network Name')
param virtualNetworkName string = 'vnet-hosted-agents-${environmentType}-${locationShortCode}'

@description('The Subnet Name')
param subnetName string = 'snet-hosted-agents'

//
// Azure Verified Modules

module createResourceGroup 'br/public:avm/res/resources/resource-group:0.4.0' = {
  name: 'createResourceGroup'
  params: {
    name: resourceGroupName
    location: location
  }
}

module createNetworkSecurityGroup 'br/public:avm/res/network/network-security-group:0.5.0' = {
  name: 'createNetworkSecurityGroup'
  scope: resourceGroup(resourceGroupName)
  params: {
    name: networkSecurityGroupName
    location: location
    securityRules: [
      {
        name: 'ALLOW_SSH_INBOUND_TCP'
        properties: {
          priority: 100
          access: 'Allow'
          direction: 'Inbound'
          protocol: 'Tcp'
          sourceAddressPrefix: publicIp
          sourcePortRange: '*'
          destinationAddressPrefix: '*'
          destinationPortRange: '22'
        }
      }
    ]
  }
  dependsOn: [
    createResourceGroup
  ]
}

module createVirtualNetwork 'br/public:avm/res/network/virtual-network:0.5.1' = {
  name: 'create-virtual-network'
  scope: resourceGroup(resourceGroupName)
  params: {
    name: virtualNetworkName
    location: location
    addressPrefixes: [
      '10.0.0.0/24'
    ]
    subnets: [
      {
        name: subnetName
        addressPrefix: '10.0.0.0/24'
        networkSecurityGroupResourceId: createNetworkSecurityGroup.outputs.resourceId
      }
    ]
  }
  dependsOn: [
    createResourceGroup
  ]
}

module createVirtualMachine 'br/public:avm/res/compute/virtual-machine:0.8.0' = {
  name: 'create-virtual-machine'
  scope: resourceGroup(resourceGroupName)
  params: {
    name: vmHostName
    adminUsername: vmUserName
    adminPassword: vmUserPassword
    location: location
    osType: 'Linux'
    vmSize: 'Standard_B2ms'
    zone: 0
    bootDiagnostics: true
    secureBootEnabled: true
    vTpmEnabled: true
    securityType: 'TrustedLaunch'
    imageReference: {
      publisher: 'Canonical'
      offer: 'ubuntu-24_04-lts'
      sku: 'server'
      version: 'latest'
    }
    nicConfigurations: [
      {
        ipConfigurations: [
          {
            name: 'ipconfig01'
            pipConfiguration: {
              name: '${vmHostName}-pip-01'
            }
            subnetResourceId: createVirtualNetwork.outputs.subnetResourceIds[0]
          }
        ]
        nicSuffix: '-nic-01'
        enableAcceleratedNetworking: false
      }
    ]
    osDisk: {
      caching: 'ReadWrite'
      diskSizeGB: 128
      managedDisk: {
        storageAccountType: 'Premium_LRS'
      }
    }
  }
  dependsOn: [
    createVirtualNetwork
  ]
}

Next you’ll need a deployment Wrapper, Save the PowerShell script for example as Invoke-AzDeployment.ps1, then calling the script:

1
.\Invoke-AzDeployment.ps1 -targetScope 'sub' -subscriptionId 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' -location 'westeurope' -environmentType dev -deploy

Invoke-AzDeployment Wrapper

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
param (
    [Parameter(Mandatory = $true, Position = 0, HelpMessage = "Deployment Guid is required")]
    [validateSet('tenant', 'mgmt', 'sub')] [string] $targetScope,

    [Parameter(Mandatory = $true, Position = 1, HelpMessage = "Subscription ID is required.")]
    [string] $subscriptionId,

    [Parameter(Mandatory = $true, Position = 2, HelpMessage = "Location is required.")]
    [ValidateSet(
        "eastus", "eastus2", "eastus3", "westus", "westus2", "westus3",
        "northcentralus", "southcentralus", "centralus",
        "canadacentral", "canadaeast", "brazilsouth",
        "northeurope", "westeurope", "uksouth", "ukwest",
        "francecentral", "francesouth", "germanywestcentral",
        "germanynorth", "switzerlandnorth", "switzerlandwest",
        "norwayeast", "norwaywest", "swedencentral", "swedensouth",
        "polandcentral", "qatarcentral", "uaenorth", "uaecentral",
        "southafricanorth", "southafricawest", "eastasia", "southeastasia",
        "japaneast", "japanwest", "australiaeast", "australiasoutheast",
        "australiacentral", "australiacentral2", "centralindia", "southindia",
        "westindia", "koreacentral", "koreasouth",
        "chinaeast", "chinanorth", "chinaeast2", "chinanorth2",
        "usgovvirginia", "usgovarizona", "usgovtexas", "usgoviowa"
    )][string]$location,

    [Parameter(Mandatory = $true, Position = 3, HelpMessage = "Environment Type is required.")]
    [validateSet('dev', 'acc', 'prod')] [string] $environmentType,

    [Parameter(Mandatory = $false, Position = 4, HelpMessage = "Enabled Bicep Deployment")]
    [switch] $deploy
)

# PowerShell Functions

# Function - New-RandomPassword
function New-RandomPassword {
    param (
        [Parameter(Mandatory)]
        [int] $length,
        [int] $amountOfNonAlphanumeric = 2
    )

    $nonAlphaNumericChars = '!@$'
    $nonAlphaNumericPart = -join ((Get-Random -Count $amountOfNonAlphanumeric -InputObject $nonAlphaNumericChars.ToCharArray()))

    $alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789'
    $alphabetPart = -join ((Get-Random -Count ($length - $amountOfNonAlphanumeric) -InputObject $alphabet.ToCharArray()))

    $password = ($alphabetPart + $nonAlphaNumericPart).ToCharArray() | Sort-Object { Get-Random }

    return -join $password
}

# Function - Get-BicepVersion
function Get-BicepVersion {

    #
    Write-Output `r "Checking for Bicep CLI..."

    # Get the installed version of Bicep
    $installedVersion = az bicep version --only-show-errors | Select-String -Pattern 'Bicep CLI version (\d+\.\d+\.\d+)' | ForEach-Object { $_.Matches.Groups[1].Value }

    if (-not $installedVersion) {
        Write-Output "Bicep CLI is not installed or version couldn't be determined."
        return
    }

    Write-Output "Installed Bicep version: $installedVersion"

    # Get the latest release version from GitHub
    $latestRelease = Invoke-RestMethod -Uri "https://api.github.com/repos/Azure/bicep/releases/latest"

    if (-not $latestRelease) {
        Write-Output "Unable to fetch the latest release."
        return
    }

    $latestVersion = $latestRelease.tag_name.TrimStart('v')  # GitHub version starts with 'v'

    # Compare versions
    if ($installedVersion -eq $latestVersion) {
        Write-Output "Bicep is up to date." `r
    }
    else {
        Write-Output "A new version of Bicep is available. Latest Release is: $latestVersion."
        # Prompt for user input (Yes/No)
        $response = Read-Host "Do you want to update? (Y/N)"

        if ($response -match '^[Yy]$') {
            Write-Output "" # Required for Verbose Spacing
            az bicep upgrade
            Write-Output "Bicep has been updated to version $latestVersion."
        }
        elseif ($response -match '^[Nn]$') {
            Write-Output "Update canceled."
        }
        else {
            Write-Output "Invalid response. Please answer with Y or N."
        }
    }
}

# PowerShell Location Shortcode Map
$LocationShortcodeMap = @{
    "eastus"             = "eus"
    "eastus2"            = "eus2"
    "eastus3"            = "eus3"
    "westus"             = "wus"
    "westus2"            = "wus2"
    "westus3"            = "wus3"
    "northcentralus"     = "ncus"
    "southcentralus"     = "scus"
    "centralus"          = "cus"
    "canadacentral"      = "canc"
    "canadaeast"         = "cane"
    "brazilsouth"        = "brs"
    "northeurope"        = "neu"
    "westeurope"         = "weu"
    "uksouth"            = "uks"
    "ukwest"             = "ukw"
    "francecentral"      = "frc"
    "francesouth"        = "frs"
    "germanywestcentral" = "gwc"
    "germanynorth"       = "gn"
    "switzerlandnorth"   = "chn"
    "switzerlandwest"    = "chw"
    "norwayeast"         = "noe"
    "norwaywest"         = "now"
    "swedencentral"      = "sec"
    "swedensouth"        = "ses"
    "polandcentral"      = "plc"
    "qatarcentral"       = "qtc"
    "uaenorth"           = "uan"
    "uaecentral"         = "uac"
    "southafricanorth"   = "san"
    "southafricawest"    = "saw"
    "eastasia"           = "ea"
    "southeastasia"      = "sea"
    "japaneast"          = "jpe"
    "japanwest"          = "jpw"
    "australiaeast"      = "aue"
    "australiasoutheast" = "ause"
    "australiacentral"   = "auc"
    "australiacentral2"  = "auc2"
    "centralindia"       = "cin"
    "southindia"         = "sin"
    "westindia"          = "win"
    "koreacentral"       = "korc"
    "koreasouth"         = "kors"
    "chinaeast"          = "ce"
    "chinanorth"         = "cn"
    "chinaeast2"         = "ce2"
    "chinanorth2"        = "cn2"
    "usgovvirginia"      = "usgv"
    "usgovarizona"       = "usga"
    "usgovtexas"         = "usgt"
    "usgoviowa"          = "usgi"
}

$shortcode = $LocationShortcodeMap[$location]

# Create Deployment Guid for Tracking in Azure
$deployGuid = (New-Guid).Guid

# Get User Public IP Address
$publicIp = (Invoke-RestMethod -Uri 'https://ifconfig.me')

# Virtual Machine Credentials
$vmUserName = 'azurevmuser'
$vmUserPassword = New-RandomPassword -length 16

# Check Azure Bicep Version
Get-BicepVersion

# Azure CLI Authentication
Write-Output "> Logging into Azure for $subscriptionId"
az login --output none --only-show-errors

# Configure Azure Cli User Experience
az config set core.login_experience_v2=off --only-show-errors

Write-Output "> Setting subscription to $subscriptionId"
az account set --subscription $subscriptionId

Write-Output `r "Pre Flight Variable Validation"
Write-Output "Bicep File............: $bicepFile"
Write-Output "Deployment Guid......: $deployGuid"
Write-Output "Location.............: $location"
Write-Output "Location Shortcode...: $shortcode"

if ($deploy) {
    $deployStartTime = Get-Date -Format 'HH:mm:ss'

    # Deploy Bicep Template
    $azDeployGuidLink = "`e]8;;https://portal.azure.com/#view/HubsExtension/DeploymentDetailsBlade/~/overview/id/%2Fsubscriptions%2F$subscriptionId%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fiac-$deployGuid`e\iac-$deployGuid`e]8;;`e\"
    Write-Output `r "> Deployment [$azDeployGuidLink] Started at $deployStartTime"

    az deployment sub create `
        --name iac-$deployGuid `
        --location $location `
        --template-file .\main.bicep `
        --parameters `
        location=$location `
        locationShortCode=$shortcode `
        environmentType=$environmentType `
        publicIp=$publicIp `
        vmUserName=$vmUserName `
        vmUserPassword=$vmUserPassword `
        --confirm-with-what-if `
        --output none

    $deployEndTime = Get-Date -Format 'HH:mm:ss'
    $timeDifference = New-TimeSpan -Start $deployStartTime -End $deployEndTime ; $deploymentDuration = "{0:hh\:mm\:ss}" -f $timeDifference
    Write-Output `r "> Deployment [iac-$deployGuid] Started at $deployEndTime - Deployment Duration: $deploymentDuration"

    Write-Output `r "Credentials"
    Write-Output "VM Username: $vmUserName"
    Write-Output "VM Password: $vmUserPassword"
}

Installing the Agent

Once you have you’re machine deployed you want to head the Actions settings either under the a Repository or Organisation

From here, Click New runner

The Bicep in this post, will deploy a Linux Ubuntu 24.04 LTS Image!

Pick the Operating System flavour and download the actions runner agent

Once the agent has been installed, We can look at starting the runner

Starting the Agent

When setting up your self-hosted Azure DevOps agent, you can start it in one of three modes:

Manual
Single Run
Service

Each method has its use cases depending on your needs.

Manual Run This mode is useful when you want to start the agent manually in a terminal session. The agent will continue to run as long as the terminal session remains active. To start the agent manually, use the following command:

1
./run.sh

Single Run If you want to run the agent for a single job and automatically exit once the job completes, use the –once flag. This is typically used when you need to trigger the agent for a specific task without leaving it running indefinitely. To execute the agent in single-run mode, use:

1
./run.sh --once

Service For persistent operation, you can install the agent as a service. This method ensures that the agent runs in the background and starts automatically with the system. It’s ideal for long-term setups where you want the agent to always be available.

To install the agent as a service, run:

1
sudo ./svc.sh install ; sudo ./svc.sh start ; sudo ./svc.sh status

This setup will ensure the agent runs as a service and is managed by your system’s init system (e.g., systemd on Linux), providing greater reliability for ongoing automation tasks.

Once the agent service has been brough online, You should see the agent online and happy in the portal!

Using Self-Hosted Agents in Pipelines

By default, GitHub Self-Hosted agents are only available for Private Repositories. To use them with Public Repositories as well, you need to update the default agent group configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
name: Test Agent Deployment

on:
  push:
    branches:
      - main

jobs:
  test-agent-deployment:
    runs-on: self-hosted
    steps:
      - name: Test Agent Deployment
        run: |
          echo "Hello, This executed! $(date)"
          echo "Agent Pool: Self Hosted Agents"
          echo "Agent Host: ${{ runner.name }}"

Wrap-Up

In this post, we’ve explored how to set up and manage GitHub Actions self-hosted runners, focusing on installation, configuration, and best practices. Using a self-hosted runner can significantly improve the speed and customization of your CI/CD pipelines by leveraging your own infrastructure. Additionally, by running the runner in service mode, you ensure that it remains available and resilient, ready to handle GitHub Actions jobs without requiring constant attention.

Key takeaways include

The steps to download, configure, and register a self-hosted GitHub Actions runner for both GitHub and GitHub Enterprise accounts.
How to leverage service mode for continuous availability and automatic recovery after system reboots.
Benefits of running the runner as a service, such as persistent uptime, reduced manual intervention, and enhanced reliability for production environments.
Tips for monitoring and maintaining the runner, ensuring your workflows run smoothly and without interruption.

By taking these steps, you ensure your self-hosted runner becomes a reliable, scalable part of your DevOps toolchain, helping you streamline your CI/CD processes with minimal overhead. If you’re working with Azure or other cloud platforms, these same principles can be applied to ensure your automation processes are efficient and always ready to run.

We hope this guide helps you set up your self-hosted GitHub Actions runner with ease and confidence. Happy automating!