Swap

Author: bobbyiliev

README.md

@@ -188,6 +188,7 @@ No providers.
 | <a name="module_networking"></a> [networking](#module\_networking) | ./modules/networking | n/a |
 | <a name="module_operator"></a> [operator](#module\_operator) | github.com/MaterializeInc/terraform-helm-materialize | v0.1.31 |
 | <a name="module_storage"></a> [storage](#module\_storage) | ./modules/storage | n/a |
+| <a name="module_swap_nodepool"></a> [swap\_nodepool](#module\_swap\_nodepool) | ./modules/nodepool | n/a |
 
 ## Resources
 
@@ -202,8 +203,8 @@ No resources.
 | <a name="input_cert_manager_install_timeout"></a> [cert\_manager\_install\_timeout](#input\_cert\_manager\_install\_timeout) | Timeout for installing the cert-manager helm chart, in seconds. | `number` | `300` | no |
 | <a name="input_cert_manager_namespace"></a> [cert\_manager\_namespace](#input\_cert\_manager\_namespace) | The name of the namespace in which cert-manager is or will be installed. | `string` | `"cert-manager"` | no |
 | <a name="input_database_config"></a> [database\_config](#input\_database\_config) | Azure Database for PostgreSQL configuration | <pre>object({<br/>    sku_name         = optional(string, "GP_Standard_D2s_v3")<br/>    postgres_version = optional(string, "15")<br/>    password         = string<br/>    username         = optional(string, "materialize")<br/>    db_name          = optional(string, "materialize")<br/>  })</pre> | n/a | yes |
-| <a name="input_disk_setup_image"></a> [disk\_setup\_image](#input\_disk\_setup\_image) | Docker image for the disk setup script | `string` | `"materialize/ephemeral-storage-setup-image:v0.3.4"` | no |
-| <a name="input_disk_support_config"></a> [disk\_support\_config](#input\_disk\_support\_config) | Advanced configuration for disk support (only used when enable\_disk\_support = true) | <pre>object({<br/>    install_openebs       = optional(bool, true)<br/>    run_disk_setup_script = optional(bool, true)<br/>    create_storage_class  = optional(bool, true)<br/>    openebs_version       = optional(string, "4.2.0")<br/>    openebs_namespace     = optional(string, "openebs")<br/>    storage_class_name    = optional(string, "openebs-lvm-instance-store-ext4")<br/>  })</pre> | `{}` | no |
+| <a name="input_disk_setup_image"></a> [disk\_setup\_image](#input\_disk\_setup\_image) | Docker image for the disk setup script | `string` | `"materialize/ephemeral-storage-setup-image:v0.4.0"` | no |
+| <a name="input_disk_support_config"></a> [disk\_support\_config](#input\_disk\_support\_config) | Advanced configuration for disk support (only used when enable\_disk\_support = true) | <pre>object({<br/>    install_openebs       = optional(bool, true)<br/>    run_disk_setup_script = optional(bool, true)<br/>    create_storage_class  = optional(bool, true)<br/>    openebs_version       = optional(string, "4.3.3")<br/>    openebs_namespace     = optional(string, "openebs")<br/>    storage_class_name    = optional(string, "openebs-lvm-instance-store-ext4")<br/>  })</pre> | `{}` | no |
 | <a name="input_enable_disk_support"></a> [enable\_disk\_support](#input\_enable\_disk\_support) | Enable disk support for Materialize using OpenEBS and local SSDs. When enabled, this configures OpenEBS, runs the disk setup script, and creates appropriate storage classes. | `bool` | `true` | no |
 | <a name="input_helm_chart"></a> [helm\_chart](#input\_helm\_chart) | Chart name from repository or local path to chart. For local charts, set the path to the chart directory. | `string` | `"materialize-operator"` | no |
 | <a name="input_helm_values"></a> [helm\_values](#input\_helm\_values) | Additional Helm values to merge with defaults | `any` | `{}` | no |
@@ -218,6 +219,7 @@ No resources.
 | <a name="input_orchestratord_version"></a> [orchestratord\_version](#input\_orchestratord\_version) | Version of the Materialize orchestrator to install | `string` | `null` | no |
 | <a name="input_prefix"></a> [prefix](#input\_prefix) | Prefix to be used for resource names | `string` | `"materialize"` | no |
 | <a name="input_resource_group_name"></a> [resource\_group\_name](#input\_resource\_group\_name) | The name of an existing resource group to use | `string` | n/a | yes |
+| <a name="input_swap_enabled"></a> [swap\_enabled](#input\_swap\_enabled) | Enable swap for Materialize. When enabled, this configures swap on a new nodepool, and adds it to the clusterd node selectors. | `bool` | `false` | no |
 | <a name="input_tags"></a> [tags](#input\_tags) | Tags to apply to all resources | `map(string)` | `{}` | no |
 | <a name="input_use_local_chart"></a> [use\_local\_chart](#input\_use\_local\_chart) | Whether to use a local chart instead of one from a repository | `bool` | `false` | no |
 | <a name="input_use_self_signed_cluster_issuer"></a> [use\_self\_signed\_cluster\_issuer](#input\_use\_self\_signed\_cluster\_issuer) | Whether to install and use a self-signed ClusterIssuer for TLS. To work around limitations in Terraform, this will be treated as `false` if no materialize instances are defined. | `bool` | `true` | no |

examples/simple/main.tf

@@ -91,6 +91,8 @@ module "materialize" {
 
   materialize_instances = var.materialize_instances
 
+  swap_enabled = var.swap_enabled
+
   database_config = {
     sku_name = "GP_Standard_D2s_v3"
     version  = "15"
@@ -188,6 +190,12 @@ variable "use_self_signed_cluster_issuer" {
   default     = true
 }
 
+variable "swap_enabled" {
+  description = "Enable swap for Materialize. When enabled, this configures swap on a new nodepool, and adds it to the clusterd node selectors."
+  type        = bool
+  default     = false
+}
+
 # Output the Materialize instance details
 output "aks_cluster" {
   description = "AKS cluster details"

main.tf

@@ -62,6 +62,31 @@ module "aks" {
   tags = local.common_labels
 }
 
+module "swap_nodepool" {
+  count      = var.swap_enabled ? 1 : 0
+  source     = "./modules/nodepool"
+  depends_on = [module.aks]
+
+  prefix     = "${var.prefix}-mz-swap"
+  cluster_id = module.aks.cluster_id
+  subnet_id  = module.networking.aks_subnet_id
+
+  vm_size      = var.aks_config.vm_size
+  disk_size_gb = var.aks_config.disk_size_gb
+
+  autoscaling_config = {
+    enabled   = true
+    min_nodes = var.aks_config.min_nodes
+    max_nodes = var.aks_config.max_nodes
+  }
+
+  swap_enabled     = true
+  disk_setup_image = var.disk_setup_image
+
+  labels = local.common_labels
+  tags   = local.common_labels
+}
+
 module "database" {
   source = "./modules/database"
 
@@ -123,7 +148,7 @@ locals {
         region = var.location
       }
       clusters = {
-        swap_enabled = false
+        swap_enabled = var.swap_enabled
       }
     }
     observability = {
@@ -227,6 +252,7 @@ module "operator" {
 
   depends_on = [
     module.aks,
+    module.swap_nodepool,
     module.database,
     module.storage,
     module.certificates,

modules/aks/main.tf

@@ -1,5 +1,6 @@
 locals {
-  nodepool_name = substr(replace(var.prefix, "-", ""), 0, 12)
+  nodepool_name   = substr(replace(var.prefix, "-", ""), 0, 12)
+  disk_setup_name = "disk-setup-scratchfs"
 }
 
 resource "azurerm_user_assigned_identity" "aks_identity" {
@@ -84,6 +85,12 @@ resource "azurerm_kubernetes_cluster_node_pool" "materialize" {
     "materialize.cloud/disk-config-required" = var.enable_disk_setup ? "true" : "false"
   }
 
+  upgrade_settings {
+    max_surge                     = "10%"
+    drain_timeout_in_minutes      = 0
+    node_soak_duration_in_minutes = 0
+  }
+
   # Taints can not be removed: https://github.com/Azure/AKS/issues/2934
   # node_taints = var.enable_disk_setup ? ["materialize.cloud/disk-unconfigured=true:NoSchedule"] : []
 
@@ -182,7 +189,7 @@ resource "kubernetes_namespace" "disk_setup" {
   count = var.enable_disk_setup ? 1 : 0
 
   metadata {
-    name = "disk-setup"
+    name = local.disk_setup_name
     labels = {
       "app.kubernetes.io/managed-by" = "terraform"
       "app.kubernetes.io/part-of"    = "materialize"
@@ -198,26 +205,26 @@ resource "kubernetes_daemonset" "disk_setup" {
   count = var.enable_disk_setup ? 1 : 0
 
   metadata {
-    name      = "disk-setup"
+    name      = local.disk_setup_name
     namespace = kubernetes_namespace.disk_setup[0].metadata[0].name
     labels = {
       "app.kubernetes.io/managed-by" = "terraform"
       "app.kubernetes.io/part-of"    = "materialize"
-      "app"                          = "disk-setup"
+      "app"                          = local.disk_setup_name
     }
   }
 
   spec {
     selector {
       match_labels = {
-        app = "disk-setup"
+        app = local.disk_setup_name
       }
     }
 
     template {
       metadata {
         labels = {
-          app = "disk-setup"
+          app = local.disk_setup_name
         }
       }
 
@@ -236,7 +243,7 @@ resource "kubernetes_daemonset" "disk_setup" {
             required_during_scheduling_ignored_during_execution {
               node_selector_term {
                 match_expressions {
-                  key      = "materialize.cloud/disk"
+                  key      = "materialize.cloud/scratch-fs"
                   operator = "In"
                   values   = ["true"]
                 }
@@ -346,15 +353,15 @@ resource "kubernetes_daemonset" "disk_setup" {
 resource "kubernetes_service_account" "disk_setup" {
   count = var.enable_disk_setup ? 1 : 0
   metadata {
-    name      = "disk-setup"
+    name      = local.disk_setup_name
     namespace = kubernetes_namespace.disk_setup[0].metadata[0].name
   }
 }
 
 resource "kubernetes_cluster_role" "disk_setup" {
   count = var.enable_disk_setup ? 1 : 0
   metadata {
-    name = "disk-setup"
+    name = local.disk_setup_name
   }
   rule {
     api_groups = [""]
@@ -366,7 +373,7 @@ resource "kubernetes_cluster_role" "disk_setup" {
 resource "kubernetes_cluster_role_binding" "disk_setup" {
   count = var.enable_disk_setup ? 1 : 0
   metadata {
-    name = "disk-setup"
+    name = local.disk_setup_name
   }
   role_ref {
     api_group = "rbac.authorization.k8s.io"

modules/aks/outputs.tf

@@ -3,6 +3,11 @@ output "cluster_name" {
   value       = azurerm_kubernetes_cluster.aks.name
 }
 
+output "cluster_id" {
+  description = "The ID of the AKS cluster"
+  value       = azurerm_kubernetes_cluster.aks.id
+}
+
 output "cluster_endpoint" {
   description = "The endpoint of the AKS cluster"
   value       = azurerm_kubernetes_cluster.aks.kube_config[0].host

modules/aks/variables.tf

@@ -88,5 +88,5 @@ variable "openebs_version" {
 variable "disk_setup_image" {
   description = "Docker image for the disk setup script"
   type        = string
-  default     = "materialize/ephemeral-storage-setup-image:v0.1.2"
+  default     = "materialize/ephemeral-storage-setup-image:v0.4.0"
 }

modules/nodepool/README.md

@@ -0,0 +1,57 @@
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
+| <a name="requirement_azurerm"></a> [azurerm](#requirement\_azurerm) | ~> 4.0 |
+| <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | >= 2.10.0 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_azurerm"></a> [azurerm](#provider\_azurerm) | ~> 4.0 |
+| <a name="provider_kubernetes"></a> [kubernetes](#provider\_kubernetes) | >= 2.10.0 |
+
+## Modules
+
+No modules.
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [azurerm_kubernetes_cluster_node_pool.primary_nodes](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster_node_pool) | resource |
+| [kubernetes_cluster_role.disk_setup](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/cluster_role) | resource |
+| [kubernetes_cluster_role_binding.disk_setup](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/cluster_role_binding) | resource |
+| [kubernetes_daemonset.disk_setup](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/daemonset) | resource |
+| [kubernetes_namespace.disk_setup](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace) | resource |
+| [kubernetes_service_account.disk_setup](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/service_account) | resource |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_autoscaling_config"></a> [autoscaling\_config](#input\_autoscaling\_config) | Auto-scaling configuration for the node pool | <pre>object({<br/>    enabled    = bool<br/>    min_nodes  = optional(number)<br/>    max_nodes  = optional(number)<br/>    node_count = optional(number)<br/>  })</pre> | <pre>{<br/>  "enabled": true,<br/>  "max_nodes": 10,<br/>  "min_nodes": 1,<br/>  "node_count": null<br/>}</pre> | no |
+| <a name="input_cluster_id"></a> [cluster\_id](#input\_cluster\_id) | The ID of the AKS cluster | `string` | n/a | yes |
+| <a name="input_disk_setup_container_resource_config"></a> [disk\_setup\_container\_resource\_config](#input\_disk\_setup\_container\_resource\_config) | Resource configuration for disk setup init container | <pre>object({<br/>    memory_limit   = string<br/>    memory_request = string<br/>    cpu_request    = string<br/>  })</pre> | <pre>{<br/>  "cpu_request": "50m",<br/>  "memory_limit": "128Mi",<br/>  "memory_request": "128Mi"<br/>}</pre> | no |
+| <a name="input_disk_setup_image"></a> [disk\_setup\_image](#input\_disk\_setup\_image) | Docker image for the disk setup script | `string` | `"materialize/ephemeral-storage-setup-image:v0.4.0"` | no |
+| <a name="input_disk_size_gb"></a> [disk\_size\_gb](#input\_disk\_size\_gb) | Size of the disk attached to each node | `number` | n/a | yes |
+| <a name="input_labels"></a> [labels](#input\_labels) | Additional labels to apply to Kubernetes resources | `map(string)` | `{}` | no |
+| <a name="input_node_taints"></a> [node\_taints](#input\_node\_taints) | Taints to apply to the node pool. Note: Once applied via Terraform, these taints cannot be manually removed by users due to AKS webhook restrictions. | <pre>list(object({<br/>    key    = string<br/>    value  = string<br/>    effect = string<br/>  }))</pre> | `[]` | no |
+| <a name="input_pause_container_resource_config"></a> [pause\_container\_resource\_config](#input\_pause\_container\_resource\_config) | Resource configuration for pause container | <pre>object({<br/>    memory_limit   = string<br/>    memory_request = string<br/>    cpu_request    = string<br/>  })</pre> | <pre>{<br/>  "cpu_request": "1m",<br/>  "memory_limit": "8Mi",<br/>  "memory_request": "8Mi"<br/>}</pre> | no |
+| <a name="input_prefix"></a> [prefix](#input\_prefix) | Prefix to be used for resource names | `string` | n/a | yes |
+| <a name="input_subnet_id"></a> [subnet\_id](#input\_subnet\_id) | The ID of the subnet | `string` | n/a | yes |
+| <a name="input_swap_enabled"></a> [swap\_enabled](#input\_swap\_enabled) | Whether to enable swap on the local NVMe disks. | `bool` | `false` | no |
+| <a name="input_tags"></a> [tags](#input\_tags) | Tags to apply to resources | `map(string)` | `{}` | no |
+| <a name="input_vm_size"></a> [vm\_size](#input\_vm\_size) | VM size for AKS nodes | `string` | n/a | yes |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| <a name="output_nodepool_id"></a> [nodepool\_id](#output\_nodepool\_id) | The ID of the node pool |
+| <a name="output_nodepool_max_count"></a> [nodepool\_max\_count](#output\_nodepool\_max\_count) | The maximum count of nodes in the node pool |
+| <a name="output_nodepool_min_count"></a> [nodepool\_min\_count](#output\_nodepool\_min\_count) | The minimum count of nodes in the node pool |
+| <a name="output_nodepool_name"></a> [nodepool\_name](#output\_nodepool\_name) | The name of the node pool |
+| <a name="output_nodepool_vm_size"></a> [nodepool\_vm\_size](#output\_nodepool\_vm\_size) | The VM size of the node pool |