Merge pull request #21 from byu-oit/rc-4.0.0

feat!: Release 4.0.0 scheduler support, and better existing_ecs_cluster support

Author: yoshutch

.github/workflows/ci.yml

@@ -22,14 +22,14 @@ jobs:
 
       - name: Terraform Format
         working-directory: "./"
-        run: terraform fmt -check -recursive
+        run: terraform fmt -check -recursive -diff
 
   plan:
     name: Terraform Plan
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        terraform: [ '1.3.x' ]
+        terraform: [ '1.3.x', '1.4.x', '1.5.x' ]
       fail-fast: false
     permissions:
       contents: read

README.md

@@ -0,0 +1,72 @@
+# [short title of solved problem and solution]
+
+* Status: [proposed | rejected | accepted | deprecated | … | superseded by [ADR-0005](0005-example.md)] <!-- optional -->
+* Deciders: [list everyone involved in the decision] <!-- optional -->
+* Date: [YYYY-MM-DD when the decision was last updated] <!-- optional -->
+
+Technical Story: [description | ticket/issue URL] <!-- optional -->
+
+## Context and Problem Statement
+
+[Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question.]
+
+## Decision Drivers <!-- optional -->
+
+* [driver 1, e.g., a force, facing concern, …]
+* [driver 2, e.g., a force, facing concern, …]
+* … <!-- numbers of drivers can vary -->
+
+## Considered Options
+
+* [option 1]
+* [option 2]
+* [option 3]
+* … <!-- numbers of options can vary -->
+
+## Decision Outcome
+
+Chosen option: "[option 1]", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | … | comes out best (see below)].
+
+### Positive Consequences <!-- optional -->
+
+* [e.g., improvement of quality attribute satisfaction, follow-up decisions required, …]
+* …
+
+### Negative Consequences <!-- optional -->
+
+* [e.g., compromising quality attribute, follow-up decisions required, …]
+* …
+
+## Pros and Cons of the Options <!-- optional -->
+
+### [option 1]
+
+[example | description | pointer to more information | …] <!-- optional -->
+
+* Good, because [argument a]
+* Good, because [argument b]
+* Bad, because [argument c]
+* … <!-- numbers of pros and cons can vary -->
+
+### [option 2]
+
+[example | description | pointer to more information | …] <!-- optional -->
+
+* Good, because [argument a]
+* Good, because [argument b]
+* Bad, because [argument c]
+* … <!-- numbers of pros and cons can vary -->
+
+### [option 3]
+
+[example | description | pointer to more information | …] <!-- optional -->
+
+* Good, because [argument a]
+* Good, because [argument b]
+* Bad, because [argument c]
+* … <!-- numbers of pros and cons can vary -->
+
+## Links <!-- optional -->
+
+* [Link type] [Link to ADR] <!-- example: Refined by [ADR-0005](0005-example.md) -->
+* … <!-- numbers of links can vary -->

docs/adr-template.md

@@ -0,0 +1,80 @@
+# Use EventBridge Scheduler to Trigger Scheduled Fargate Tasks
+
+* Status: accepted
+* Deciders:
+  * Jamie Visker
+  * Josh Gubler
+  * Scott Hutchings
+  * Spencer Visker
+* Date: 2023-09-26
+
+## Context and Problem Statement
+
+We used CloudWatch Event Rules and Targets to trigger scheduled Fargate tasks.
+AWS has a newer service called [EventBridge Scheduler](https://docs.aws.amazon.com/eventbridge/latest/userguide/scheduler.html).
+Do we start using the Scheduler to take advantage of those newer features?
+
+## Decision Drivers <!-- optional -->
+
+* Would be nice if our tasks could support time zone scheduled jobs
+* Needs to be easy to use
+
+## Considered Options
+
+* Keep using Event Rules and Targets
+* Use EventBridge Scheduler (keep event trigger functionality)
+* Use EventBridge Scheduler (move event trigger functionality to new module)
+* Use EventBridge Scheduler and refactor fargate modules
+
+## Decision Outcome
+
+Chosen option: "Use EventBridge Scheduler (keep event trigger functionality)", because it'll give us the best of both worlds with using the new Scheduler and still supporting events.
+
+### Positive Consequences <!-- optional -->
+
+* We were able to implement in such a way that a fargate task can be both scheduled and triggered by an event with the same module.
+
+### Negative Consequences <!-- optional -->
+
+* Using the Scheduler removes the fargate task from the "Scheduled tasks" portion of the ECS cluster on the AWS console.
+
+## Pros and Cons of the Options <!-- optional -->
+
+### Keep using Event Rules and Targets
+
+* Good, because there'll be no change to our module
+* Good, because it still works
+* Bad, because we can't take advantage of the EventBridge Scheduler features (time zone, start-end dates, retries etc.)
+
+### Use EventBridge Scheduler (keep event trigger functionality)
+
+* Good, because we can take advantage of the EventBridge Scheduler features (time zone, start-end dates, retries etc.)
+* Good, because we can still support tasks triggered by tasks
+* Bad, because the module will be more complex (it will include resources for both EventBridge Scheduler, and CloudWatch Rules and Targets)
+* Bad, because using the Scheduler removes the task from the list of "Scheduled Tasks" in the ECS cluster
+
+### Use EventBridge Scheduler (move event trigger functionality to new module)
+
+* Good, because we can take advantage of the EventBridge Scheduler features (time zone, start-end dates, retries etc.)
+* Good, because this module will be simplified
+* Bad, because in order to support tasks triggered by tasks we'll need to create a new module (with lots of copy/paste terraform configuration)
+* Bad, because using the Scheduler removes the task from the list of "Scheduled Tasks" in the ECS cluster
+
+### Use EventBridge Scheduler and refactor fargate modules
+
+We can refactor the fargate-api, and scheduled-fargate modules into smaller pieces that work together in order to reduce duplicate terraform code.
+Maybe something like:
+* `fargate-definition` module that sets up the fargate task definition, ECS cluster, SGs, related IAM roles etc.
+* `fargate-service` module that takes task definition and cluster inputs from 'fargate-defintion' module, and creates an ECS service with ALB, CodeDeploy, related IAM roles etc.
+* `fargate-scheduled` module that takes task definition and cluster inputs from 'fargate-defintion' module, and creates Scheduler that runs the task on the schedule, related IAM roles etc.
+* `fargate-event` module that takes task definition and cluster inputs from 'fargate-defintion' module, and creates CloudWatch Event to trigger task, Rule(s), Target(s), and related IAM roles etc.
+
+* Good, because we can reduce duplicate terraform
+* Good, because our modules can be smaller and simpler, thus easier to maintain
+* Bad, because it completely breaks our exising terraform module contract, so upgrades will be more difficult
+* Bad, because it makes our apps a little more complex having to use interconnected modules instead of just one. It borders on not needing modules in the first place if our apps have to know how everything integrates.
+* Bad, because using the Scheduler removes the task from the list of "Scheduled Tasks" in the ECS cluster
+
+## Links <!-- optional -->
+
+* [EventBridge Scheduler](https://docs.aws.amazon.com/eventbridge/latest/userguide/scheduler.html)

docs/adr/000-use-scheduler.md

@@ -0,0 +1,67 @@
+# Migration to v4
+V4 has a significant amount of breaking changes from v3 and earlier:
+
+* How we use an existing ECS cluster allowing the creating of a cluster alongside using it as an existing cluster in the same terraform configuration.
+* This module now uses the [EventBridge Scheduler](https://docs.aws.amazon.com/eventbridge/latest/userguide/scheduler.html) instead of CloudWatch EventBridge Rules.
+
+## For scheduled tasks
+1. Change `schedule_expression` variable to be inside the `expression` block
+2. Remove `event_role_arn`
+3. Change `ecs_cluster` output to `new_ecs_cluster` output if you're using it
+   > **Note:** that the `new_ecs_cluster` will only be populated if you let the module create the cluster, if you provide your own existing cluster this output will be `null`
+
+```diff
+module "scheduled_fargate" {
+-  source = "github.com/byu-oit/terraform-aws-scheduled-fargate?ref=v3.0.1"
+  source = "github.com/byu-oit/terraform-aws-scheduled-fargate?ref=v4.0.0"
+  app_name = "test-scheduled-fargate-dev"
+-  schedule_expression = "rate(5 minutes)"
++  schedule = {
++    expression = "rate(5 minutes)"
++  }  
+  primary_container_definition = {
+    # ...
+  }
+-  event_role_arn                = module.acs.power_builder_role.arn
+  vpc_id                        = module.acs.vpc.id
+  private_subnet_ids            = module.acs.private_subnet_ids
+  role_permissions_boundary_arn = module.acs.role_permissions_boundary.arn
+}
+```
+
+## For event triggered tasks
+1. Change the `event_pattern` variable to be inside the `event` block
+2. Remove `event_role_arn`
+3. Change `ecs_cluster` output to `new_ecs_cluster` output if you're using it
+    > **Note:** that the `new_ecs_cluster` will only be populated if you let the module create the cluster, if you provide your own existing cluster this output will be `null`
+
+```diff
+module "scheduled_fargate" {
+-  source = "github.com/byu-oit/terraform-aws-scheduled-fargate?ref=v3.0.1"
++  source = "github.com/byu-oit/terraform-aws-scheduled-fargate?ref=v4.0.0"
+  app_name      = "event-triggered-fargate-example-dev"
+-  event_pattern = <<EOF
++  event = {
++    pattern = <<EOF
+  {
+    "source": ["aws.s3"],
+    "detail-type": ["Object Created"],
+    "detail": {
+      "bucket": {
+        "name": ["${aws_s3_bucket.test_bucket.bucket}"]
+      }
+    }
+  }
+EOF
++  }
+  primary_container_definition = {
+    # ...
+  }
+-  event_role_arn                = module.acs.power_builder_role.arn
+  vpc_id                        = module.acs.vpc.id
+  private_subnet_ids            = module.acs.private_subnet_ids
+  role_permissions_boundary_arn = module.acs.role_permissions_boundary.arn
+}
+```
+
+View the release notes for more detailed changes

docs/migration-to-v4.md

@@ -4,7 +4,7 @@ terraform {
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = "~> 3.69"
+      version = "~> 4.0"
     }
   }
 }
@@ -14,19 +14,20 @@ provider "aws" {
 }
 
 module "acs" {
-  source = "github.com/byu-oit/terraform-aws-acs-info?ref=v3.5.0"
+  source = "github.com/byu-oit/terraform-aws-acs-info?ref=v4.0.0"
 }
 
 module "scheduled_fargate" {
-  source = "../../"
-
-  app_name            = "test-scheduled-fargate-dev"
-  schedule_expression = "rate(5 minutes)"
+  #  source              = "github.com/byu-oit/terraform-aws-scheduled-fargate?ref=v4.0.0"
+  source   = "../../"
+  app_name = "test-scheduled-fargate-dev"
+  schedule = {
+    expression = "rate(5 minutes)"
+  }
   primary_container_definition = {
     name  = "test"
     image = "hello-world"
   }
-  event_role_arn                = module.acs.power_builder_role.arn
   vpc_id                        = module.acs.vpc.id
   private_subnet_ids            = module.acs.private_subnet_ids
   role_permissions_boundary_arn = module.acs.role_permissions_boundary.arn
@@ -36,9 +37,8 @@ module "scheduled_fargate" {
   }
 }
 
-
-output "scheduled_fargate_ecs_cluster" {
-  value = module.scheduled_fargate.ecs_cluster
+output "scheduled_fargate_new_ecs_cluster" {
+  value = module.scheduled_fargate.new_ecs_cluster
 }
 
 output "scheduled_fargate_security_group" {
@@ -49,12 +49,8 @@ output "scheduled_task_definition" {
   value = module.scheduled_fargate.task_definition
 }
 
-output "scheduled_event_rule" {
-  value = module.scheduled_fargate.event_rule
-}
-
-output "scheduled_event_target" {
-  value     = module.scheduled_fargate.event_target
+output "schedule" {
+  value     = module.scheduled_fargate.schedule
   sensitive = true
 }
 
@@ -71,3 +67,8 @@ output "task_role" {
   value     = module.scheduled_fargate.task_role
   sensitive = true
 }
+
+output "run_task_cli_command" {
+  value     = module.scheduled_fargate.run_task_cli_command
+  sensitive = true
+}

examples/ci/ci.tf

@@ -4,13 +4,18 @@ terraform {
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = "~> 3.69"
+      version = "~> 4.0"
     }
   }
 }
 
+provider "aws" {
+  region = "us-west-2"
+}
+
+
 module "acs" {
-  source = "github.com/byu-oit/terraform-aws-acs-info?ref=v3.5.0"
+  source = "github.com/byu-oit/terraform-aws-acs-info?ref=v4.0.0"
 }
 
 variable "env" {
@@ -31,6 +36,9 @@ locals {
   }
 }
 
+resource "aws_ecs_cluster" "existing" {
+  name = "test-existing-cluster"
+}
 resource "aws_ecr_repository" "repo" {
   name = "test-existing-ecr"
 }
@@ -39,12 +47,26 @@ output "repo_url" {
   value = aws_ecr_repository.repo.repository_url
 }
 
+resource "aws_scheduler_schedule_group" "group" {
+  name = "test"
+}
+
 // Scheduled fargate
 module "scheduled_fargate" {
-  source              = "github.com/byu-oit/terraform-aws-scheduled-fargate?ref=v3.0.1"
-  app_name            = local.name
-  ecs_cluster_name    = aws_ecr_repository.repo.name
-  schedule_expression = "rate(5 minutes)"
+  #  source              = "github.com/byu-oit/terraform-aws-scheduled-fargate?ref=v4.0.0"
+  source   = "../../"
+  app_name = local.name
+  existing_ecs_cluster = {
+    arn = aws_ecs_cluster.existing.arn
+  }
+  schedule = {
+    expression = "rate(5 minutes)"
+    timezone   = "UTC"
+    start_date = "2023-10-01T00:00:00.000Z"
+    end_date   = "2023-10-02T00:00:00.000Z"
+  }
+
+  log_group_name = aws_scheduler_schedule_group.group.name
   primary_container_definition = {
     name  = "test-dynamo"
     image = "${aws_ecr_repository.repo.repository_url}:${var.image_tag}"
@@ -61,7 +83,6 @@ module "scheduled_fargate" {
     ]
   }
   task_policies                 = [aws_iam_policy.my_dynamo_policy.arn]
-  event_role_arn                = module.acs.power_builder_role.arn
   vpc_id                        = module.acs.vpc.id
   private_subnet_ids            = module.acs.private_subnet_ids
   role_permissions_boundary_arn = module.acs.role_permissions_boundary.arn
@@ -127,7 +148,7 @@ resource "aws_security_group" "efs_sg" {
 }
 
 resource "aws_efs_mount_target" "efs_target" {
-  for_each = toset(module.acs.private_subnet_ids)
+  for_each = nonsensitive(toset(module.acs.private_subnet_ids))
 
   file_system_id  = aws_efs_file_system.my_efs.id
   subnet_id       = each.key