Release - Lucia Training Platform v1.4 (#131)

**Description**

Merge bug fixes from v1.4 to dev branch.

**Major Revisions**
* ModelProxy: fix uncommit changes (#122) 
* Fix cicd errors when branch creation and building all imges (#123) 
* Bugfix - fix bug of image regex deployment (#125) 
* Bugfix - fix bugs in alert manager test and fix kusto alert query issue (#126)
* Fix openpai runtime build on arm64 (#128)
* Fix fluentd build on arm64 (#129)
* Bugfix - bug fix for baremetal support(#127)
* Doc - Add Release note for v1.4.0 (#130)

---------

Co-authored-by: zhogu <[email protected]>
Co-authored-by: Yifan Xiong <[email protected]>

Author: 3 people

.github/workflows/build-all.yaml

@@ -5,7 +5,9 @@ permissions:
 
 on:
   push:
-    branches: ['release/*']
+    branches: ["release/*"]
+  pull_request:
+    branches: ["release/*"]
   release:
     types: [published]
   workflow_dispatch:
@@ -56,8 +58,9 @@ jobs:
       - name: Install Package
         if: steps.all.outputs.services != ''
         run: |
-          DEBIAN_FRONTEND=noninteractive apt install -y python3 python-is-python3 pip git unzip docker-cli ca-certificates curl apt-transport-https lsb-release gnupg parallel
+          DEBIAN_FRONTEND=noninteractive apt install -y python3 python-is-python3 pip git unzip ca-certificates curl apt-transport-https lsb-release gnupg parallel
           curl -sL https://aka.ms/InstallAzureCLIDeb | bash
+          curl -fsSL https://get.docker.com | sh
 
       - name: Install python libs
         if: steps.all.outputs.services != ''
@@ -78,6 +81,9 @@ jobs:
           mv $GITHUB_WORKSPACE/config/auth-configuration /tmp/
           ls -l /tmp/auth-configuration
 
+      - name: Log in to GHCR
+        run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+
       - name: Build Images of Services
         if: steps.all.outputs.services != ''
         run: |
@@ -86,7 +92,7 @@ jobs:
           echo "--------------------------------"
           failed_services=""
           for service in $all_services; do
-            if [[ "$service" =~ alert-manager ]]; then
+            if echo "$service" | grep -q "alert-manager"; then
               echo "alert-manager is in the changed services"
               # Build specific images in alert-manager
               echo "Building specific alert-manager images"

.github/workflows/build-deploy-changes.yaml

@@ -49,7 +49,12 @@ jobs:
             base_sha=$(git merge-base origin/${{ github.event.pull_request.base.ref }} ${{ github.event.pull_request.head.sha }})
             head_sha="${{ github.event.pull_request.head.sha }}"
           else
-            base_sha="${{ github.event.before }}"
+            if [ "${{ github.event.before }}" = "0000000000000000000000000000000000000000" ]; then
+              # Get the previous commit on branch
+              base_sha=$(git rev-parse ${{ github.sha }}^)
+            else
+              base_sha="${{ github.event.before }}"
+            fi
             head_sha="${{ github.sha }}"
           fi
 

docs/LuciaTrainingPlatform/blog/2025-11-28-release-1-4.md

@@ -0,0 +1,43 @@
+---
+slug: release-ltp-v1.4
+title: Releasing Lucia Training Platform v1.4
+author: Lucia Training Platform Team
+tags: [ltp, announcement, release]
+---
+
+We are pleased to announce the official release of **Lucia Training Platform v1.4.0**!
+
+## Lucia Training Platform v1.4.0 Release Notes
+
+This release brings significant improvements across platform stability, inference capabilities, CI/CD automation, security enhancements, and user experience.
+
+## Platform Features & Stability
+- Fixed Blobfuse2 installation failure
+- Fixed slow portal response issues
+- Fixed deployment issues on arm64 architecture and bare metal machines
+- Added image regex to support valid image limitation
+- Added webportal plugin for cluster local storage
+- Added tolerance and priority class in cluster-local-storage
+- Created PostgreSQL backend and support backend switch with Kusto
+
+## Inference Plugin
+- Inference plugin support for model proxy
+- Added support for training/inference job type in restserver
+- Added support for training/inference job type in webportal
+- Support inference job type in model proxy
+
+## CI/CD
+- Added CI/CD workflow to build all service images
+- Enabled imagelist argument for image build script
+- Fixed copy error in building image process
+- Fix cicd errors when branch creation and building all images
+
+## Security
+- Upgraded Dockerfiles with latest system updates
+- Fixed credentials exposure in source code
+- Updated Cilium to latest version
+
+## User Experience
+- Added comprehensive user manual documentation
+- Improved Copilot test experience
+

src/alert-manager/build/alertmanager.common.dockerfile

@@ -0,0 +1,4 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+
+FROM prom/alertmanager:v0.29.0

src/alert-manager/deploy/alert-manager-deployment.yaml.template

@@ -42,6 +42,7 @@ spec:
       containers:
       - name: alertmanager
         image: {{ cluster_cfg['cluster']['docker-registry']['prefix'] }}alertmanager:{{ cluster_cfg['cluster']['docker-registry']['tag'] }}
+        imagePullPolicy: Always
         args:
           - '--config.file=/etc/alertmanager/config.yml'
           - '--storage.path=/alertmanager'
@@ -196,8 +197,8 @@ spec:
           value: {{ cluster_cfg["alert-manager"]["alert-parser"]["log-level"] | default(cluster_cfg["alert-manager"]["log-level"]) | default("INFO") }}
         - name: UPDATE_INTERVAL
           value: "{{ cluster_cfg["alert-manager"]["alert-parser"]["update-interval"] | default("10") }}"
-        - name: REST_SERVER_URI
-          value: {{ cluster_cfg["alert-manager"]["alert-parser"]["uri"] | default(cluster_cfg["alert-manager"]["prometheus-uri"]) }}
+        - name: PROMETHEUS_SERVER_URI
+          value: {{ cluster_cfg["prometheus"]["url"] }}
         - name: PAI_TOKEN
           value: {{ cluster_cfg["alert-manager"]["pai-bearer-token"] }}
 {% if cluster_cfg["alert-manager"]["alert-parser"].get("storage-backend", cluster_cfg["alert-manager"].get("storage-backend", "kusto")) == "kusto" %}
@@ -217,11 +218,11 @@ spec:
           value: "{{ cluster_cfg['alert-manager']['alert-parser'].get('alert-kusto-database', 'DefaultWorkspace-id-westus2') }}"
         - name: LTP_KUSTO_VM_INFO_TABLE_NAME
           value: {{ cluster_cfg["alert-manager"]["alert-parser"].get("kusto-vm-table", cluster_cfg["alert-manager"]["kusto"].get("kusto-vm-table", "")) }}
-{% endif %}
         - name: LTP_VMSS_IDS
           value: {{ cluster_cfg["alert-manager"]["alert-parser"]["vmss_ids"] | default(cluster_cfg["alert-manager"]["vmss_ids"]) }}
         - name: AZURE_CLIENT_ID
           value: {{ cluster_cfg["alert-manager"]["alert-parser"]["vmss_client_id"] | default(cluster_cfg["alert-manager"]["vmss_client_id"]) }}
+{% endif %}
         # Storage backend selection
         - name: LTP_STORAGE_BACKEND_DEFAULT
           value: "{{ cluster_cfg['alert-manager']['alert-parser'].get('storage-backend', cluster_cfg['alert-manager'].get('storage-backend', 'kusto')) }}"
@@ -332,7 +333,7 @@ spec:
         - name: LOG_LEVEL
           value: {{ cluster_cfg["alert-manager"]["node-failure-detection"]["monitor"]["log-level"] | default("INFO") }}
         - name: PROMETHEUS_SERVER_URI
-          value: {{ cluster_cfg["alert-manager"]["node-failure-detection"]["monitor"]["prometheus-server-uri"] }}
+          value: {{ cluster_cfg["prometheus"]["url"] }}
         - name: REST_SERVER_URI
           value: {{ cluster_cfg["rest-server"]["uri"] }}
         - name: PAI_TOKEN
@@ -405,7 +406,7 @@ spec:
         - name: REST_SERVER_URI
           value: {{ cluster_cfg["rest-server"]["uri"] }}
         - name: PROMETHEUS_SERVER_URI
-          value: {{ cluster_cfg["alert-manager"]["job-data-recorder"]["prometheus-uri"] | default(cluster_cfg["alert-manager"]["prometheus-uri"]) }}
+          value: {{ cluster_cfg["prometheus"]["url"] }}
         - name: PAI_TOKEN
           value: {{ cluster_cfg["alert-manager"]["pai-bearer-token"] }}
 {% if cluster_cfg["alert-manager"]["job-data-recorder"].get("storage-backend", cluster_cfg["alert-manager"].get("storage-backend", "kusto")) == "kusto" %}

src/alert-manager/deploy/redis-deployment.yaml.template

@@ -72,11 +72,13 @@ spec:
           periodSeconds: 10
           successThreshold: 1
           failureThreshold: 3
+      imagePullSecrets:
+      - name: {{ cluster_cfg["cluster"]["docker-registry"]["secret-name"] }}
       volumes:
       - name: redis-data
         hostPath:
           path: /data/redis-data
           type: ""
       - name: redis-config
         configMap:
-          name: redis-config
+          name: redis-config

src/alert-manager/src/alert-parser/node_alert_monitor.py

@@ -34,7 +34,7 @@ def __init__(self):
         """
         self.endpoint = os.getenv("CLUSTER_ID")
         self.update_interval = int(os.getenv("UPDATE_INTERVAL", 10))  # Default to 10 minutes
-        self.rest_server_uri = os.getenv("REST_SERVER_URI", "http://localhost:8080")
+        self.prometheus_uri = os.getenv("PROMETHEUS_SERVER_URI", "http://localhost:8080")
         self.is_running = False
         self.last_update_time = None
         self.tolerance_time = int(os.getenv("TOLERANCE_TIME", 300))
@@ -54,7 +54,7 @@ def query_availability_changes(self, end_time: int, time_offset: str, interval:
             'query?query=(avg_over_time(avg by (node_name) (pai_node_count{unschedulable="true",node_name!~"aks-.*"} or pai_node_count{unschedulable="false",node_name!~"aks-.*"}*0)'
             + f"[{time_offset}:{interval}] @ {end_time} )) >0")
 
-        result = RequestUtil.prometheus_query(query=query, data={}, uri=self.rest_server_uri)
+        result = RequestUtil.prometheus_query(query=query, data={}, uri=f"{self.prometheus_uri}/prometheus")
 
         if result is not None:
             result = result["result"]
@@ -68,7 +68,7 @@ def query_availability_changes(self, end_time: int, time_offset: str, interval:
 
         query = ('query?query=avg by (node_name) (pai_node_count{unschedulable="false",node_name!~"aks-.*"})'
                 + f"[{interval}:{interval}] @ {end_time}")
-        result = RequestUtil.prometheus_query(query=query, data={}, uri=self.rest_server_uri)
+        result = RequestUtil.prometheus_query(query=query, data={}, uri=f"{self.prometheus_uri}/prometheus")
         if result is not None:
             result = result["result"]
             for node_result in result:
@@ -84,7 +84,7 @@ def get_node_status_changes(self, node: str, end_time: int, time_offset: str, in
                 f'unschedulable="false",node_name="{node}"' + "}*0) " +
                 f"[{time_offset}:{interval}] @ {end_time}")
 
-        result = RequestUtil.prometheus_query(query=query, data={}, uri=self.rest_server_uri)
+        result = RequestUtil.prometheus_query(query=query, data={}, uri=f"{self.prometheus_uri}/prometheus")
         status_changes = {}
         raw_values = {}
 
@@ -279,7 +279,7 @@ def process_node_changes(self, node: str, changes: Dict[float, float], end_time:
                 return 
             # fetch alerts for the node
             alerts = self.alert_fetcher.get_node_alert_records(
-                end_time, f"{time_offset}s", endpoint=self.endpoint, nodes=[node]
+                end_time, f"{time_offset}s", nodes=[node], severity="error"
             )
 
             sorted_changes = sorted(changes.items(), key=lambda x: x[0])

src/alert-manager/src/alert-parser/tests/conftest.py

@@ -9,13 +9,7 @@
 def mock_env_vars():
     """Mock environment variables used in the tests"""
     env_vars = {
-        'CLUSTER_ID': 'test-cluster',
-        'LTP_KUSTO_CLUSTER_URI': 'test-kusto-cluster',
-        'LTP_KUSTO_DATABASE_NAME': 'test-db',
-        'KUSTO_NODE_STATUS_TABLE_NAME': 'NodeStatusRecord',
-        'KUSTO_NODE_STATUS_ATTRIBUTE_TABLE_NAME': 'NodeStatusAttributes',
-        'KUSTO_NODE_ACTION_TABLE_NAME': 'NodeActionRecord',
-        'KUSTO_NODE_ACTION_ATTRIBUTE_TABLE_NAME': 'NodeActionAttributes'
+        'CLUSTER_ID': 'test-cluster'
     }
     with pytest.MonkeyPatch.context() as m:
         for key, value in env_vars.items():