DNS resolution failures in Alpine-based docker container

[arturfromtabnine]

What Happened?

Portkey gateway running in Alpine-based docker container fails to resolve some DNS queries for external services, causing connectivity issue. Alpine based images are know for its issues with musl dns client, so in some cases it fails to resolve. https://gitlab.alpinelinux.org/alpine/aports/-/issues/9017?__goaway_challenge=cookie&__goaway_id=532e6a3acc2ec7453ce670715eb7a81f

Error Details

$ nslookup test.openai.azure.com
;; Got recursion not available from 10.44.0.10, trying next server
;; Got recursion not available from 2001:cafe:42:1::a
Server:		10.44.0.10
Address:	10.44.0.10#53

Non-authoritative answer:
test.openai.azure.com	canonical name = test.privatelink.openai.azure.com.
Name:	test.privatelink.openai.azure.com
Address: 10.104.2.4
;; Got SERVFAIL reply from 10.44.0.10, trying next server
;; Got SERVFAIL reply from 2001:cafe:42:1::a
** server can't find test.privatelink.openai.azure.com: SERVFAIL

What Should Have Happened?

Debian's dns resolver handle cluster's private endpoint DNS configuration in more graceful manner using glibc instead of musl
For the context, we had this issue on one of our clients installations where they do use hosted models / private endpoints. Having debian base image solved the issue.

Relevant Code Snippet

index 684a905..0f1c474 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,5 +1,5 @@
 # Use the official Node.js runtime as a parent image
-FROM node:20-alpine AS build
+FROM node:20-slim AS build
 
 # Set the working directory in the container
 WORKDIR /app
@@ -8,8 +8,10 @@ WORKDIR /app
 COPY package*.json ./
 COPY patches ./
 
-# Upgrade system packages
-RUN apk update && apk upgrade --no-cache
+# Update system packages and install necessary dependencies
+RUN apt-get update && apt-get upgrade -y && apt-get install -y \
+    curl \
+    && rm -rf /var/lib/apt/lists/*
 
 # Upgrade npm to version 10.9.2
 RUN npm install -g npm@10.9.2
@@ -22,14 +24,16 @@ COPY . .
 
 # Build the application and clean up
 RUN npm run build \
-&& rm -rf node_modules \
-&& npm install --omit=dev
+    && rm -rf node_modules \
+    && npm install --omit=dev
 
 # Use the official Node.js runtime as a parent image
-FROM node:20-alpine
+FROM node:20-slim
 
-# Upgrade system packages
-RUN apk update && apk upgrade --no-cache
+# Update system packages and install necessary dependencies
+RUN apt-get update && apt-get upgrade -y && apt-get install -y \
+    curl \
+    && rm -rf /var/lib/apt/lists/*
 
 # Upgrade npm to version 10.9.2
 RUN npm install -g npm@10.9.2

Your Twitter/LinkedIn

No response

[narengogi]

I dont think the right fix for this is to change from something lightweight like alpine to slim where the build size could be 5x larger

[arturfromtabnine]

looks like diff is not that big 88 mb vs 62 mb

[arturfromtabnine]

bump

[narengogi]

@sk-portkey for comments

[narengogi]

@arturfromtabnine If we may know, what cloud provider was this causing an issue in?
we have deployments on AWS, Azure and GCP using the same base image but we've not experienced anything similar

[arturfromtabnine]

@narengogi its a azure open ai provider. our own azure deployment works fine, but we had this problem in customers environment
I believe its not about provider itself
its about DNS failures

[narengogi]

@arturfromtabnine Yes I was referring to the deployment environment
If you could help with where your customers have the gateway deployed that could help narrow down the problem

[arturfromtabnine]

ah
its azure as well