Service and Endpoints for the node exporters are not correctly configured

[SSvilen]

The windows node exporter is installed on all windows worker nodes, but the required Service and Endpoint resources are no created at all.
There is a service object created, but it's from type ClusterIP, which in this case won't work.
The Service should be of type 'ExternalName' and the Endpoints should be updated by the operator on every node join/deletion operation.
For instance:

apiVersion: v1
kind: Service
metadata:
 labels:
   name: windows-exporter
 name: windows-exporter
 namespace: openshift-windows-machine-config-operator
spec:
 type: ExternalName
 ports:
   - name: metrics
     port: 9182
     protocol: TCP
     targetPort: 9182
 externalName: nodexporter
---
apiVersion: v1
kind: Endpoints
metadata:
 labels:
   name: windows-exporter
 name: windows-exporter
 namespace: openshift-windows-machine-config-operator
subsets:
 - addresses:
     - ip: 1.1.1.1
       targetRef:
         kind: Node
         name: winmach-q84jj
         uid: ab8028e7-a0ed-4f83-89e5-b577be2231ed
     - ip: 1.1.1.1
       targetRef:
         kind: Node
         name: winmach-t5vgm
         uid: 1b710328-88d5-4142-a78f-dd414705cc19
   ports:
     - name: metrics
       port: 9182
       protocol: TCP

[mansikulkarni96]

@SSvilen thanks for the provided information.

As you can see in the manifests/windows-exporter_v1_service.yaml, the type is not set to ClusterIP.
The required service and endpoint names should be both windows-exporter as that name is used to get the resources in the operator code.
I suspect monitoring is not enabled in the operator namespace. Please ensure label openshift.io/cluster-monitoring=true is present in the openshift-windows-machine-config-operator namespace which is required for monitoring resources to be created by WMCO in that namespace.
If it is not enabled you can see a log like: install the prometheus-operator to enable Prometheus configuration in the WMCO logs.
Community Operators have a checkbox to enable monitoring in the operator namespace, if you are building from source you can use oc label ns openshift-windows-machine-config-operator openshift.io/cluster-monitoring=true --overwrite to set the label.
Let us know if that resolves the issue!

[SSvilen]

@mansikulkarni96,

The monitoring for the namespace is enabled. The problem is that the node exporter is installed on the windows worker nodes and it's not running as a pod, like it is for the linux based os. So prometheus operator can not properly discover the endpoint for that servicemonitor.
So I had to recreate the service and manually create the endpoint object, which in turn points to the windows nodes.
Or am I overthinking this?

[mansikulkarni96]

@SSvilen Thanks for confirming that.
The behaviour you see is the expected behaviour, windows-exporter runs as a Windows service on the Windows nodes which is different from it's linux counterpart due to support reasons.
Prometheus operator should be able to discover the endpoint, you can take a look at the service_monitor.yaml, you can see how the re-labellings are applied to make the endpoint discoverable.
What you are expecting is exactly what the operator does, it updates the endpoint objects on every node join/deletion operation, more details in metrics.go if you are interested in the code base.
If you could provide Windows Machine Config Operator logs and details about the exact operator version, OCP version and the steps followed to reach this point, I should be able to help you out further.

[SSvilen]

OK, I see what's happening.

controller.windowsmachine    invalid Machine    {"name": "winmach-t5vgm", "error": "no internal IP address associated",

and based on the code in metrics.go an internal IP address is expected.

The status field of the machine shows type 'InternalDNS'

status:                                                                                                                                                                                                                                    
  addresses:                                                                                                                                                                                                                               
   - address: winmach-q84jj                                                                                                                                                                                                                 
     type: InternalDNS

I'm not sure why that is.

[mansikulkarni96]

@SSvilen can you provide details about the WMCO version, cloud provider, OCP version and the Windows Server version used for the VM?
This is what the support matrix looks like Supported Cloud Providers based on OKD/OCP Version and WMCO version and Supported Windows Server versions.

[SSvilen]

@mansikulkarni96 ,

WMCO 3.1
OCP 4.8
Windows 20H2

But it would be also beneficial, if there is a bit more logging. For instance here. That would make the troubleshooting easier.

[mansikulkarni96]

@SSvilen logging info noted.
According to your comment, the Windows worker node is present, is that added by using WMCO?
If yes, then the "no internal IP address associated" error should have resolved on its own as the IP address is not just required for metrics but also for the SSH connection to the VM.
I would request you some more information for the deubgging further:

1. Cloud provider information: is it vmware vSphere?
2. Node configuration method used here, provide info from one of the two:

• BYOH
• machinesSets

3. Full output of oc logs -f deployment/windows-machine-config-operator -n openshift-windows-machine-config-operator
4. Windows MachineSet yaml/ configMap yaml depending on the Node configuration method used.
5. Output of oc get network.operator cluster -o yaml

[SSvilen]

@mansikulkarni96 ,

1.Cloud provider information: is it vmware vSphere?

Yes.

2. Node configuration method used here, provide info from one of the two:
- BYOH
- machinesSet

machinesSet

network.txt
operatorlogs.txt
machineSet.txt

Thanks!

[mansikulkarni96]

@SSvilen Thanks for providing the logs, from the operator logs I can see the IP address cannot be found to configure the Windows machine into a node. You should see the same issue if you try to oc describe the machine object, it is trying to configure. I suspect it has to do with the golden image creation for vSphere. Please make sure you have followed all the steps described in vsphere-golden-image.md

[SSvilen]

@mansikulkarni96,

ok thanks. We'll look at it again.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale