SSL Cert errors

[Yeraze]

I just upgraded from 4.13.1 to 4.14, and now the indexer won't start.. Keeps throwing SSL errors like so :

[2025-11-02T20:37:44,850][ERROR][o.o.h.n.s.SecureNetty4HttpServerTransport] [wazuh.indexer] Exception during establishing a SSL connection: javax.net.ssl.SSLHands
hakeException: Received fatal alert: bad_certificate                                                                                                              
javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate                                                                                        
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]                                                                             
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]                                                                             
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:370) ~[?:?]                                                                    
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287) ~[?:?]                                                                          
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:209) ~[?:?]                                                                 
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]                                                                           
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]                                                                         
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]                                                                     
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]                                                                         
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]                                                                         
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]                    

and

[2025-11-02T20:33:04,697][WARN ][o.o.h.AbstractHttpServerTransport] [wazuh.indexer] caught exception while handling client http traffic, closing connection Netty4
HttpChannel{localAddress=/172.21.0.3:9200, remoteAddress=/172.21.0.2:58656}                                                                                       
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than ta
g size (16)                                                                                                                                                       
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500) ~[netty-codec-4.1.121.Final.jar:4.1.121.Final]                   
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.121.Final.jar:4.1.121.Final]                  
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.121.Final.jar:4.1.121.Fin
al]                                                                                                                                                               
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.121.Final.jar:4.1.121.Fin
al]                                                                                                                                                               
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.121.Final.jar:4.1.121.Final
]                                                                                                                                                                 
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.121.Final.jar:4.1.121.Final]    
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.121.Final.jar:4.1.121.Fin
al]                                                                                                                                                               
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.121.Final.jar:4.1.121.Fin
al]                                                                                                                                                               
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.121.Final.jar:4.1.121.Final]             
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.121.Final.jar:4.1.121.Final]      
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) [netty-transport-4.1.121.Final.jar:4.1.121.Final]                          
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) [netty-transport-4.1.121.Final.jar:4.1.121.Final]                    
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) [netty-transport-4.1.121.Final.jar:4.1.121.Final]                         
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.121.Final.jar:4.1.121.Final]                                         
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) [netty-common-4.1.121.Final.jar:4.1.121.Final]            
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.121.Final.jar:4.1.121.Final]                               
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]                                                                                                 
Caused by: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)                   
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]                                                                             
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:383) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:134) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]                                                                                    
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:308) ~[netty-handler-4.1.121.Final.jar:4.1.121.Final]                           
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) ~[netty-handler-4.1.121.Final.jar:4.1.121.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377) ~[netty-handler-4.1.121.Final.jar:4.1.121.Final]                             
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1428) ~[netty-handler-4.1.121.Final.jar:4.1.121.Final]                                          
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[netty-codec-4.1.121.Final.jar:4.1.121.Final
]                                                                                                                                                                 
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[netty-codec-4.1.121.Final.jar:4.1.121.Final]
        ... 16 more                                                                                                                                               
Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
        at java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1864) ~[?:?]                                      
        at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:239) ~[?:?]                
at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:196) ~[?:?]                                                           
        at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:159) ~[?:?]                                                           
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ~[?:?]                                                                           
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]                                                                         
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]                                                                     
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]                                                                         
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]                                                                         
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]                                                                                    
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:308) ~[netty-handler-4.1.121.Final.jar:4.1.121.Final]                           
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486) ~[netty-handler-4.1.121.Final.jar:4.1.121.Final]                                          
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377) ~[netty-handler-4.1.121.Final.jar:4.1.121.Final]                             
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1428) ~[netty-handler-4.1.121.Final.jar:4.1.121.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[netty-codec-4.1.121.Final.jar:4.1.121.Final
]                                                                                                                                                                 
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[netty-codec-4.1.121.Final.jar:4.1.121.Final]                   
        ... 16 more                                                                                                                          

[hydrandt]

It seems the configs and certs path has changed.

The configs used to be in /usr/share/wazuh-indexer/, now they are in /usr/share/wazuh-indexer/config/. The certs folder moved into the config directory too.

So my indexer was using a default config, and the problem was wazuh.manager not connecting because the cert is for demo.indexer and signed with a different key (which is, I think, the cause of the error you pasted).

So I had to modify:

docker-compose.yaml, wazuh.indexer:

    volumes:
      - wazuh-indexer-data:/var/lib/wazuh-indexer
      - ./certs/wazuh_indexer_ssl_certs/root-ca.pem:/usr/share/wazuh-indexer/config/certs/root-ca.pem
      - ./certs/wazuh_indexer_ssl_certs/wazuh.indexer-key.pem:/usr/share/wazuh-indexer/config/certs/wazuh.indexer.key
      - ./certs/wazuh_indexer_ssl_certs/wazuh.indexer.pem:/usr/share/wazuh-indexer/config/certs/wazuh.indexer.pem
      - ./certs/wazuh_indexer_ssl_certs/admin.pem:/usr/share/wazuh-indexer/config/certs/admin.pem
      - ./certs/wazuh_indexer_ssl_certs/admin-key.pem:/usr/share/wazuh-indexer/config/certs/admin-key.pem
      - ./config/wazuh_indexer/opensearch.yml:/usr/share/wazuh-indexer/config/opensearch.yml
      - ./config/wazuh_indexer/internal_users.yml:/usr/share/wazuh-indexer/config/opensearch-security/internal_users.yml

and in the opensearch.yml, I just went for absolute paths:

plugins.security.ssl.http.pemcert_filepath: /usr/share/wazuh-indexer/config/certs/wazuh.indexer.pem
plugins.security.ssl.http.pemkey_filepath: /usr/share/wazuh-indexer/config/certs/wazuh.indexer.key
plugins.security.ssl.http.pemtrustedcas_filepath: /usr/share/wazuh-indexer/config/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /usr/share/wazuh-indexer/config/certs/wazuh.indexer.pem
plugins.security.ssl.transport.pemkey_filepath: /usr/share/wazuh-indexer/config/certs/wazuh.indexer.key
plugins.security.ssl.transport.pemtrustedcas_filepath: /usr/share/wazuh-indexer/config/certs/root-ca.pem

I had more issues: I had to re-enrol most of my agents (docker compose exec wazuh.manager /var/ossec/bin/manage_agents -r 006), and disable checking agent source IP when enrolling, by editing manager's ossec.conf and setting <use_source_ip>no</use_source_ip> in the <auth> section (all the agents seem to come through the manager's host docker IP: Duplicate IP: 172.18.0.1. Unable to add agent).

It is also not clear to me if we should do some of the steps in https://documentation.wazuh.com/current/upgrade-guide/upgrading-central-components.html , when running a single node setup with docker compose. I guess so, because I keep having warnings in the manager like: wazuh-analysisd: WARNING: Mitre Technique ID 'T1078' not found in database.

Update: There is a separate upgrade guide in the docs: https://documentation.wazuh.com/current/deployment-options/docker/upgrading-wazuh-docker.html

After recreating the containers with docker compose down and docker compose up, I don't get the wazuh-analysisd warnings anymore.

[awptechnologies]

same issue i think in my manager logs i get this

wazuh_manager.1.x60cpi6hmnfb@net3server | 2025-11-03T10:20:05.846-0500 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://wazuh.indexer:9200)): Get "https://wazuh.indexer:9200": x509: certificate is valid for demo.indexer, not wazuh.indexer

this is indexer error

wazuh_indexer.1.gkrg24a2rh4d@net3server | [2025-11-03T10:21:17,836][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [wazuh.indexer] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security) wazuh_indexer.1.gkrg24a2rh4d@net3server | [2025-11-03T10:21:17,837][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [wazuh.indexer] Failure no such index

and this is dashboard error

wazuh_dashboard.1.ny0vvm56wsge@net3server | {"type":"log","@timestamp":"2025-11-03T15:21:51Z","tags":["error","opensearch","data"],"pid":54,"message":"[ConnectionError]: unable to verify the first certificate"}

[Yeraze]

Well that got me a bit further along, but now I'm having SSO problems where it seems the SAML configuration requires some updates too.. and that's not included in the instructions...

[7djs7]

I had the same issue. I followed the upgrade log but still had issues. I ended up starting over. Not ideal... Next time I'll wait a month before changing versions. Never had real problems even after upgrading from version 4.8.

[awptechnologies]

Something like this has happen to me before and i started over. i now have 20 agents and thats a lot of work.

[7djs7]

Something like this has happen to me before and i started over. i now have 20 agents and thats a lot of work.

I have over 30. Yes, it's a lot of work but I didn't want to spend hours and hours troubleshooting so I started over.

[awptechnologies]

how do you go about doing it? stopping the service on the agent host reinstall all of wazuh and go back and start agent on hosts?

[7djs7]

how do you go about doing it? stopping the service on the agent host reinstall all of wazuh and go back and start agent on hosts?

I stopped the docker container. Renamed all the volumes it was using with this " docker run --rm -v <old_volume_name>:/from -v <new_volume_name>:/to alpine ash -c "cd /from ; cp -av . /to" instead of deleting the volumes. then i renamed the wazuh directory from "wazuh-docker" to "wazuh-docker-old" I then followed these directions for a single node. https://documentation.wazuh.com/current/deployment-options/docker/wazuh-container.html

I then went back in later and deleted the old volumes and old docker config files once my new wazuh instance was running with all the added old configuration that I had to manually put back in. It was a pain.

[awptechnologies]

So i basically said screw that lol and wanted to try something. i use nfs volumes so i deleted all the data folders rebuilt docker compose from fresh wazuh docker pull. (i use docker swarm so its a bit different.) After signing all the certs with generator i started it up and done. Didn't stop an agent on a single host and they all added there selves back. The only thing different is the order of agents numbers. i dont mind this since i use groups.