Skip to content

[Bug]: CosmosDBEmulatorContainer 'EOFException: SSL peer shut down incorrectly' when building keystore #8324

New issue
New issue
Closed
Closed
[Bug]: CosmosDBEmulatorContainer 'EOFException: SSL peer shut down incorrectly' when building keystore#8324
Labels
modules/azuretype/bug

Description

@rvervaek
rvervaek
opened on Feb 19, 2024

Module

Azure

Testcontainers version

1.18.3

Using the latest Testcontainers version?

Yes

Host OS

Windows

Host Arch

x86

Docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:44 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:02 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

I'm trying to use the CosmosDBEmulatorContainer as described in https://java.testcontainers.org/modules/azure/.

However when I call the buildNewKeyStore() method of the started CosmosDBEmulatorContainer, the call fails during SSL handshake:

java.io.EOFException: SSL peer shut down incorrectly

Relevant log output

Exception in thread "main" java.lang.ExceptionInInitializerError
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:375)
	at com.intellij.junit5.JUnit5TestRunnerUtil.loadMethodByReflection(JUnit5TestRunnerUtil.java:126)
	at com.intellij.junit5.JUnit5TestRunnerUtil.buildRequest(JUnit5TestRunnerUtil.java:102)
	at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:43)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.IllegalStateException: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
	at org.testcontainers.containers.KeyStoreBuilder.buildByDownloadingCertificate(KeyStoreBuilder.java:33)
	at org.testcontainers.containers.CosmosDBEmulatorContainer.buildNewKeyStore(CosmosDBEmulatorContainer.java:37)
	at ExampleTest.<clinit>(ExampleTest.java:30)
	... 10 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
	at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1719)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1518)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
	at org.testcontainers.shaded.okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:379)
	at org.testcontainers.shaded.okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337)
	at org.testcontainers.shaded.okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209)
	at org.testcontainers.shaded.okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
	at org.testcontainers.shaded.okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
	at org.testcontainers.shaded.okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
	at org.testcontainers.shaded.okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
	at org.testcontainers.shaded.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
	at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.testcontainers.shaded.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.testcontainers.shaded.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.testcontainers.shaded.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at org.testcontainers.shaded.okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at org.testcontainers.shaded.okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at org.testcontainers.containers.KeyStoreBuilder.buildByDownloadingCertificate(KeyStoreBuilder.java:30)
	... 12 more
	Suppressed: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
		... 35 more
	Caused by: java.io.EOFException: SSL peer shut down incorrectly
		at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
		at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
		at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
		at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
		at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
		... 33 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
	at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
	... 33 more

Additional Information

Tested with testcontainers versions 1.18.3 and 1.19.5.
Tested with mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest and mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:mongodb images.

Example project illustrating the issue:
testcontainers-cosmosdb-em-ssl-handshake.zip

Activity

  1. rvervaek
    added
    type/bug
    on Feb 19, 2024

  2. eddumelendez commented on Feb 20, 2024

    @eddumelendez
    eddumelendez
    on Feb 20, 2024
    Member

    Hi, you can override the configuration

    waitingFor(Wait.forHttps("/_explorer/emulator.pem").forStatusCode(200).allowInsecure());
withStartupTimeout(Duration.ofMinutes(3));

    Looks like there has been some changes in the image provided and it has affected the default strategy. Those changes have not been applied because the image is pretty unstable in GHA. See Azure/azure-cosmos-db-emulator-docker#45

  3. eddumelendez
    added
    modules/azure
    on Feb 20, 2024

  4. rvervaek commented on Feb 21, 2024

    @rvervaek
    rvervaek
    on Feb 21, 2024
    Author

    Thanks @eddumelendez. The workaround you provided indeed works.

  5. kiview commented on Apr 25, 2024

    @kiview
    kiview
    on Apr 25, 2024
    Member

    Since the workaround solved the immediate issue, I am going to close, given that we are still waiting for the image to stabilize in upstream.

  6. kiview
    closed this as completedon Apr 25, 2024

  7. OleksandrShkurat commented on May 14, 2024

    @OleksandrShkurat
    OleksandrShkurat
    on May 14, 2024

    I'd suggest adding those details to the documentation
    https://java.testcontainers.org/modules/azure/

  8. vyu-talend commented on May 14, 2024

    @vyu-talend
    vyu-talend
    on May 14, 2024

    @eddumelendez Hi, the workaround works for me. But I prefer a more flexible way to configure the wait strategy, because it has to wait for 3 minutes every time on CI build. I've been searching for some other signs that the Cosmos emulator has fully started, but no luck. Could you please give some advice about the wait strategy? Thanks in advance!

  9. OleksandrShkurat commented on May 14, 2024

    @OleksandrShkurat
    OleksandrShkurat
    on May 14, 2024

    I've observed that CosmosDB Emulator starting long for real (almost two minutes on my local machine). In general it looks unreal to make it responsible earlier than its web-context became available. So, no chance to make it faster. The question is to CosmosDB here rather than to testcontainers.

  10. vyu-talend commented on May 14, 2024

    @vyu-talend
    vyu-talend
    on May 14, 2024

    @OleksandrShkurat Yeah, it's the same as my local machine. From here we can see it indeed has the strategy on testcontainers side, like get the exposed port or the log marked that the cosmos emulator has already been started. But although the container has been started, the web service is still unavailable. And I've been using the docker env variable with "AZURE_COSMOS_EMULATOR_PARTITION_COUNT" to restrict the number of partitions to be started, in order to make it faster. However, I still couldn't find an accurate signal indicating that the web service is available.

  11. vyu-talend commented on May 15, 2024

    @vyu-talend
    vyu-talend
    on May 15, 2024

    @eddumelendez Sorry, the wait strategy here worked. The specific startup time is 106 seconds on my local, and the timeout setting has not been reached

    Hi, you can override the configuration

    waitingFor(Wait.forHttps("/_explorer/emulator.pem").forStatusCode(200).allowInsecure());
withStartupTimeout(Duration.ofMinutes(3));

    Looks like there has been some changes in the image provided and it has affected the default strategy. Those changes have not been applied because the image is pretty unstable in GHA. See Azure/azure-cosmos-db-emulator-docker#45

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    modules/azuretype/bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions