Repository navigation
Issue during docker build in microclimate jenkins pipeline #78
Description
Issue type: Bug
Reported in: 1.10 installed on ICP 3.1.0
Description of issue: If the pipeline is initiated docker build is failing with the following error
Step 11/17 : COPY . .
---> 39e33e89aed0
Step 12/17 : RUN chown -R node:node /usr/src/app
---> Running in afcc3b39aea2
**OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 40\"": unknown**
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE
Workaround (if possible): I came to understand the link opencontainers/runc#1740 it could be an issue with page memory and for fixing the issue
we need to run "echo 1 > /proc/sys/vm/drop_caches" in the worker node hosting the microclimate jenkins pod. This issue is happening more than 3 times per day and each time we need to run the above command. I have created a cronjob for the same to run every 5 hours but cannot stop the issue . While checking the node i can see the node is not fully utilized ie we have enough memory available.
May i get some recommendation from Microclimate team for fixing the issue permanently. ?
Activity
sishida commentedon May 24, 2019
Hello! Could you please try to update to the newest version of Microclimate?
https://github.com/IBM/charts/blob/master/stable/ibm-microclimate/RELEASENOTES.md
If problems persist after the update, please let us know! Thank you!
arjunsbabu commentedon May 27, 2019
@sishida we cant do it because of microclimate limitation. We can only use ICp 3.1.0 because that is a requirement for another chart ( Watson assistant 1.1.0 ) . Microclimate 1.10 is the highest supported on ICp 3.1.0 . So we cannot proceed with highest version of MC without updating ICp
So let us know how to avoid such issues in MC 1.10.
Is it a known issue ?
sishida commentedon May 28, 2019
I have notified my team of this situation. I will let you know what we find out.
arjunsbabu commentedon Jun 6, 2019
@sishida any update
sishida commentedon Jun 6, 2019
@arjunsbabu My team has asked another team to look into this issue. Hopefully we can get answers from them.
arjunsbabu commentedon Jun 7, 2019
@sishida thanks for the update
jagraj commentedon Jun 7, 2019
I and Arjun had a call on this issue and here is more information.
HA Cluster configuration.
Masters - 3 nodes
Management - 2 nodes
Proxy - 2 nodes
VA - 1 node
Workers for Watson Assistant - 4 nodes
Other workers shared - 3 nodes (Microclimate, Health coach, Cognitive virtual Assistant, Redis)
Worker config - (2 CPU cores, 8 GB RAM)
This issue happens in pre-prod but not in production environment. Customer restarted one of the node where we were seeing this problem and after that we have not seen this problem so far.
The next action item is to reproduce this problem in one of customer POC environment and I requested to capture monitoring information on these three worker nodes where Microclimate is running.