ES Rest High Level Client throws SocketTimeoutException after being idle for sometime

`RestHighLevelClient` is used to connect to ES 6.4(hosted on AWS) in a spring-boot app. When the app goes idle for some time, and request arrives, then the `RestHighLevelClient` throws `SocketTimeoutException`:
```
[Request processing failed; nested exception is org.springframework.data.elasticsearch.ElasticsearchException: Error while bulk for request: org.elasticsearch.action.bulk.BulkRequest@21511b6c] w
java.net.SocketTimeoutException: 5,000 milliseconds timeout on connection http-outgoing-38 [ACTIVE]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
```
RestHighLevelClient is created using:
```
@Bean
RestHighLevelClient client() {

ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo(elasticsearchHostAndPort)
.build();

return RestClients.create(clientConfiguration).rest();
}
```

```spring-data-elasticsearch``` version ```3.2.0.M2``` is used.

Any hints/workarounds?

In `RestClientBuilder.createHttpClient()` the defaults for socket timeout and connection timeout are set to 30 and 10 seconds.

You can override this defaults by implementing `RestClientBuilder.RequestConfigCallback` and calling `setRequestConfigCallback(...)` on your `RestHighLevelClient`

We did something like

@Override
public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder builder) {
return builder.setSocketTimeout(socketTimeout); // try to prevent SocketTimeoutException
}

it should be also configurable thru properties:

spring.elasticsearch.jest.connection-timeout=3000 # Connection timeout in milliseconds.
spring.elasticsearch.jest.multi-threaded=true # Enable connection requests from multiple execution threads.

Here is how I solved this issue:

@Configuration
@EnableElasticsearchRepositories(basePackages = "com.hamdos.repositories.es")
@ComponentScan(basePackages = { "com.hamdos.services.es" })
@Order(2)
public class Config {

@Bean
RestHighLevelClient client() {

RestClientBuilder builder = RestClient.builder(
new HttpHost("localhost", 9289))
.setRequestConfigCallback(
new RestClientBuilder.RequestConfigCallback() {
@Override
public RequestConfig.Builder customizeRequestConfig(
RequestConfig.Builder requestConfigBuilder) {
return requestConfigBuilder
.setConnectTimeout(5000)
.setSocketTimeout(60000);
}
});

return new RestHighLevelClient(builder);
}

@Bean
public ElasticsearchOperations elasticsearchTemplate() {
return new ElasticsearchRestTemplate(client());
}
}

Info about system:


- spring-data-elasticsearch: 4.0.0.RELEASE
- spring-boot: 2.3.0.RELEASE
- Elasticsearch: 7.7.0

I have also tried setting the connection/socket timeout to 0, as suggested here and in other places.
It didn't help eventually.

There is another solution/workaround, suggested by spring-data-elasticsearch in

[To see links please register here]

.
It simply perform an internal retry in case of such exception. It does not really solve the issue, but your client will not get and error. Instead, the first request after idle time will take additional 5 seconds (or whatever timeout your configured).

If you use Springboot-data-elasticsearch version 4+ (staring with Springboot 2.3.0), then you can apply the solution.

I can confirm the following solution is working (with the limitations I mentioned above):

```java

@Configuration
public class ElasticSearchRestClientConfig extends AbstractElasticsearchConfiguration {

@Autowired
private RestHighLevelClient restHighLevelClient;

@Override
public RestHighLevelClient elasticsearchClient() {
return restHighLevelClient;
}

@Bean
@Override
public ElasticsearchCustomConversions elasticsearchCustomConversions() {
return new ElasticsearchCustomConversions();
}

@Override
public ElasticsearchOperations elasticsearchOperations(ElasticsearchConverter elasticsearchConverter) {
return new ElasticsearchRestTemplate(elasticsearchClient(), elasticsearchConverter) {
@Override
public <T> T execute(ClientCallback<T> callback) {
int retryCount = 0;
T t = null;
while (retryCount <= RestClientBuilder.DEFAULT_MAX_CONN_PER_ROUTE && t == null) {
try {
t = super.execute(callback);
} catch (DataAccessResourceFailureException e) {
// retry
if (e.getCause() != null && (e.getCause().getCause() instanceof SocketTimeoutException) &&
(retryCount < RestClientBuilder.DEFAULT_MAX_CONN_PER_ROUTE)) {
retryCount++;
log.warn("Elasticsearch client - performing retry {} after caught DataAccessResourceFailureException: {}", retryCount, e.getMessage());
}
else {
throw e;
}
}
}
return t;
}
};
}

```

@Bean
override fun elasticsearchClient(): RestHighLevelClient {
val clientConfiguration = ClientConfiguration
.builder()
.connectedTo("localhost:9200")
.withSocketTimeout(enter time in millisecond)
.build()`enter code here`
return RestClients.create(clientConfiguration).rest()
}

just use withSocketTimeout(30000) mthode like shown above