We are seeing these errors on the client side sporadically.

Caused by: io.grpc.StatusRuntimeException: UNKNOWN: channel closed
    at io.grpc.Status.asRuntimeException(Status.java:532)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:398)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at (ThreadPoolExecutor.java:1128)
    at (ThreadPoolExecutor.java:628)
    at (Thread.java:834)
Caused by: java.nio.channels.ClosedChannelException: null
    at io.grpc.netty.Utils.statusFromThrowable(Utils.java:166)
    at io.grpc.netty.NettyClientHandler.onConnectionError(NettyClientHandler.java:474)
    at io.netty.handler.codec.http2.Http2ConnectionHandler.onError(Http2ConnectionHandler.java:641)
    at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:225)
    at io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeHeaders(DecoratingHttp2FrameWriter.java:53)
    at io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:157)
    at io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:141)
    at io.grpc.netty.NettyClientHandler.createStream(NettyClientHandler.java:543)
    at io.grpc.netty.NettyClientHandler.write(NettyClientHandler.java:312)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:716)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:708)
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:791)
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:701)
    at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1026)
    at io.netty.channel.AbstractChannel.write(AbstractChannel.java:288)
    at io.grpc.netty.WriteQueue$AbstractQueuedCommand.run(WriteQueue.java:174)
    at io.grpc.netty.WriteQueue.flush(WriteQueue.java:112)
    at io.grpc.netty.WriteQueue.access$000(WriteQueue.java:32)
    at io.grpc.netty.WriteQueue$1.run(WriteQueue.java:44)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:515)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    ... 1 common frames omitted
Caused by: java.nio.channels.ClosedChannelException: null
    at io.netty.channel.AbstractChannel$AbstractUnsafe.newClosedChannelException(AbstractChannel.java:955)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:863)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1378)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:716)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:708)
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:791)
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:701)
    at io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeHeadersInternal(DefaultHttp2FrameWriter.java:528)
    at io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeHeaders(DefaultHttp2FrameWriter.java:268)
    at io.netty.handler.codec.http2.Http2OutboundFrameLogger.writeHeaders(Http2OutboundFrameLogger.java:60)
    at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:208)
    ... 22 common frames omitted
Caused by: java.io.IOException: Connection reset by peer
    at (Native Method)
    at (SocketDispatcher.java:47)
    at (IOUtil.java:113)
    at (IOUtil.java:58)
    at (IOUtil.java:50)
    at (SocketChannelImpl.java:466)
    at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:405)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:928)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:356)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:895)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1383)
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:749)
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:741)
    at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:727)
    at io.grpc.netty.NettyClientHandler.sendPingFrame(NettyClientHandler.java:646)
    at io.grpc.netty.NettyClientHandler.write(NettyClientHandler.java:318)
    ... 17 common frames omitted

Both the server and the client run in Kubernetes with the server being a ClusterIP service. The client's channel builder looks like this:

NettyChannelBuilder.forAddress("some-service", 8090)
                .nameResolverFactory(DnsNameResolverProvider.asFactory())
                .defaultLoadBalancingPolicy("round_robin")
                .idleTimeout(60000, TimeUnit.MILLISECONDS)
                .usePlaintext();

The client sends requests in a burst of 4-5 (concurrently using project-reactor) every 5 minutes. We see failures happening once or twice a day. We have tried setting both keepAlive and idleTimeout on separate occasions but nothing seems to be working. We have retries defined for UNAVAILABLE and DEADLINE_EXCEEDED status codes but not UNKNOWN since that could mean retrying on potentially non-retriable errors. Is there a way to fix this on the client side without having to retry on UNKNOWN errors?

Client gRPC version: 1.20.0 Server gRPC version: 1.20.0 Netty Version: 4.1.39.Final