Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface. It enables easy submission of Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark Context management, all via a simple REST interface or an RPC client library.

Why is Livy used?

Livy is an open source Apache licensed REST web service for managing long running Spark Contexts and submitting Spark jobs.

Why is the cluster mode strongly recommended when using Livy?

It is strongly recommended to use cluster deploy mode when using Livy with YARN. This ensures that the host running the Livy server doesn't become overloaded when multiple sessions/jobs are running. livy. ... If the cluster is busy, Livy might not be able to launch the job within this time limit.