Failed spark job reports on YARN as successful when deployed with client-mode

If you deploy your spark job on YARN with client deploy mode, be aware that the YARN application is always shown as SUCCEEDED on YARN resource manager site even if the Spark job fails.

See the following Jira ticket

https://issues.apache.org/jira/browse/SPARK-11058

Note from Spark docs:

--deploy-mode: Whether to deploy your driver on the worker nodes (cluster) or locally as an external client (client) (default: client)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s