Here are some interesting papers about automatic feature engineering.
Great news ! Spark 2.3.0 was released yesterday. This new release comes with many great features, checkout the release note
One of the many notable features is the Spark on Kubernetes support. It provides a new kubernetes scheduler backend that supports native submission of spark jobs to a kubernetes cluster.
For those who are interested in the details of implementation, you can check out the Spark on Kubernetes Design Proposal
Here is the associated Jira ticket https://issues.apache.org/jira/browse/SPARK-18278