What exactly are zero copy techniques and how these techniques can be employed to achieve better performance in distributed system ?
If you browse the Hadoop MapReduce and Spark JIRA tickets, you will find a number of tickets related to the use of zero copy techniques such as MMap memory mapped files and sendFile() to improve the system.
Zero copy techniques are these techniques used to eliminate unnecessary data copy and context switches across application and kernel space. Please refer to the following excellent post for an in depth explanation of these techniques.
http://www.ibm.com/developerworks/library/j-zerocopy/
Traditionally, if a server wants to send data over the network to a client, it needs to read the data from the disk into kernel memory before storing it in the user memory. Then it transfers the data again from the user memory space to kernel buffer associated with the network stack before sending to the network interface card. See Figure 1 (taken from the above paper)
A popular zero copy technique is called sendFile() or transferTo. Please see the following figure. (taken from the same paper).
As you see in Hadoop, it has already reverted to use zero copy transferTo way back in version 0.18.
https://issues.apache.org/jira/browse/HADOOP-3164
Also, it uses the same technique in sending shuffle files. Please see
In Spark, there is also plan to use the same technique in sending shuffle files, targeted for upcoming version 1.2.0.
Spark-2468 : introduce the same transferTo technique in sending shuffle files.
Wei Shung Chung 