In TaskTracker, there is a daemon thread called TaskMemoryManagerThread running in the background. It monitors the memory usage of each task and kills the process if necessary. It uses the ShellCommandExecutor class which wraps around the ProcessBuilder to issue the sigkill command.
TaskTracker is essentially a Thread running on a node. Internally it launches two inner TaskLauncher threads called mapLauncher and reduceLauncher. TaskLauncher thread uses a List to keep track of assigned tasks and starts the task in another TaskRunner thread. TaskRunner thread will then launch a JVM process for the task.
If we dig inside the codes deeper, you will find that TaskRunner contains a JVMmanager instance. Inside JVMmanager, it contains two different JvmManagerForType instances, one for map tasks (mapJvmManager) and another for reduce tasks (reduceJvmManager). Each is responsible of spawning JvmRunner thread which will actually launch a JVM process.