Here is a great presentation about new Hive ORC file format (Optimized Record Columnar file)
If you check out Hive 0.11, you can find the ORCFile codes in this package
org/apache/hadoop/hive/ql/io/orc
As for information about the RCFile (Record Columnar File), you can take a look at the following paper.