Here is a great presentation about new Hive ORC file format (Optimized Record Columnar file)

If you check out Hive 0.11, you can find the ORCFile codes in this package

org/apache/hadoop/hive/ql/io/orc

As for information about the RCFile (Record Columnar File), you can take a look at the following paper.

Click to access TR-11-4.pdf