Protobuf Basics: Java

A basic Java programmers introduction to working with Protobuf.

This tutorial provides a basic Java programmer’s introduction to working with Protobuf. By walking through creating a simple example application, it shows you how to

Define message formats in a .proto file.
Use the Protobuf compiler.
Use the Java Protobuf API to write and read messages.

This isn’t a comprehensive guide to using Protobuf in Java. For more detailed reference information, see the Protobuf Language Guide (proto2), the Protobuf Language Guide (proto3), the Java API Reference, the Java Generated Code Guide, and the Encoding Reference.

The Problem Domain

The example we’re going to use is a very simple “address book” application that can read and write people’s contact details to and from a file. Each person in the address book has a name, an ID, an email address, and a contact phone number.

How do you serialize and retrieve structured data like this? There are a few ways to solve this problem:

Use Java Serialization. This is the default approach since it’s built into the language, but it has a host of well-known problems (see Effective Java, by Josh Bloch pp. 213), and also doesn’t work very well if you need to share data with applications written in C++ or Python.
You can invent an ad-hoc way to encode the data items into a single string – such as encoding 4 ints as “12:3:-23:67”. This is a simple and flexible approach, although it does require writing one-off encoding and parsing code, and the parsing imposes a small run-time cost. This works best for encoding very simple data.
Serialize the data to XML. This approach can be very attractive since XML is (sort of) human readable and there are binding libraries for lots of languages. This can be a good choice if you want to share data with other applications/projects. However, XML is notoriously space intensive, and encoding/decoding it can impose a huge performance penalty on applications. Also, navigating an XML DOM tree is considerably more complicated than navigating simple fields in a class normally would be.

Instead of these options, you can use Protobuf. Protobuf is the flexible, efficient, automated solution to solve exactly this problem. With Protobuf, you write a .proto description of the data structure you wish to store. From that, the Protobuf compiler creates a class that implements automatic encoding and parsing of the Protobuf data with an efficient binary format. The generated class provides getters and setters for the fields that make up a Protobuf message and takes care of the details of reading and writing the message as a unit. Importantly, the Protobuf format supports the idea of extending the format over time in such a way that the code can still read data encoded with the old format.

Where to Find the Example Code

The example code is included in the source code package, under the “examples” directory. Download it here.

Defining Your Protocol Format

To create your address book application, you’ll need to start with a .proto file. The definitions in a .proto file are simple: you add a message for each data structure you want to serialize, then specify a name and a type for each field in the message. Here is the .proto file that defines your messages, addressbook.proto.

syntax = "proto2";

package tutorial;

option java_multiple_files = true;
option java_package = "com.example.tutorial.protos";
option java_outer_classname = "AddressBookProtos";

message Person {
  optional string name = 1;
  optional int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    PHONE_TYPE_UNSPECIFIED = 0;
    PHONE_TYPE_MOBILE = 1;
    PHONE_TYPE_HOME = 2;
    PHONE_TYPE_WORK = 3;
  }

  message PhoneNumber {
    optional string number = 1;
    optional PhoneType type = 2 [default = PHONE_TYPE_HOME];
  }

  repeated PhoneNumber phones = 4;
}

message AddressBook {
  repeated Person people = 1;
}

As you can see, the syntax is similar to C++ or Java. Let’s go through each part of the file and see what it does.

The .proto file starts with a package declaration, which helps to prevent naming conflicts between different projects. In Java, the package name is used as the Java package unless you have explicitly specified a java_package, as we have here. Even if you do provide a java_package, you should still define a normal package as well to avoid name collisions in the Protobuf name space as well as in non-Java languages.

After the package declaration, you can see three options that are Java-specific: java_multiple_files, java_package, and java_outer_classname. java_package specifies in what Java package name your generated classes should live. If you don’t specify this explicitly, it simply matches the package name given by the package declaration, but these names usually aren’t appropriate Java package names (since they usually don’t start with a domain name). The java_outer_classname option defines the class name of the wrapper class which will represent this file. If you don’t give a java_outer_classname explicitly, it will be generated by converting the file name to upper camel case. For example, “my_proto.proto” would, by default, use “MyProto” as the wrapper class name. The java_multiple_files = true option enables generating a separate .java file for each generated class (instead of the legacy behavior of generating a single .java file for the wrapper class, using the wrapper class as an outer class, and nesting all the other classes inside the wrapper class).

Next, you have your message definitions. A message is just an aggregate containing a set of typed fields. Many standard simple data types are available as field types, including bool, int32, float, double, and string. You can also add further structure to your messages by using other message types as field types – in the above example the Person message contains PhoneNumber messages, while the AddressBook message contains Person messages. You can even define message types nested inside other messages – as you can see, the PhoneNumber type is defined inside Person. You can also define enum types if you want one of your fields to have one of a predefined list of values – here you want to specify that a phone number can be one of the following phone types: PHONE_TYPE_MOBILE, PHONE_TYPE_HOME, or PHONE_TYPE_WORK.

The " = 1", " = 2" markers on each element identify the unique “tag” that field uses in the binary encoding. Tag numbers 1-15 require one less byte to encode than higher numbers, so as an optimization you can decide to use those tags for the commonly used or repeated elements, leaving tags 16 and higher for less-commonly used optional elements. Each element in a repeated field requires re-encoding the tag number, so repeated fields are particularly good candidates for this optimization.

Each field must be annotated with one of the following modifiers:

optional: the field may or may not be set. If an optional field value isn’t set, a default value is used. For simple types, you can specify your own default value, as we’ve done for the phone number type in the example. Otherwise, a system default is used: zero for numeric types, the empty string for strings, false for bools. For embedded messages, the default value is always the “default instance” or “prototype” of the message, which has none of its fields set. Calling the accessor to get the value of an optional (or required) field which has not been explicitly set always returns that field’s default value.
repeated: the field may be repeated any number of times (including zero). The order of the repeated values will be preserved in the Protobuf message. Think of repeated fields as dynamically sized arrays.
required: a value for the field must be provided, otherwise the message will be considered “uninitialized”. Trying to build an uninitialized message will throw a RuntimeException. Parsing an uninitialized message will throw an