Protocol Buffers (part 1, basics), follow-up to JSON & XML

I quickly introduced Protocol Buffers in my last post. It is an amazing tool that every programmer should know. The .Net version protobuf-net can be downloaded from here.

On its homepage Protocol Buffers is described with the following words:

Protocol buffers is the name of the binary serialization format used by Google for much of their data communications. It is designed to be:

small in size – efficient data storage (far smaller than xml)
cheap to process – both at the client and server
platform independent – portable between different programming architectures
extensible – to add new data to old messages

This is succinct. Protobuf-net also supports the WCF.

Add the protobuf-net.dll, which is located in the net30 installation directory, to your solution references.
We need some data structure to start with.

public class Book {
    public string author;
    public List<Fable> stories;
    public DateTime edition;
    public int pages;
    public double price;
    public bool isEbook;
} // class

public class Fable {
    public string title;
    public double[] customerRatings;
} // class

Protocol Buffers uses attributes to identify types to serialize. The ProtoMember attribute needs a positive integer. This can be painful, because you have to avoid overlapping when using inheritance. But integers have a distinct advantage as well. They are much faster than strings. As you are already aware, Protocol Buffers is about speed.

[ProtoContract]
public class Book {
    [ProtoMember(1)]
    public string author;
    [ProtoMember(2)]
    public List stories;
    [ProtoMember(3)]
    public DateTime edition;
    [ProtoMember(4)]
    public int pages;
    [ProtoMember(5)]
    public double price;
    [ProtoMember(6)]
    public bool isEbook;

    public override string ToString() {
        StringBuilder s = new StringBuilder();
        s.Append("by "); s.Append(author);
        s.Append(", edition "); s.Append(edition.ToString("dd MMM yyyy"));
        s.Append(", pages "); s.Append(pages);
        s.Append(", price "); s.Append(price);
        s.Append(", isEbook "); s.Append(isEbook);
        s.AppendLine();
        if (stories != null) foreach (Fable lFable in stories) {
                s.Append("title "); s.Append(lFable.title);
                s.Append(", rating "); s.Append(lFable.customerRatings.Average());
                s.AppendLine();
            }

        return s.ToString();
    } //
} // class

[ProtoContract]
public class Fable {
    [ProtoMember(1)]
    public string title;
    [ProtoMember(2)]
    public double[] customerRatings;
} // class

public static Book GetData() {
    return new Book {
        author = "Aesop",
        price = 1.99,
        isEbook = false,
        edition = new DateTime(1975, 03, 13),
        pages = 203,
        stories = new List<Fable>(new Fable[] {
            new Fable{ title = "The Fox & the Grapes", customerRatings = new double[]{ 0.7, 0.7, 0.8} },
            new Fable{ title = "The Goose that Laid the Golden Eggs", customerRatings = new double[]{ 0.6, 0.75, 0.5, 1.0} },
            new Fable{ title = "The Cat & the Mice", customerRatings = new double[]{ 0.1, 0.0, 0.3} },
            new Fable{ title = "The Mischievous Dog", customerRatings = new double[]{ 0.45, 0.5, 0.4, 0.0, 0.5} }
    })
    };
} //

Let’s serialize the data now.

public static void SerializeData() {
    MemoryStream lStream = new MemoryStream();
    BinaryWriter lWriter = new BinaryWriter(lStream); // no "using", because it would close the MemoryStream automatically
    Book lBook = GetData();
    ProtoBuf.Serializer.Serialize<Book>(lStream, lBook);
    lWriter.Flush();
    lStream.Position = 0;

    using (BinaryReader lReader = new BinaryReader(lStream)) {
        for (long i = 0, n = lStream.Length; i < n; i++) {
            byte b = lReader.ReadByte();
            Console.Write(string.Format("{0:X2} ", b));
            if ((i+1) % 20 == 0) Console.WriteLine();
        }
        Console.WriteLine();
        Console.WriteLine();
        Console.WriteLine("number of bytes: " + lStream.Length);
    }
    Console.ReadLine();
} //

example output:
0A 05 41 65 73 6F 70 12 31 0A 14 54 68 65 20 46 6F 78 20 26
20 74 68 65 20 47 72 61 70 65 73 11 66 66 66 66 66 66 E6 3F
11 66 66 66 66 66 66 E6 3F 11 9A 99 99 99 99 99 E9 3F 12 49
0A 23 54 68 65 20 47 6F 6F 73 65 20 74 68 61 74 20 4C 61 69
64 20 74 68 65 20 47 6F 6C 64 65 6E 20 45 67 67 73 11 33 33
33 33 33 33 E3 3F 11 00 00 00 00 00 00 E8 3F 11 00 00 00 00
00 00 E0 3F 11 00 00 00 00 00 00 F0 3F 12 2F 0A 12 54 68 65
20 43 61 74 20 26 20 74 68 65 20 4D 69 63 65 11 9A 99 99 99
99 99 B9 3F 11 00 00 00 00 00 00 00 00 11 33 33 33 33 33 33
D3 3F 12 42 0A 13 54 68 65 20 4D 69 73 63 68 69 65 76 6F 75
73 20 44 6F 67 11 CD CC CC CC CC CC DC 3F 11 00 00 00 00 00
00 E0 3F 11 9A 99 99 99 99 99 D9 3F 11 00 00 00 00 00 00 00
00 11 00 00 00 00 00 00 E0 3F 1A 03 08 D2 1D 20 CB 01 29 D7
A3 70 3D 0A D7 FF 3F

number of bytes: 267

And back again: deserialize data.

public static void ToAndFro() {
    using (MemoryStream lStream = new MemoryStream()) {
        BinaryWriter lWriter = new BinaryWriter(lStream);
        Book lBook = GetData();
        ProtoBuf.Serializer.Serialize<Book>(lStream, lBook);
        lWriter.Flush();
        lStream.Position = 0;

        Book lCopy = ProtoBuf.Serializer.Deserialize<Book>(lStream);
        Console.WriteLine(lCopy.ToString());
    }

    Console.ReadLine();
} //

example output:
by Aesop, edition 13 Mar 1975, pages 203, price 1.99, isEbook False
title The Fox & the Grapes, rating 0.733333333333333
title The Goose that Laid the Golden Eggs, rating 0.7125
title The Cat & the Mice, rating 0.133333333333333
title The Mischievous Dog, rating 0.37