Category Archives: Serialization
JSON to LINQ to JSON
It is about 9:30pm and I am sitting on the train home. It has been quite tough organizing my life in the last months. This is not the promised follow-up on OpenGL. There has been no time to read any book.
Anyway, this post shows how to run LINQ queries on JSON. The use can be quite broad. Imagine a shell command executing your LINQ on any JSON file. You can query into depth and build complex return tree structures with 2 or more levels. LINQ itself is highly flexible. And – just to give it the right flair – it returns results in the good old JSON format.
You could even go a bit further and insert more than just a LINQ query. In theory, you could add any C# code at run-time. The changes to this program would be minuscule.
I have sufficiently commented the code. There is a minimum overhead to get the job done.
How it works:
- The JSON data is imported using the JavaScriptSerializer, which is part of the System.Web.Extensions library reference. We try to force the result into a Dictionary<string, object>. Do not lazily use ‘object’. Some type information would get lost. I played with this and got eg. arrays instead of ArrayLists.
- The resulting structure is then used to build classes. These are converted to legible C# source code.
- The LINQ command is inserted. You can see the result in the TextBox titled ‘C# source code output’.
- A compiler instance is created and some library references are added. We compile the generated source code at run-time and then execute the binary.
- A standard JSON object is returned, rudimentary formatted and displayed as the final result.
Btw. I have inserted a horizontal and a vertical GridSplitter into the WPF XAML. You can easily change the size of the TextBoxes while playing with this tool. A few JSON examples were also added.
<Window xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" mc:Ignorable="d" x:Class="LinqJson.MainWindow" Title="JSON LINQ JSON" Height="Auto" Width="876" d:DesignHeight="695">
<Grid>
<Grid.RowDefinitions>
<RowDefinition Height="1*"/>
<RowDefinition Height="43*"/>
<RowDefinition Height="30"/>
<RowDefinition Height="150*"/>
<RowDefinition Height="30"/>
<RowDefinition Height="150*"/>
<RowDefinition Height="10*"/>
</Grid.RowDefinitions>
<Grid.ColumnDefinitions>
<ColumnDefinition Width="10*"/>
<ColumnDefinition Width="250*"/>
<ColumnDefinition Width="10"/>
<ColumnDefinition Width="250*"/>
<ColumnDefinition Width="10*"/>
<ColumnDefinition/>
</Grid.ColumnDefinitions>
<Button Content="convert" Width="50" Height="30" Click="Button_Click" Grid.Row="1" HorizontalAlignment="Left" Grid.Column="1" Margin="0,8,0,7"/>
<StackPanel Grid.Column="1" Grid.Row="2" VerticalAlignment="Bottom" HorizontalAlignment="Left" Orientation="Horizontal" Height="27" >
<Label Content="JSON input" VerticalAlignment="Bottom" HorizontalAlignment="Left" Margin="0,1" />
<ComboBox Width="240" Margin="50,3,3,3" SelectionChanged="ComboBox_SelectionChanged" VerticalAlignment="Bottom">
<ComboBoxItem Name="ex1">Example 1</ComboBoxItem>
<ComboBoxItem Name="ex2">Example 2</ComboBoxItem>
<ComboBoxItem Name="ex3">Example 3</ComboBoxItem>
<ComboBoxItem Name="ex4">Example 4</ComboBoxItem>
</ComboBox>
</StackPanel>
<TextBox x:Name="JsonIn" HorizontalScrollBarVisibility="Auto" VerticalScrollBarVisibility="Auto" Grid.Column="1" Grid.Row="3"/>
<GridSplitter Grid.Column="2" Grid.RowSpan="999" HorizontalAlignment="Stretch" Width="10" Background="Transparent" ResizeBehavior="PreviousAndNext"/>
<GridSplitter Grid.Row="4" Grid.ColumnSpan="999" VerticalAlignment="Top" HorizontalAlignment="Stretch" Height="10" Background="Transparent" ResizeBehavior="PreviousAndNext" />
<Label Content="JSON output" Grid.Column="3" Grid.Row="2" VerticalAlignment="Bottom" />
<TextBox x:Name="JsonOut" VerticalAlignment="Stretch" HorizontalScrollBarVisibility="Auto" VerticalScrollBarVisibility="Auto" Grid.Column="3" Grid.Row="3" />
<Label Content="C# source code output" Grid.Column="3" Grid.Row="4" VerticalAlignment="Bottom" />
<TextBox x:Name="CSharpSourceCode" HorizontalScrollBarVisibility="Auto" VerticalScrollBarVisibility="Auto" Grid.Column="3" Grid.Row="5" />
<Label Content="LINQ input" Grid.Column="1" Grid.Row="4" VerticalAlignment="Bottom"/>
<TextBox x:Name="LinqIn" HorizontalScrollBarVisibility="Auto" VerticalScrollBarVisibility="Auto" Grid.Column="1" Grid.Row="5" />
</Grid>
</Window>
using Microsoft.CSharp;
using System;
using System.CodeDom.Compiler;
using System.Collections.Generic;
using System.Reflection;
using System.Text;
using System.Windows;
using System.Windows.Controls;
namespace LinqJson {
public partial class MainWindow : Window {
public MainWindow() {
InitializeComponent();
} // constructor
private void Button_Click(object sender, RoutedEventArgs e) {
// --------------------------------------------------
// get the JSON input
// --------------------------------------------------
string lJsonIn = JsonIn.Text;
// --------------------------------------------------
// construct the C# source code (class hierarchy)
// --------------------------------------------------
Converter lConverter = new Converter();
Dictionary<string, object> lTree = lConverter.JsonToDictionary(lJsonIn);
lConverter.DictionariesToClasses("root", 0, lTree);
var lStringBuilder = new StringBuilder();
lConverter.BuildClasses(lStringBuilder);
string lCSharpSourceCode = lConverter.GetUsings() + lStringBuilder.ToString();
// --------------------------------------------------
// add the LINQ command to the source code
// --------------------------------------------------
string lLinq = LinqIn.Text;
string lEntryPoint = "\n\n";
lEntryPoint += "public class baseClass {\n";
lEntryPoint += " public static object executeLinq(string xJson) {\n";
lEntryPoint += " xJson = xJson.Trim();";
lEntryPoint += " if (xJson[0] == '[') xJson = \"{home: \" + xJson + \"}\";";
lEntryPoint += " var lSerializer = new JavaScriptSerializer();\n";
lEntryPoint += " var root = lSerializer.Deserialize<Class_root>(xJson);\n";
lEntryPoint += " var lResult = " + lLinq.Replace("\n", "\n ") + ";\n";
lEntryPoint += " return lSerializer.Serialize(lResult);\n";
lEntryPoint += " }\n";
lEntryPoint += "}\n";
lCSharpSourceCode += lEntryPoint;
// --------------------------------------------------
// display the source code
// --------------------------------------------------
CSharpSourceCode.Text = lCSharpSourceCode;
// --------------------------------------------------
// compile the source code
// --------------------------------------------------
var lProviderOptions = new Dictionary<string, string>();
lProviderOptions.Add("CompilerVersion", "v4.0");
var lCSharpCodeProvider = new CSharpCodeProvider(lProviderOptions);
var lCompilerParameters = new CompilerParameters();
lCompilerParameters.ReferencedAssemblies.Add("System.dll");
lCompilerParameters.ReferencedAssemblies.Add("System.Core.dll");
lCompilerParameters.ReferencedAssemblies.Add("System.Data.Linq.dll");
lCompilerParameters.ReferencedAssemblies.Add("System.Threading.dll");
lCompilerParameters.ReferencedAssemblies.Add("System.Web.Extensions.dll");
lCompilerParameters.ReferencedAssemblies.Add("System.Xml.Linq.dll");
lCompilerParameters.GenerateInMemory = true;
lCompilerParameters.GenerateExecutable = false; // not required, we don't have a Main() method
lCompilerParameters.IncludeDebugInformation = true;
var lCompilerResults = lCSharpCodeProvider.CompileAssemblyFromSource(lCompilerParameters, lCSharpSourceCode);
if (lCompilerResults.Errors.HasErrors) {
var lError = new StringBuilder();
foreach (CompilerError lCompilerError in lCompilerResults.Errors) {
lError.AppendLine(lCompilerError.ErrorNumber + " => " + lCompilerError.ErrorText + Environment.NewLine);
}
JsonOut.TextWrapping = TextWrapping.Wrap;
JsonOut.Text = lError.ToString();
return;
}
JsonOut.TextWrapping = TextWrapping.NoWrap;
// --------------------------------------------------
// execute the compiled code
// --------------------------------------------------
Assembly lAssembly = lCompilerResults.CompiledAssembly;
Type lProgram = lAssembly.GetType("baseClass");
MethodInfo lMethod = lProgram.GetMethod("executeLinq");
object lQueryResult = lMethod.Invoke(null, new object[] { lJsonIn }); // returns a JSON string object
// --------------------------------------------------
// rudimentary JSON output formatting
// --------------------------------------------------
string lJsonOut = lQueryResult.ToString();
lJsonOut = lJsonOut.Replace(",", ",\n");
lJsonOut = lJsonOut.Replace(",{", ",{\n");
lJsonOut = lJsonOut.Replace("]", "]\n");
JsonOut.Text = lJsonOut;
} //
private void ComboBox_SelectionChanged(object xSender, SelectionChangedEventArgs e) {
var lComboBox = xSender as ComboBox;
switch (lComboBox.SelectedIndex) {
case 0:
JsonIn.Text = "{\n \"number\": 108.541,\n \"datetime\": \"1975-03-13T10:30:00\" ,\n \"serialnumber\": \"SN1234\",\n \"more\": {\n \"field1\": 123,\n \"field2\": \"hello\"\n },\n \"array\": [\n {\"x\": 2.0},\n {\"x\": 3.0},\n {\"x\": 4.0}\n ]\n}";
LinqIn.Text = "from a in root.array\nwhere a.x > 2.0M\nselect a";
break;
case 1:
JsonIn.Text = "[ 1, 9, 5, 7, 1, 4 ]";
LinqIn.Text = "from a in root.home\nwhere ((a == 4) || (a == 1))\nselect a";
break;
case 2:
JsonIn.Text = "{myLuckyNumbers: [ 1, 9, 5, 26, 7, 1, 4 ]}";
LinqIn.Text = "from x in root.myLuckyNumbers\nwhere x % 2 == 0\nselect new { simple=x, square=x*x, text=\"Field\"+x.ToString(), classInClass=new {veryOdd=x/7.0, lol=\":D\"}}";
break;
case 3:
string s;
s = "{\"mainMenu\": {\n";
s += " \"info\": \"Great tool.\",\n";
s += " \"value\": 100.00,\n";
s += " \"menu\": {\n";
s += " \"subMenu\": [\n";
s += " {\"text\": \"New\", \"onclick\": \"NewObject()\"},\n";
s += " {\"text\": \"Open\", \"onclick\": \"Load()\"},\n";
s += " {\"text\": \"Close\", \"onclick\": \"ByeBye()\"},\n";
s += " {\"text\": \"NNN\", \"onclick\": \"Useless()\"}\n";
s += " ]}\n";
s += "}}\n";
JsonIn.Text = s;
LinqIn.Text = "root.mainMenu.menu.subMenu.Where(t => t.text.StartsWith(\"N\")).Last();";
break;
default:
break;
}
} //
} // class
} // namespace
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Web.Script.Serialization; // requires reference to System.Web.Extensions, used by the JavaScriptSerializer
namespace LinqJson {
public class Converter {
private Dictionary<string, object> _Classes = new Dictionary<string, object>();
public string GetUsings() {
string s;
s = "using System;\n";
s += "using System.Collections.Generic;\n";
s += "using System.Linq;\nusing System.Text;\n";
s += "using System.Threading.Tasks;\n";
s += "using System.Collections;\n";
s += "using System.Web.Script.Serialization;\n\n";
return s;
} //
public Dictionary<string, object> JsonToDictionary(string xJson) {
xJson = xJson.Trim();
if (xJson[0] == '[') xJson = "{home: " + xJson + "}"; // nameless arrays cannot be converted to dictionaries
var lJavaScriptSerializer = new JavaScriptSerializer();
try { return lJavaScriptSerializer.Deserialize<Dictionary<string, object>>(xJson); }
catch (Exception) { return null; }
} //
public void BuildClasses(StringBuilder xStringBuilder) {
foreach (var lClass in _Classes) {
Dictionary<string, object> lMembers = lClass.Value as Dictionary<string, object>;
if (lMembers == null) continue;
if (lMembers.Count <= 0) continue;
xStringBuilder.Append("public class Class_");
xStringBuilder.Append(lClass.Key);
xStringBuilder.AppendLine(" {");
foreach (var lMember in lMembers) {
object lValue = lMember.Value;
string lKey = lMember.Key;
Type lType = (lValue == null) ? typeof(object) : lMember.Value.GetType();
xStringBuilder.Append(new String(' ', 2));
xStringBuilder.Append("public ");
if (lType.IsValueType || (lValue is string)) {
xStringBuilder.Append(lType.Name);
xStringBuilder.Append(" ");
xStringBuilder.Append(lKey);
xStringBuilder.AppendLine(";");
}
else if (lValue is Dictionary<string, object>) {
xStringBuilder.Append("Class_");
xStringBuilder.Append(lKey);
xStringBuilder.Append(" ");
xStringBuilder.Append(lKey);
xStringBuilder.AppendLine(";");
}
else if (lValue is ArrayList) {
ArrayList lArrayList = lValue as ArrayList;
var lMemberType = ArrayListType(lArrayList); // differentiate between the contents of the list
if (lMemberType.IsValueType || (lMemberType.Name == "String")) {
//xStringBuilder.Append(lMemberType.Name.Replace("`2")); // Dictionaries use name "Dictionary`2"
xStringBuilder.Append(" List<"); xStringBuilder.Append(lMemberType.Name); xStringBuilder.Append("> ");
xStringBuilder.Append(lKey);
xStringBuilder.AppendLine(";");
}
else {
// a class
xStringBuilder.Append("List<Class_" + lKey + "> ");
xStringBuilder.Append(lKey);
xStringBuilder.AppendLine(";");
}
}
}
xStringBuilder.AppendLine(" }");
xStringBuilder.AppendLine();
}
} //
public void DictionariesToClasses(string xRootName, int xIndent, object xObject) {
if (xObject == null) return;
Type lType = xObject.GetType();
if (lType.IsValueType || (xObject is string)) return;
var lDictionary = xObject as Dictionary<string, object>;
if (lDictionary != null) {
object lObj;
if (!_Classes.TryGetValue(xRootName, out lObj)) {
_Classes.Add(xRootName, lDictionary);
}
else {
foreach (var lKeyValuePair in lDictionary) {
// This is a weakness of the program.
// Two JSON objects must not use the same name.
// We would have to compare the JSON objects and determine wether to create multiple or just one C# class.
_Classes[lKeyValuePair.Key] = lKeyValuePair.Value; // object type will be overridden !!!!!!
}
return;
}
foreach (var lKeyValuePair in lDictionary) {
DictionariesToClasses(lKeyValuePair.Key, xIndent, lKeyValuePair.Value);
}
return;
}
var lArrayList = xObject as ArrayList;
if (lArrayList != null) {
object lObj;
if (!_Classes.TryGetValue(xRootName, out lObj)) {
lDictionary = new Dictionary<string, object>();
_Classes.Add(xRootName, lDictionary);
}
else lDictionary = lObj as Dictionary<string, object>;
var lElementType = ArrayListType(lArrayList);
if (lElementType == typeof(Dictionary<string, object>)) {
var lList = lArrayList.Cast<Dictionary<string, object>>().ToList(); // upgrade our object to have stronger types
foreach (var lDict in lList) {
foreach (var lKeyValuePair in lDict) {
lDictionary[lKeyValuePair.Key] = lKeyValuePair.Value; // object type will be overridden !!!!!!
}
}
foreach (var lKeyValuePair in lDictionary) {
DictionariesToClasses(lKeyValuePair.Key, xIndent, lKeyValuePair.Value);
}
}
return;
}
} //
private void Append(StringBuilder xStringbuilder, int xIndent, string xString) {
xStringbuilder.Append(new String(' ', xIndent));
xStringbuilder.Append(xString);
} //
private void Newline(StringBuilder xStringbuilder) {
xStringbuilder.AppendLine();
} //
private Type ArrayListType(ArrayList xArrayList) {
var lTypes = new Dictionary<string, Type>();
Type lType;
string lTypeName;
foreach (object o in xArrayList) {
if (o == null) lType = typeof(object);
else lType = o.GetType();
lTypeName = lType.Name;
if (!lTypes.ContainsKey(lTypeName)) lTypes.Add(lTypeName, lType);
}
if (lTypes.Count == 1) return lTypes.Values.First(); // distinct
return typeof(object);
} //
} // class
} // namespace
Protocol Buffers (part 3, advanced), Tcp Networking
Ok guys, hardcore. Unfortunately in a different way than I thought 😉
Once again it took a lot of time to prepare the next source code example. It extends the code of my last post.
I removed the ProtoType class, because inheritance is not a real strength of Protobuf-Net. It is possible, but the ease of code maintenance is a definite argument against it. The complexity increases slowly, we cannot afford code that is difficult to change.
In theory we could write a Serialize() method for each class, the result would be extreme performance. You could use the BitConverter class to convert and then merge values without causing too much overhead. And by using Assembler I guess you could even make it 10 times faster again.
I think that all modern protocols lose a lot of time due to the required reflection, which is mainly the mapping of methods and properties. And unlike Assembler the modern languages do not allow to eg. simply write an integer to a memory address and only read the first byte of it.
You can rotate bits or multiply and divide, what is much less efficient.
In C# you can use the Parallel class to boost your performance. Of course the processing time of each code piece has to be substantial, otherwise it takes longer to create tasks than to solve the problems. In slow networks it might even be worth considering packing data before it is sent.
I replaced the ProtoType class by the Header class, which tells the type of the succeeding data block and also adds a serial identifier. This identifier can be used to receive vital feedbacks. For instance the server tells the client if the data transmission was successful or not. It can also send calculation results or error messages.
The data stream has the following order: header, data, header, data, header, data … except for feedback headers; they do not need data blocks. The enum eType inside the Header class defines all possible data types: ErrorMessage, Feedback, Book or Fable.
As said before I did not use inheritance for the Header class. The code remains legible and there is no spaghetti code to achieve indirect multiple inheritance.
In my last post, the client was sending and the server was receiving data. Now the communication is bidirectional. The client uses two threads, one to send and one to receive. The sending method is still using the BlockingCollection construction and an endless loop on a separate thread.
The server can be connected to several clients simultaneously. To keep it simple I decided sending data without any context switching. This usually blocks threads during the IO operation. I have added tasks inside the event OnMessageBook to give an example on how to avoid this. Nevertheless the send method uses a lock on the client preventing simultaneous write actions on the socket. This is bad practice; you should not apply locks on objects that might be locked in other areas as well. This is out of your scope, you don’t know if the .Net framework or any other code uses a lock on the same object, which could cause undesired behavior. In the below example it would have been better to wrap the client object into another class and apply the lock on that outer shell object. But I guess this is ok in our shorter example. The code was long enough already.
public static void Send(TcpClient xClient, ProtoBufExample.Header xHeader) {
if (xHeader == null) return;
if (xClient == null) return;
lock (xClient) {
NetworkStream lNetworkStream = xClient.GetStream();
....
The above lock problem could be avoided with this:
public class ClientData {
public TcpClient Client { get; private set; }
private DateTime _ConnectedSince;
private string _UserName;
....
public static void Send(ClientData xClient, ProtoBufExample.Header xHeader) {
if (xHeader == null) return;
if (xClient == null) return;
lock (xClient) {
NetworkStream lNetworkStream = xClient.Client.GetStream();
....
And here is the entire source code:
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using ProtoBuf;
namespace DemoApp {
public class ProtoBufExample {
public enum eType : byte { eError = 0, eFeedback, eBook, eFable };
[ProtoContract]
public class Header {
[ProtoMember(1)] public eType objectType;
[ProtoMember(2)] public readonly int serialMessageId;
public object data;
private static int _HeaderSerialId = 0;
public Header(object xData, eType xObjectType, int xSerialMessageId = 0) {
data = xData;
serialMessageId = (xSerialMessageId == 0) ? Interlocked.Increment(ref _HeaderSerialId) : xSerialMessageId;
objectType = xObjectType; // we could use "if typeof(T) ...", but it would be slower, harder to maintain and less legible
} // constructor
// parameterless constructor needed for Protobuf-net
public Header() {
} // constructor
} // class
[ProtoContract]
public class ErrorMessage {
[ProtoMember(1)]
public string Text;
} // class
[ProtoContract]
public class Book {
[ProtoMember(1)] public string author;
[ProtoMember(2, DataFormat = DataFormat.Group)] public List<Fable> stories;
[ProtoMember(3)] public DateTime edition;
[ProtoMember(4)] public int pages;
[ProtoMember(5)] public double price;
[ProtoMember(6)] public bool isEbook;
public override string ToString() {
StringBuilder s = new StringBuilder();
s.Append("by "); s.Append(author);
s.Append(", edition "); s.Append(edition.ToString("dd MMM yyyy"));
s.Append(", pages "); s.Append(pages);
s.Append(", price "); s.Append(price);
s.Append(", isEbook "); s.Append(isEbook);
s.AppendLine();
if (stories != null) foreach (Fable lFable in stories) {
s.Append("title "); s.Append(lFable.title);
s.Append(", rating "); s.Append(lFable.customerRatings.Average()); // Average() is an extension method of "using System.Linq;"
s.AppendLine();
}
return s.ToString();
} //
} // class
[ProtoContract]
public class Fable {
[ProtoMember(1)] public string title;
[ProtoMember(2, DataFormat = DataFormat.Group)]
public double[] customerRatings;
public override string ToString() {
return "title " + title + ", rating " + customerRatings.Average();
} //
} // class
public static Book GetData() {
return new Book {
author = "Aesop",
price = 1.99,
isEbook = false,
edition = new DateTime(1975, 03, 13),
pages = 203,
stories = new List<Fable>(new Fable[] {
new Fable{ title = "The Fox & the Grapes", customerRatings = new double[]{ 0.7, 0.7, 0.8} },
new Fable{ title = "The Goose that Laid the Golden Eggs", customerRatings = new double[]{ 0.6, 0.75, 0.5, 1.0} },
new Fable{ title = "The Cat & the Mice", customerRatings = new double[]{ 0.1, 0.0, 0.3} },
new Fable{ title = "The Mischievous Dog", customerRatings = new double[]{ 0.45, 0.5, 0.4, 0.0, 0.5} }
})
};
} //
} // class
public class PendingFeedbacks {
private readonly ConcurrentDictionary<int, ProtoBufExample.Header> _Messages = new ConcurrentDictionary<int, ProtoBufExample.Header>();
public int Count { get { return _Messages.Count; } }
public void Add(ProtoBufExample.Header xHeader) {
if (xHeader == null) throw new Exception("cannot add a null header");
if (!_Messages.TryAdd(xHeader.serialMessageId, xHeader)) {
throw new Exception("there must be a programming error somewhere");
}
} //
public void Remove(ProtoBufExample.Header xHeader) {
ProtoBufExample.Header lHeader;
if (!_Messages.TryRemove(xHeader.serialMessageId, out lHeader)) {
throw new Exception("there must be a programming error somewhere");
}
switch (xHeader.objectType) {
case ProtoBufExample.eType.eError:
Console.WriteLine("error: " + ((ProtoBufExample.ErrorMessage)xHeader.data).Text);
Console.WriteLine("the message that was sent out was: " + lHeader.objectType + " with serial id " + lHeader.serialMessageId);
Console.WriteLine("please check the log files" + Environment.NewLine);
break;
case ProtoBufExample.eType.eFeedback:
// all ok !
break;
default:
Console.WriteLine("warning: This message type was not expected.");
break;
}
} //
} // class
public static class NetworkTest {
public static void Test() {
NetworkListener lServer = new NetworkListener("127.0.0.1", 65432, "Server");
NetworkClient lClient = new NetworkClient("127.0.0.1", 65432, "Client");
lServer.Connect();
lServer.OnMessageBook += new NetworkListener.dOnMessageBook(OnMessageBook);
lServer.OnMessageFable += new NetworkListener.dOnMessageFable(OnMessageFable);
lClient.Connect();
ProtoBufExample.Header lHeader;
// send a book across the network
ProtoBufExample.Book lBook = ProtoBufExample.GetData();
lHeader = new ProtoBufExample.Header(lBook, ProtoBufExample.eType.eBook);
lClient.Send(lHeader);
System.Threading.Thread.Sleep(1000); // remove this to see the asynchonous processing (the output will look terrible)
// send a fable across the network
lHeader = new ProtoBufExample.Header(lBook.stories[1], ProtoBufExample.eType.eFable);
lClient.Send(lHeader);
System.Threading.Thread.Sleep(1000);
lClient.Disconnect();
lServer.Disconnect();
Console.ReadLine();
}
// demo: synchronous processing
static void OnMessageFable(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Fable xFable) {
Console.WriteLine(Environment.NewLine + "received a fable: ");
Console.WriteLine(xFable.ToString());
// demo: we tell the server that something went wrong
ProtoBufExample.ErrorMessage lErrorMessage = new ProtoBufExample.ErrorMessage() { Text = "The fable was rejected. It is far too short." };
ProtoBufExample.Header lErrorHeader = new ProtoBufExample.Header(lErrorMessage, ProtoBufExample.eType.eError, xHeader.serialMessageId);
NetworkListener.Send(xSender, lErrorHeader);
} //
// demo: asynchronous processing
static void OnMessageBook(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Book xBook) {
Task.Factory.StartNew(() => {
Console.WriteLine(Environment.NewLine + "received a book: ");
Console.WriteLine(xBook.ToString());
// send a feedback without any body to signal all was ok
ProtoBufExample.Header lFeedback = new ProtoBufExample.Header(null, ProtoBufExample.eType.eFeedback, xHeader.serialMessageId);
NetworkListener.Send(xSender, lFeedback);
return;
});
Console.WriteLine("Book event was raised");
} //
} // class
public class NetworkListener {
private bool _ExitLoop = true;
private TcpListener _Listener;
public delegate void dOnMessageBook(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Book xBook);
public event dOnMessageBook OnMessageBook;
public delegate void dOnMessageFable(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Fable xFable);
public event dOnMessageFable OnMessageFable;
private List<TcpClient> _Clients = new List<TcpClient>();
public int Port { get; private set; }
public string IpAddress { get; private set; }
public string ThreadName { get; private set; }
public NetworkListener(string xIpAddress, int xPort, string xThreadName) {
Port = xPort;
IpAddress = xIpAddress;
ThreadName = xThreadName;
} //
public bool Connect() {
if (!_ExitLoop) {
Console.WriteLine("Listener running already");
return false;
}
_ExitLoop = false;
try {
_Listener = new TcpListener(IPAddress.Parse(IpAddress), Port);
_Listener.Start();
Thread lThread = new Thread(new ThreadStart(LoopWaitingForClientsToConnect));
lThread.IsBackground = true;
lThread.Name = ThreadName + "WaitingForClients";
lThread.Start();
return true;
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
return false;
} //
public void Disconnect() {
_ExitLoop = true;
lock (_Clients) {
foreach (TcpClient lClient in _Clients) lClient.Close();
_Clients.Clear();
}
} //
private void LoopWaitingForClientsToConnect() {
try {
while (!_ExitLoop) {
Console.WriteLine("waiting for a client");
TcpClient lClient = _Listener.AcceptTcpClient();
string lClientIpAddress = lClient.Client.LocalEndPoint.ToString();
Console.WriteLine("new client connecting: " + lClientIpAddress);
if (_ExitLoop) break;
lock (_Clients) _Clients.Add(lClient);
Thread lThread = new Thread(new ParameterizedThreadStart(LoopRead));
lThread.IsBackground = true;
lThread.Name = ThreadName + "CommunicatingWithClient";
lThread.Start(lClient);
}
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
finally {
_ExitLoop = true;
if (_Listener != null) _Listener.Stop();
}
} //
private void LoopRead(object xClient) {
TcpClient lClient = xClient as TcpClient;
NetworkStream lNetworkStream = lClient.GetStream();
while (!_ExitLoop) {
try {
ProtoBufExample.Header lHeader = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Header>(lNetworkStream, ProtoBuf.PrefixStyle.Fixed32);
if (lHeader == null) break; // happens during shutdown process
switch (lHeader.objectType) {
case ProtoBufExample.eType.eBook:
ProtoBufExample.Book lBook = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Book>(lNetworkStream, ProtoBuf.PrefixStyle.Fixed32);
if (lBook == null) break;
lHeader.data = lBook; // not necessary, but nicer
dOnMessageBook lEventBook = OnMessageBook;
if (lEventBook == null) continue;
lEventBook(lClient, lHeader, lBook);
break;
case ProtoBufExample.eType.eFable:
ProtoBufExample.Fable lFable = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Fable>(lNetworkStream, ProtoBuf.PrefixStyle.Fixed32);
if (lFable == null) break;
lHeader.data = lFable; // not necessary, but nicer
dOnMessageFable lEventFable = OnMessageFable;
if (lEventFable == null) continue;
lEventFable(lClient, lHeader, lFable);
break;
default:
Console.WriteLine("Mayday, mayday, we are in big trouble.");
break;
}
}
catch (System.IO.IOException) {
if (_ExitLoop) Console.WriteLine("user requested client shutdown");
else Console.WriteLine("disconnected");
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
Console.WriteLine("server: listener is shutting down");
} //
public static void Send(TcpClient xClient, ProtoBufExample.Header xHeader) {
if (xHeader == null) return;
if (xClient == null) return;
lock (xClient) {
try {
NetworkStream lNetworkStream = xClient.GetStream();
// send header (most likely a simple feedback)
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Header>(lNetworkStream, xHeader, ProtoBuf.PrefixStyle.Fixed32);
// send errors
if (xHeader.objectType != ProtoBufExample.eType.eError) return;
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.ErrorMessage>(lNetworkStream, (ProtoBufExample.ErrorMessage)xHeader.data, ProtoBuf.PrefixStyle.Fixed32);
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
} //
} // class
public class NetworkClient {
public int Port { get; private set; }
public string IpAddress { get; private set; }
public string ThreadName { get; private set; }
private NetworkStream _NetworkStream = null;
private TcpClient _Client = null;
private bool _ExitLoop = true;
private BlockingCollection<ProtoBufExample.Header> _Queue = new BlockingCollection<ProtoBufExample.Header>();
private readonly PendingFeedbacks _PendingFeedbacks = new PendingFeedbacks();
public NetworkClient(string xIpAddress, int xPort, string xThreadName) {
Port = xPort;
IpAddress = xIpAddress;
ThreadName = xThreadName;
} //
public void Connect() {
if (!_ExitLoop) return; // running already
_ExitLoop = false;
_Client = new TcpClient();
_Client.Connect(IpAddress, Port);
_NetworkStream = _Client.GetStream();
Thread lLoopWrite = new Thread(new ThreadStart(LoopWrite));
lLoopWrite.IsBackground = true;
lLoopWrite.Name = ThreadName + "Write";
lLoopWrite.Start();
Thread lLoopRead = new Thread(new ThreadStart(LoopRead));
lLoopRead.IsBackground = true;
lLoopRead.Name = ThreadName + "Read";
lLoopRead.Start();
} //
public void Disconnect() {
_ExitLoop = true;
_Queue.Add(null);
if (_Client != null) _Client.Close();
//if (_NetworkStream != null) _NetworkStream.Close();
} //
public void Send(ProtoBufExample.Header xHeader) {
if (xHeader == null) return;
_PendingFeedbacks.Add(xHeader);
_Queue.Add(xHeader);
} //
private void LoopWrite() {
while (!_ExitLoop) {
try {
ProtoBufExample.Header lHeader = _Queue.Take();
if (lHeader == null) break;
// send header
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Header>(_NetworkStream, lHeader, ProtoBuf.PrefixStyle.Fixed32);
// send data
switch (lHeader.objectType) {
case ProtoBufExample.eType.eBook:
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Book>(_NetworkStream, (ProtoBufExample.Book)lHeader.data, ProtoBuf.PrefixStyle.Fixed32);
break;
case ProtoBufExample.eType.eFable:
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Fable>(_NetworkStream, (ProtoBufExample.Fable)lHeader.data, ProtoBuf.PrefixStyle.Fixed32);
break;
default:
break;
}
}
catch (System.IO.IOException) {
if (_ExitLoop) Console.WriteLine("user requested client shutdown.");
else Console.WriteLine("disconnected");
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
_ExitLoop = true;
Console.WriteLine("client: writer is shutting down");
} //
private void LoopRead() {
while (!_ExitLoop) {
try {
ProtoBufExample.Header lHeader = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Header>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
if (lHeader == null) break;
if (lHeader.objectType == ProtoBufExample.eType.eError) {
ProtoBufExample.ErrorMessage lErrorMessage = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.ErrorMessage>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
lHeader.data = lErrorMessage;
}
_PendingFeedbacks.Remove(lHeader);
}
catch (System.IO.IOException) {
if (_ExitLoop) Console.WriteLine("user requested client shutdown");
else Console.WriteLine("disconnected");
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
Console.WriteLine("client: reader is shutting down");
} //
} // class
} // namespace
example output:
waiting for a client
new client connecting: 127.0.0.1:65432
waiting for a client
Book event was raisedreceived a book:
by Aesop, edition 13 Mar 1975, pages 203, price 1.99, isEbook False
title The Fox & the Grapes, rating 0.733333333333333
title The Goose that Laid the Golden Eggs, rating 0.7125
title The Cat & the Mice, rating 0.133333333333333
title The Mischievous Dog, rating 0.37received a fable:
title The Goose that Laid the Golden Eggs, rating 0.7125
error: The fable was rejected. It is far too short.
the message that was sent out was: eFable with serial id 2
please check the log filesclient: writer is shutting down
server: listener is shutting down
user requested client shutdown
client: reader is shutting down
Protocol Buffers (part 2, advanced), Tcp Networking
There has been no post yesterday. As usual I am preparing the source code for you during my spare time. This is not easy in my current environment. I will have to slow down the post frequency as the source codes are getting longer and more complex.
So, what is on the agenda now? Today we are going to send serialized binary data across the network. I am using the localhost to have a running example. Simply amend the local IP-addresses “127.0.0.1” to whatever suits you.
I wrote a small class (“ProtoType”) to identify objects quickly during the serialization and deserialization process. It is abstract and forms our base class. The class “Book” and “Fable” are nearly unchanged from the previous posts. I just added an override “ToString” in “Fable” to have a nicer output. Both classes must inherit from our abstract base class called “ProtoType”, which avoids implementing an interface and impacting the code legibility more than needed.
public class ProtoBufExample {
public enum eType { Unknown = 0, eBook = 1, eFable };
public abstract class ProtoType {
public readonly eType objectId;
public readonly byte[] objectIdAsBytes;
public ProtoType() {
Type t = this.GetType();
if (t == typeof(ProtoBufExample.Book)) objectId = eType.eBook; // to identify the object before deserialization
else if (t == typeof(ProtoBufExample.Fable)) objectId = eType.eFable; // to identify the object before deserialization
else throw new Exception("object type unknown");
objectIdAsBytes = BitConverter.GetBytes((Int16)objectId);
} // constructor
} // class
[ProtoContract]
public class Book : ProtoType {
[ProtoMember(1)]
public string author;
[ProtoMember(2)]
public List<Fable> stories;
[ProtoMember(3)]
public DateTime edition;
[ProtoMember(4)]
public int pages;
[ProtoMember(5)]
public double price;
[ProtoMember(6)]
public bool isEbook;
public override string ToString() {
StringBuilder s = new StringBuilder();
s.Append("by "); s.Append(author);
s.Append(", edition "); s.Append(edition.ToString("dd MMM yyyy"));
s.Append(", pages "); s.Append(pages);
s.Append(", price "); s.Append(price);
s.Append(", isEbook "); s.Append(isEbook);
s.AppendLine();
if (stories != null) foreach (Fable lFable in stories) {
s.Append("title "); s.Append(lFable.title);
s.Append(", rating "); s.Append(lFable.customerRatings.Average()); // Average() is an extension method of "using System.Linq;"
s.AppendLine();
}
return s.ToString();
} //
} // class
[ProtoContract]
public class Fable : ProtoType {
[ProtoMember(1)]
public string title;
[ProtoMember(2)]
public double[] customerRatings;
public override string ToString() {
return "title " + title + ", rating " + customerRatings.Average();
} //
} // class
public static Book GetData() {
return new Book {
author = "Aesop",
price = 1.99,
isEbook = false,
edition = new DateTime(1975, 03, 13),
pages = 203,
stories = new List<Fable>(new Fable[] {
new Fable{ title = "The Fox & the Grapes", customerRatings = new double[]{ 0.7, 0.7, 0.8} },
new Fable{ title = "The Goose that Laid the Golden Eggs", customerRatings = new double[]{ 0.6, 0.75, 0.5, 1.0} },
new Fable{ title = "The Cat & the Mice", customerRatings = new double[]{ 0.1, 0.0, 0.3} },
new Fable{ title = "The Mischievous Dog", customerRatings = new double[]{ 0.45, 0.5, 0.4, 0.0, 0.5} }
})
};
} //
} // class
The server-side always needs to be started before the client. Otherwise the client would not receive an immediate response from the server. We set up a listener on localhost port 65432 (arbitrary choice) and wait for a client connection. I chose the TCP protocol. It is easy to use and very reliable. We do not need to care about transmission problems. The downside is slowness. I will cover that one in comming posts.
The deserializer on the server-side cannot be run right away. We need to determine the object type first. And here we make use of our “ProtoType” class. The first two bytes of the transmission tell us the type. This is quite important for the speed of the deserialization. You see that the ProtoBuf.Serializer.DeserializeWithLengthPrefix method is generic. The type casting is minimized that way.
public class NetworkListener {
private bool _ExitLoop = true;
private TcpListener _Listener;
public delegate void dOnMessage(object xSender, ProtoBufExample.Book xBook);
public event dOnMessage OnMessage;
private NetworkStream _NetworkStream = null;
public int Port { get; private set; }
public string IpAddress { get; private set; }
public string ThreadName { get; private set; }
public NetworkListener(string xIpAddress, int xPort, string xThreadName) {
Port = xPort;
IpAddress = xIpAddress;
ThreadName = xThreadName;
} //
public bool Connect() {
if (!_ExitLoop) {
Console.WriteLine("Listener running already");
return false;
}
_ExitLoop = false;
try {
_Listener = new TcpListener(IPAddress.Parse(IpAddress), Port);
_Listener.Start();
Thread lThread = new Thread(new ThreadStart(Loop));
lThread.IsBackground = true;
lThread.Name = ThreadName;
lThread.Start();
return true;
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
return false;
} //
public void Disconnect() {
_ExitLoop = true;
_NetworkStream.WriteTimeout = 5; // immediate timeout
} //
private void Loop() {
try {
while (!_ExitLoop) {
Console.WriteLine("Waiting for a client");
using (TcpClient lClient = _Listener.AcceptTcpClient()) {
string lClientIpAddress = lClient.Client.LocalEndPoint.ToString();
Console.WriteLine("New client connecting: " + lClientIpAddress);
using (_NetworkStream = lClient.GetStream()) {
while (!_ExitLoop) {
try {
byte[] lHeader = new byte[2]; // to indentify the object
if (_NetworkStream.Read(lHeader, 0, 2) != 2) break;
int lObjectType = BitConverter.ToInt16(lHeader, 0);
ProtoBufExample.eType lType = (ProtoBufExample.eType)lObjectType;
switch (lType) {
case ProtoBufExample.eType.Unknown:
break;
case ProtoBufExample.eType.eBook:
ProtoBufExample.Book lBook = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Book>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
Console.WriteLine(Environment.NewLine + "received a book: ");
Console.WriteLine(lBook.ToString());
// raise an event
dOnMessage lEvent = OnMessage;
if (lEvent == null) continue;
lEvent(lClient, lBook);
break;
case ProtoBufExample.eType.eFable:
ProtoBufExample.Fable lFable = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Fable>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
Console.WriteLine(Environment.NewLine + "received a fable: ");
Console.WriteLine(lFable.ToString());
break;
default:
Console.WriteLine("Mayday, mayday, we are in big trouble.");
break;
}
}
catch (System.IO.IOException) {
if (_ExitLoop) Console.WriteLine("user requested TcpClient shutdown");
else Console.WriteLine("disconnected");
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
Console.WriteLine(Environment.NewLine + "server/listener: shutting down");
}
}
}
}
catch (Exception ex) {
Console.WriteLine(ex.Message);
}
finally {
_ExitLoop = true;
if (_Listener != null) _Listener.Stop();
}
} //
} // class
The client side is straight forward. There is not much processing besides choosing the correct generic type for the serialization process. I used a BlockingCollection to make the context switching easier to read. It is not the fastest solution, but it definitely makes passing objects into a thread loop easy. I am personally no fan of the concurrent collections. They are not as predictable as any custom solution. _Queue.Take(); blocks and waits for data to arrive. It is thread-safe and does not require any object locking.
public class NetworkClient {
public int Port { get; private set; }
public string IpAddress { get; private set; }
public string ThreadName { get; private set; }
private NetworkStream _NetworkStream = null;
private bool _ExitLoop = true;
private BlockingCollection<ProtoBufExample.ProtoType> _Queue = new BlockingCollection<ProtoBufExample.ProtoType>();
public NetworkClient(string xIpAddress, int xPort, string xThreadName) {
Port = xPort;
IpAddress = xIpAddress;
ThreadName = xThreadName;
} //
public void Connect() {
if (!_ExitLoop) return; // running already
_ExitLoop = false;
Thread lThread = new Thread(new ThreadStart(Loop));
lThread.IsBackground = true;
lThread.Name = ThreadName;
lThread.Start();
} //
public void Disconnect() {
_ExitLoop = true;
_Queue.Add(null);
if (_NetworkStream != null) _NetworkStream.ReadTimeout = 100;
} //
public void Send(ProtoBufExample.ProtoType xObject) {
if (xObject == null) return;
_Queue.Add(xObject);
} //
private void Loop() {
try {
using (TcpClient lClient = new TcpClient()) {
lClient.Connect(IpAddress, Port);
using (_NetworkStream = lClient.GetStream()) {
while (!_ExitLoop) {
try {
ProtoBufExample.ProtoType lObject = _Queue.Take();
if (lObject == null) break;
switch (lObject.objectId) {
case ProtoBufExample.eType.eBook:
_NetworkStream.Write(lObject.objectIdAsBytes, 0, 2);
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Book>(_NetworkStream, (ProtoBufExample.Book)lObject, ProtoBuf.PrefixStyle.Fixed32);
break;
case ProtoBufExample.eType.eFable:
_NetworkStream.Write(lObject.objectIdAsBytes, 0, 2);
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Fable>(_NetworkStream, (ProtoBufExample.Fable)lObject, ProtoBuf.PrefixStyle.Fixed32);
break;
default:
break;
}
}
catch (System.IO.IOException) {
if (_ExitLoop) Console.WriteLine("user requested TcpClient shutdown.");
else Console.WriteLine("disconnected");
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
_ExitLoop = true;
Console.WriteLine(Environment.NewLine + "client: shutting down");
}
}
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
} //
} // class
The main program is pretty neat. We get the “Book” data and send it across the localhost. Then we take a single story of the same book and send it again to see if the program deals with the different types properly. Et voilà , it does.
The framing of the serialized data is very important. If you do not frame it, then you cannot determine the type. There are weird examples on the internet. My quick google research showed that people generally do not implement that centerpiece, leaving beginners in the middle of nowhere.
public static class NetworkTest {
public static void Test() {
NetworkListener lServer = new NetworkListener("127.0.0.1", 65432, "Server");
NetworkClient lClient = new NetworkClient("127.0.0.1", 65432, "Client");
lServer.Connect();
lServer.OnMessage += new NetworkListener.dOnMessage(OnBook);
lClient.Connect();
// send a book across the network
ProtoBufExample.Book lBook = ProtoBufExample.GetData();
lClient.Send(lBook);
// send a fable across the network
lClient.Send(lBook.stories[1]);
System.Threading.Thread.Sleep(1000);
lClient.Disconnect();
lServer.Disconnect();
Console.ReadLine();
}
static void OnBook(object xSender, ProtoBufExample.Book xBook) {
Console.WriteLine("Book event was raised");
} //
} //
example output:
Waiting for a client
New client connecting: 127.0.0.1:65432received a book:
by Aesop, edition 13 Mar 1975, pages 203, price 1.99, isEbook False
title The Fox & the Grapes, rating 0.733333333333333
title The Goose that Laid the Golden Eggs, rating 0.7125
title The Cat & the Mice, rating 0.133333333333333
title The Mischievous Dog, rating 0.37Book event was raised
received a fable:
title The Goose that Laid the Golden Eggs, rating 0.7125client: shutting down
server/listener: shutting down
There is one tiny issue with the OnBook event. We receive data on a thread and block the thread as long as it takes to process the event. To avoid thread bottlenecks you should think about forwarding “Book” events to tasks and let them deal with whatever needs to be done asynchronously. I posted about tasks eg. here. ThreadPools and the Parallel class might also be interesting posts in this context.
Protocol Buffers (part 1, basics), follow-up to JSON & XML
I quickly introduced Protocol Buffers in my last post. It is an amazing tool that every programmer should know. The .Net version protobuf-net can be downloaded from here.
On its homepage Protocol Buffers is described with the following words:
Protocol buffers is the name of the binary serialization format used by Google for much of their data communications. It is designed to be:
small in size – efficient data storage (far smaller than xml)
cheap to process – both at the client and server
platform independent – portable between different programming architectures
extensible – to add new data to old messages
This is succinct. Protobuf-net also supports the WCF.
Add the protobuf-net.dll, which is located in the net30 installation directory, to your solution references.
We need some data structure to start with.
public class Book {
public string author;
public List<Fable> stories;
public DateTime edition;
public int pages;
public double price;
public bool isEbook;
} // class
public class Fable {
public string title;
public double[] customerRatings;
} // class
Protocol Buffers uses attributes to identify types to serialize. The ProtoMember attribute needs a positive integer. This can be painful, because you have to avoid overlapping when using inheritance. But integers have a distinct advantage as well. They are much faster than strings. As you are already aware, Protocol Buffers is about speed.
[ProtoContract]
public class Book {
[ProtoMember(1)]
public string author;
[ProtoMember(2)]
public List stories;
[ProtoMember(3)]
public DateTime edition;
[ProtoMember(4)]
public int pages;
[ProtoMember(5)]
public double price;
[ProtoMember(6)]
public bool isEbook;
public override string ToString() {
StringBuilder s = new StringBuilder();
s.Append("by "); s.Append(author);
s.Append(", edition "); s.Append(edition.ToString("dd MMM yyyy"));
s.Append(", pages "); s.Append(pages);
s.Append(", price "); s.Append(price);
s.Append(", isEbook "); s.Append(isEbook);
s.AppendLine();
if (stories != null) foreach (Fable lFable in stories) {
s.Append("title "); s.Append(lFable.title);
s.Append(", rating "); s.Append(lFable.customerRatings.Average());
s.AppendLine();
}
return s.ToString();
} //
} // class
[ProtoContract]
public class Fable {
[ProtoMember(1)]
public string title;
[ProtoMember(2)]
public double[] customerRatings;
} // class
public static Book GetData() {
return new Book {
author = "Aesop",
price = 1.99,
isEbook = false,
edition = new DateTime(1975, 03, 13),
pages = 203,
stories = new List<Fable>(new Fable[] {
new Fable{ title = "The Fox & the Grapes", customerRatings = new double[]{ 0.7, 0.7, 0.8} },
new Fable{ title = "The Goose that Laid the Golden Eggs", customerRatings = new double[]{ 0.6, 0.75, 0.5, 1.0} },
new Fable{ title = "The Cat & the Mice", customerRatings = new double[]{ 0.1, 0.0, 0.3} },
new Fable{ title = "The Mischievous Dog", customerRatings = new double[]{ 0.45, 0.5, 0.4, 0.0, 0.5} }
})
};
} //
Let’s serialize the data now.
public static void SerializeData() {
MemoryStream lStream = new MemoryStream();
BinaryWriter lWriter = new BinaryWriter(lStream); // no "using", because it would close the MemoryStream automatically
Book lBook = GetData();
ProtoBuf.Serializer.Serialize<Book>(lStream, lBook);
lWriter.Flush();
lStream.Position = 0;
using (BinaryReader lReader = new BinaryReader(lStream)) {
for (long i = 0, n = lStream.Length; i < n; i++) {
byte b = lReader.ReadByte();
Console.Write(string.Format("{0:X2} ", b));
if ((i+1) % 20 == 0) Console.WriteLine();
}
Console.WriteLine();
Console.WriteLine();
Console.WriteLine("number of bytes: " + lStream.Length);
}
Console.ReadLine();
} //
example output:
0A 05 41 65 73 6F 70 12 31 0A 14 54 68 65 20 46 6F 78 20 26
20 74 68 65 20 47 72 61 70 65 73 11 66 66 66 66 66 66 E6 3F
11 66 66 66 66 66 66 E6 3F 11 9A 99 99 99 99 99 E9 3F 12 49
0A 23 54 68 65 20 47 6F 6F 73 65 20 74 68 61 74 20 4C 61 69
64 20 74 68 65 20 47 6F 6C 64 65 6E 20 45 67 67 73 11 33 33
33 33 33 33 E3 3F 11 00 00 00 00 00 00 E8 3F 11 00 00 00 00
00 00 E0 3F 11 00 00 00 00 00 00 F0 3F 12 2F 0A 12 54 68 65
20 43 61 74 20 26 20 74 68 65 20 4D 69 63 65 11 9A 99 99 99
99 99 B9 3F 11 00 00 00 00 00 00 00 00 11 33 33 33 33 33 33
D3 3F 12 42 0A 13 54 68 65 20 4D 69 73 63 68 69 65 76 6F 75
73 20 44 6F 67 11 CD CC CC CC CC CC DC 3F 11 00 00 00 00 00
00 E0 3F 11 9A 99 99 99 99 99 D9 3F 11 00 00 00 00 00 00 00
00 11 00 00 00 00 00 00 E0 3F 1A 03 08 D2 1D 20 CB 01 29 D7
A3 70 3D 0A D7 FF 3Fnumber of bytes: 267
And back again: deserialize data.
public static void ToAndFro() {
using (MemoryStream lStream = new MemoryStream()) {
BinaryWriter lWriter = new BinaryWriter(lStream);
Book lBook = GetData();
ProtoBuf.Serializer.Serialize<Book>(lStream, lBook);
lWriter.Flush();
lStream.Position = 0;
Book lCopy = ProtoBuf.Serializer.Deserialize<Book>(lStream);
Console.WriteLine(lCopy.ToString());
}
Console.ReadLine();
} //
example output:
by Aesop, edition 13 Mar 1975, pages 203, price 1.99, isEbook False
title The Fox & the Grapes, rating 0.733333333333333
title The Goose that Laid the Golden Eggs, rating 0.7125
title The Cat & the Mice, rating 0.133333333333333
title The Mischievous Dog, rating 0.37
C# Hardcore Programming 