Category Archives: Streams

JSON to LINQ to JSON

JsonLinqJson

It is about 9:30pm and I am sitting on the train home. It has been quite tough organizing my life in the last months. This is not the promised follow-up on OpenGL. There has been no time to read any book.

Anyway, this post shows how to run LINQ queries on JSON. The use can be quite broad. Imagine a shell command executing your LINQ on any JSON file. You can query into depth and build complex return tree structures with 2 or more levels. LINQ itself is highly flexible. And – just to give it the right flair – it returns results in the good old JSON format.

You could even go a bit further and insert more than just a LINQ query. In theory, you could add any C# code at run-time. The changes to this program would be minuscule.

I have sufficiently commented the code. There is a minimum overhead to get the job done.

 

How it works:

  1. The JSON data is imported using the JavaScriptSerializer, which is part of the System.Web.Extensions library reference. We try to force the result into a Dictionary<string, object>. Do not lazily use ‘object’. Some type information would get lost. I played with this and got eg. arrays instead of ArrayLists.
  2. The resulting structure is then used to build classes. These are converted to legible C# source code.
  3. The LINQ command is inserted. You can see the result in the TextBox titled ‘C# source code output’.
  4. A compiler instance is created and some library references are added. We compile the generated source code at run-time and then execute the binary.
  5. A standard JSON object is returned, rudimentary formatted and displayed as the final result.

Btw. I have inserted a horizontal and a vertical GridSplitter into the WPF XAML. You can easily change the size of the TextBoxes while playing with this tool. A few JSON examples were also added.

<Window         xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"         xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"         xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" mc:Ignorable="d" x:Class="LinqJson.MainWindow"   Title="JSON LINQ JSON" Height="Auto" Width="876" d:DesignHeight="695">
  <Grid>
    <Grid.RowDefinitions>
      <RowDefinition Height="1*"/>
      <RowDefinition Height="43*"/>
      <RowDefinition Height="30"/>
      <RowDefinition Height="150*"/>
      <RowDefinition Height="30"/>
      <RowDefinition Height="150*"/>
      <RowDefinition Height="10*"/>
    </Grid.RowDefinitions>
    <Grid.ColumnDefinitions>
      <ColumnDefinition Width="10*"/>
      <ColumnDefinition Width="250*"/>
      <ColumnDefinition Width="10"/>
      <ColumnDefinition Width="250*"/>
      <ColumnDefinition Width="10*"/>
      <ColumnDefinition/>
    </Grid.ColumnDefinitions>
    <Button Content="convert" Width="50" Height="30" Click="Button_Click" Grid.Row="1" HorizontalAlignment="Left" Grid.Column="1" Margin="0,8,0,7"/>

    <StackPanel Grid.Column="1" Grid.Row="2" VerticalAlignment="Bottom" HorizontalAlignment="Left" Orientation="Horizontal" Height="27" >
      <Label Content="JSON input" VerticalAlignment="Bottom" HorizontalAlignment="Left" Margin="0,1" />
      <ComboBox Width="240" Margin="50,3,3,3" SelectionChanged="ComboBox_SelectionChanged" VerticalAlignment="Bottom">
        <ComboBoxItem  Name="ex1">Example 1</ComboBoxItem>
        <ComboBoxItem  Name="ex2">Example 2</ComboBoxItem>
        <ComboBoxItem  Name="ex3">Example 3</ComboBoxItem>
        <ComboBoxItem  Name="ex4">Example 4</ComboBoxItem>
      </ComboBox>
    </StackPanel>
    <TextBox  x:Name="JsonIn" HorizontalScrollBarVisibility="Auto" VerticalScrollBarVisibility="Auto" Grid.Column="1" Grid.Row="3"/>

    <GridSplitter Grid.Column="2" Grid.RowSpan="999" HorizontalAlignment="Stretch" Width="10" Background="Transparent" ResizeBehavior="PreviousAndNext"/>
    <GridSplitter Grid.Row="4" Grid.ColumnSpan="999" VerticalAlignment="Top" HorizontalAlignment="Stretch"  Height="10" Background="Transparent" ResizeBehavior="PreviousAndNext" />

    <Label Content="JSON output" Grid.Column="3" Grid.Row="2"  VerticalAlignment="Bottom" />
    <TextBox x:Name="JsonOut" VerticalAlignment="Stretch" HorizontalScrollBarVisibility="Auto" VerticalScrollBarVisibility="Auto" Grid.Column="3"  Grid.Row="3"  />

    <Label Content="C# source code output" Grid.Column="3" Grid.Row="4"  VerticalAlignment="Bottom" />
    <TextBox x:Name="CSharpSourceCode" HorizontalScrollBarVisibility="Auto" VerticalScrollBarVisibility="Auto" Grid.Column="3" Grid.Row="5" />
    <Label Content="LINQ input" Grid.Column="1" Grid.Row="4"  VerticalAlignment="Bottom"/>
    <TextBox x:Name="LinqIn" HorizontalScrollBarVisibility="Auto" VerticalScrollBarVisibility="Auto" Grid.Column="1" Grid.Row="5" />

  </Grid>

</Window>
using Microsoft.CSharp;
using System;
using System.CodeDom.Compiler;
using System.Collections.Generic;
using System.Reflection;
using System.Text;
using System.Windows;
using System.Windows.Controls;

namespace LinqJson {
  public partial class MainWindow : Window {

    public MainWindow() {
      InitializeComponent();
    } // constructor

    private void Button_Click(object sender, RoutedEventArgs e) {

      // --------------------------------------------------
      // get the JSON input
      // --------------------------------------------------

      string lJsonIn = JsonIn.Text;

      // --------------------------------------------------
      // construct the C# source code (class hierarchy)
      // --------------------------------------------------

      Converter lConverter = new Converter();
      Dictionary<string, object> lTree = lConverter.JsonToDictionary(lJsonIn);
      lConverter.DictionariesToClasses("root", 0, lTree);

      var lStringBuilder = new StringBuilder();
      lConverter.BuildClasses(lStringBuilder);
      string lCSharpSourceCode = lConverter.GetUsings() + lStringBuilder.ToString();

      // --------------------------------------------------
      // add the LINQ command to the source code
      // --------------------------------------------------

      string lLinq = LinqIn.Text;
      string lEntryPoint = "\n\n";
      lEntryPoint += "public class baseClass {\n";
      lEntryPoint += "  public static object executeLinq(string xJson) {\n";
      lEntryPoint += "    xJson = xJson.Trim();";
      lEntryPoint += "    if (xJson[0] == '[') xJson = \"{home: \" + xJson + \"}\";";
      lEntryPoint += "    var lSerializer = new JavaScriptSerializer();\n";
      lEntryPoint += "    var root = lSerializer.Deserialize<Class_root>(xJson);\n";
      lEntryPoint += "    var lResult = " + lLinq.Replace("\n", "\n        ") + ";\n";
      lEntryPoint += "    return lSerializer.Serialize(lResult);\n";
      lEntryPoint += "  }\n";
      lEntryPoint += "}\n";
      lCSharpSourceCode += lEntryPoint;

      // --------------------------------------------------
      // display the source code
      // --------------------------------------------------

      CSharpSourceCode.Text = lCSharpSourceCode;

      // --------------------------------------------------
      // compile the source code
      // --------------------------------------------------

      var lProviderOptions = new Dictionary<string, string>();
      lProviderOptions.Add("CompilerVersion", "v4.0");
      var lCSharpCodeProvider = new CSharpCodeProvider(lProviderOptions);
      var lCompilerParameters = new CompilerParameters();

      lCompilerParameters.ReferencedAssemblies.Add("System.dll");
      lCompilerParameters.ReferencedAssemblies.Add("System.Core.dll");
      lCompilerParameters.ReferencedAssemblies.Add("System.Data.Linq.dll");
      lCompilerParameters.ReferencedAssemblies.Add("System.Threading.dll");
      lCompilerParameters.ReferencedAssemblies.Add("System.Web.Extensions.dll");
      lCompilerParameters.ReferencedAssemblies.Add("System.Xml.Linq.dll");
      lCompilerParameters.GenerateInMemory = true;
      lCompilerParameters.GenerateExecutable = false;  // not required, we don't have a Main() method
      lCompilerParameters.IncludeDebugInformation = true;
      var lCompilerResults = lCSharpCodeProvider.CompileAssemblyFromSource(lCompilerParameters, lCSharpSourceCode);

      if (lCompilerResults.Errors.HasErrors) {
        var lError = new StringBuilder();

        foreach (CompilerError lCompilerError in lCompilerResults.Errors) {
          lError.AppendLine(lCompilerError.ErrorNumber + " => " + lCompilerError.ErrorText + Environment.NewLine);
        }

        JsonOut.TextWrapping = TextWrapping.Wrap;
        JsonOut.Text = lError.ToString();
        return;
      }
      JsonOut.TextWrapping = TextWrapping.NoWrap;

      // --------------------------------------------------
      // execute the compiled code
      // --------------------------------------------------

      Assembly lAssembly = lCompilerResults.CompiledAssembly;
      Type lProgram = lAssembly.GetType("baseClass");
      MethodInfo lMethod = lProgram.GetMethod("executeLinq");

      object lQueryResult = lMethod.Invoke(null, new object[] { lJsonIn });   // returns a JSON string object

      // --------------------------------------------------
      // rudimentary JSON output formatting
      // --------------------------------------------------

      string lJsonOut = lQueryResult.ToString();
      lJsonOut = lJsonOut.Replace(",", ",\n");
      lJsonOut = lJsonOut.Replace(",{", ",{\n");
      lJsonOut = lJsonOut.Replace("]", "]\n");
      JsonOut.Text = lJsonOut;
    } //

    private void ComboBox_SelectionChanged(object xSender, SelectionChangedEventArgs e) {
      var lComboBox = xSender as ComboBox;

      switch (lComboBox.SelectedIndex) {
        case 0:
          JsonIn.Text = "{\n \"number\": 108.541,\n \"datetime\": \"1975-03-13T10:30:00\" ,\n \"serialnumber\": \"SN1234\",\n \"more\": {\n  \"field1\": 123,\n  \"field2\": \"hello\"\n },\n \"array\": [\n  {\"x\": 2.0},\n  {\"x\": 3.0},\n  {\"x\": 4.0}\n ]\n}";
          LinqIn.Text = "from a in root.array\nwhere a.x > 2.0M\nselect a";
          break;
        case 1:
          JsonIn.Text = "[ 1, 9, 5, 7, 1, 4 ]";
          LinqIn.Text = "from a in root.home\nwhere ((a == 4) || (a == 1))\nselect a";
          break;
        case 2:
          JsonIn.Text = "{myLuckyNumbers: [ 1, 9, 5, 26, 7, 1, 4 ]}";
          LinqIn.Text = "from x in root.myLuckyNumbers\nwhere x % 2 == 0\nselect new { simple=x, square=x*x, text=\"Field\"+x.ToString(), classInClass=new {veryOdd=x/7.0, lol=\":D\"}}";
          break;
        case 3:
          string s;
          s = "{\"mainMenu\": {\n";
          s += "  \"info\": \"Great tool.\",\n";
          s += "  \"value\": 100.00,\n";
          s += "  \"menu\": {\n";
          s += "  \"subMenu\": [\n";
          s += "    {\"text\": \"New\", \"onclick\": \"NewObject()\"},\n";
          s += "    {\"text\": \"Open\", \"onclick\": \"Load()\"},\n";
          s += "    {\"text\": \"Close\", \"onclick\": \"ByeBye()\"},\n";
          s += "    {\"text\": \"NNN\", \"onclick\": \"Useless()\"}\n";
          s += "  ]}\n";
          s += "}}\n";
          JsonIn.Text = s;
          LinqIn.Text = "root.mainMenu.menu.subMenu.Where(t => t.text.StartsWith(\"N\")).Last();";

          break;
        default:
          break;
      }
    } //

  } // class
} // namespace
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Web.Script.Serialization;  // requires reference to System.Web.Extensions, used by the JavaScriptSerializer

namespace LinqJson {
  public class Converter {

    private Dictionary<string, object> _Classes = new Dictionary<string, object>();

    public string GetUsings() {
      string s;
      s = "using System;\n";
      s += "using System.Collections.Generic;\n";
      s += "using System.Linq;\nusing System.Text;\n";
      s += "using System.Threading.Tasks;\n";
      s += "using System.Collections;\n";
      s += "using System.Web.Script.Serialization;\n\n";
      return s;
    } //

    public Dictionary<string, object> JsonToDictionary(string xJson) {
      xJson = xJson.Trim();
      if (xJson[0] == '[') xJson = "{home: " + xJson + "}";  // nameless arrays cannot be converted to dictionaries

      var lJavaScriptSerializer = new JavaScriptSerializer();
      try { return lJavaScriptSerializer.Deserialize<Dictionary<string, object>>(xJson); }
      catch (Exception) { return null; }
    } //

    public void BuildClasses(StringBuilder xStringBuilder) {
      foreach (var lClass in _Classes) {
        Dictionary<string, object> lMembers = lClass.Value as Dictionary<string, object>;
        if (lMembers == null) continue;
        if (lMembers.Count <= 0) continue;

        xStringBuilder.Append("public class Class_");
        xStringBuilder.Append(lClass.Key);
        xStringBuilder.AppendLine(" {");

        foreach (var lMember in lMembers) {
          object lValue = lMember.Value;
          string lKey = lMember.Key;
          Type lType = (lValue == null) ? typeof(object) : lMember.Value.GetType();

          xStringBuilder.Append(new String(' ', 2));
          xStringBuilder.Append("public ");

          if (lType.IsValueType || (lValue is string)) {
            xStringBuilder.Append(lType.Name);
            xStringBuilder.Append(" ");
            xStringBuilder.Append(lKey);
            xStringBuilder.AppendLine(";");
          }
          else if (lValue is Dictionary<string, object>) {
            xStringBuilder.Append("Class_");
            xStringBuilder.Append(lKey);
            xStringBuilder.Append(" ");
            xStringBuilder.Append(lKey);
            xStringBuilder.AppendLine(";");
          }
          else if (lValue is ArrayList) {
            ArrayList lArrayList = lValue as ArrayList;
            var lMemberType = ArrayListType(lArrayList);   // differentiate between the contents of the list
            if (lMemberType.IsValueType || (lMemberType.Name == "String")) {
              //xStringBuilder.Append(lMemberType.Name.Replace("`2")); // Dictionaries use name "Dictionary`2"
              xStringBuilder.Append(" List<");               xStringBuilder.Append(lMemberType.Name);               xStringBuilder.Append("> ");
              xStringBuilder.Append(lKey);
              xStringBuilder.AppendLine(";");
            }
            else {
              // a class
              xStringBuilder.Append("List<Class_" + lKey + "> ");
              xStringBuilder.Append(lKey);
              xStringBuilder.AppendLine(";");
            }
          }
        }
        xStringBuilder.AppendLine(" }");
        xStringBuilder.AppendLine();
      }
    } //

    public void DictionariesToClasses(string xRootName, int xIndent, object xObject) {
      if (xObject == null) return;
      Type lType = xObject.GetType();

      if (lType.IsValueType || (xObject is string)) return;
      var lDictionary = xObject as Dictionary<string, object>;
      if (lDictionary != null) {
        object lObj;
        if (!_Classes.TryGetValue(xRootName, out lObj)) {
          _Classes.Add(xRootName, lDictionary);
        }
        else {
          foreach (var lKeyValuePair in lDictionary) {
            // This is a weakness of the program.
            // Two JSON objects must not use the same name.
            // We would have to compare the JSON objects and determine wether to create multiple or just one C# class.
            _Classes[lKeyValuePair.Key] = lKeyValuePair.Value;  // object type will be overridden !!!!!!
          }
          return;
        }

        foreach (var lKeyValuePair in lDictionary) {
          DictionariesToClasses(lKeyValuePair.Key, xIndent, lKeyValuePair.Value);
        }
        return;
      }

      var lArrayList = xObject as ArrayList;
      if (lArrayList != null) {
        object lObj;
        if (!_Classes.TryGetValue(xRootName, out lObj)) {
          lDictionary = new Dictionary<string, object>();
          _Classes.Add(xRootName, lDictionary);
        }
        else lDictionary = lObj as Dictionary<string, object>;

        var lElementType = ArrayListType(lArrayList);
        if (lElementType == typeof(Dictionary<string, object>)) {
          var lList = lArrayList.Cast<Dictionary<string, object>>().ToList(); // upgrade our object to have stronger types
          foreach (var lDict in lList) {
            foreach (var lKeyValuePair in lDict) {
              lDictionary[lKeyValuePair.Key] = lKeyValuePair.Value;  // object type will be overridden !!!!!!
            }
          }
          foreach (var lKeyValuePair in lDictionary) {
            DictionariesToClasses(lKeyValuePair.Key, xIndent, lKeyValuePair.Value);
          }
        }
        return;
      }
    } //

    private void Append(StringBuilder xStringbuilder, int xIndent, string xString) {
      xStringbuilder.Append(new String(' ', xIndent));
      xStringbuilder.Append(xString);
    } //

    private void Newline(StringBuilder xStringbuilder) {
      xStringbuilder.AppendLine();
    } //

    private Type ArrayListType(ArrayList xArrayList) {
      var lTypes = new Dictionary<string, Type>();
      Type lType;
      string lTypeName;

      foreach (object o in xArrayList) {
        if (o == null) lType = typeof(object);
        else lType = o.GetType();
        lTypeName = lType.Name;
        if (!lTypes.ContainsKey(lTypeName)) lTypes.Add(lTypeName, lType);
      }

      if (lTypes.Count == 1) return lTypes.Values.First();  // distinct
      return typeof(object);
    } //

  } // class
} // namespace
Advertisements

migration C#, Java, C++ (day 8), TCP, Qt

The last two days were characterized by incompatibilities. There is no TCP support in the C++ standard libraries. I knew why I tried to avoid Boost and Qt. Visual Studio 2013 has its problems with Boost. You get really weird error messages that cannot be fixed at all. Why should Visual Studio actively support Boost? They have their own libraries, which are pretty cool.
I was fighting with several compilers and tools that generated roughly 50 GB of library data in total. Finally I gave up as it is really as mad as it is described on some web pages. I ended up installing the Qt Creator and had to rewrite the entire C++ code. I have to say that Qt is structured in a much better way than Boost. Overall the C++ coding took roughly 20 times longer than the C# coding, which was completed within 20 minutes. Most of that time was due to the incompatibility fight.
C++ executes faster, but let’s take a company’s point of view. If someone invests 250,000 USD in extra production time, wouldn’t it be smarter to buy 10x faster computers instead?
In the stock market, there are many traders telling you that it is all about speed. Not really! Data is coming in a sequence. I do know many situations where you have to slow down the system to solve specific sequence problems. Most market participant aren’t even aware of this.

Today I compare two TCP chat programs. C# is straight forward. The C++ side looks a bit weird, because the Qt Q_OBJECT macro requires header files. I did not want to split up the code into many pieces, so I kept the code short and I also kept it in the header sections.

ADDENDUM 21 May 2014

The Java code has been added today. The code structure is better than the C# example. The short Java test program deals with 1 server and 2 clients.

Below the Java code you can find a second C# example that uses the improved object approach similar to the Java code.

TCP

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Threading;

namespace DemoApp.ToCpp {
   public class Day8 {

      public class NetworkListener {

         private volatile bool _ExitLoop = true;
         private TcpListener _Listener;

         public int Port { get; private set; }
         public string IpAddress { get; private set; }
         public string ThreadName { get; private set; }
         private List<Client> _AllClients = new List<Client>();

         public NetworkListener(string xIpAddress, int xPort, string xThreadName) {
            Port = xPort;
            IpAddress = xIpAddress;
            ThreadName = xThreadName;
         } //

         private class Client {
            private TcpClient _TcpClient;
            private NetworkStream _NetworkStream;
            public delegate void dOnShutDown(Client xClient);
            public event dOnShutDown OnShutDown;

            private volatile bool _ExitLoop = false;
            public readonly string IpAddress;

            public Client(TcpClient xTcpClient) {
               _TcpClient = xTcpClient;
               IpAddress = xTcpClient.Client.LocalEndPoint.ToString();
               Console.WriteLine("SERVER MESSAGE -> " + IpAddress + " new client connected.");
            } //

            public void Disconnect() {
               _ExitLoop = true;
               dOnShutDown lEvent = OnShutDown;
               if (lEvent != null) lEvent(this);
               if (_NetworkStream != null) _NetworkStream.WriteTimeout = 5; // immediate timeout
            } //

            public void ListenAsync() {
               Thread lThread = new Thread(new ThreadStart(Listen));
               lThread.IsBackground = true;
               lThread.Name = "Client_" + IpAddress;
               lThread.Start();
            } //

            private void Listen() {
               using (_NetworkStream = _TcpClient.GetStream()) {
                  using (StreamReader lStreamReader = new StreamReader(_NetworkStream)) {
                     while (!_ExitLoop) {
                        try {
                           string s = lStreamReader.ReadLine();
                           if (s == null) break;
                           Console.WriteLine("SERVER MESSAGE -> " + IpAddress + " Client said: " + s);
                        }
                        catch (IOException) {
                           if (_ExitLoop) Console.WriteLine("SERVER MESSAGE -> user requested TcpClient shutdown");
                           else Console.WriteLine("SERVER MESSAGE -> disconnected");
                        }
                        catch (Exception ex) { Console.WriteLine(ex.Message); }
                     }
                  }
                  Console.WriteLine("SERVER MESSAGE -> " + IpAddress + " Listener is shutting down");
               }
               _NetworkStream = null;
            } //
         } // class

         public bool WaitForClientsAsync() {
            if (!_ExitLoop) {
               Console.WriteLine("SERVER MESSAGE -> Listener running already");
               return false;
            }
            _ExitLoop = false;

            try {
               _Listener = new TcpListener(IPAddress.Parse(IpAddress), Port);
               _Listener.Start();

               Thread lThread = new Thread(new ThreadStart(WaitForAClient));
               lThread.IsBackground = true;
               lThread.Name = ThreadName;
               lThread.Start();

               return true;
            }
            catch (Exception ex) { Console.WriteLine(ex.Message); }
            return false;
         } //

         public void Disconnect() {
            _ExitLoop = true;
            List<Client> lClone; // we need a clone, because the _AllClients collection will be manipulated during the foreach loop
            lock (_AllClients) { lClone = new List<Client>(_AllClients); }
            foreach (Client lClient in lClone) lClient.Disconnect();
            _Listener.Stop();
         } //

         private void WaitForAClient() {
            try {
               while (!_ExitLoop) {
                  int lClientCount;
                  lock (_AllClients) lClientCount = _AllClients.Count;
                  Console.WriteLine("SERVER MESSAGE -> Waiting for client number " + lClientCount);
                  TcpClient lTcpClient = _Listener.AcceptTcpClient();
                  Client lClient = new Client(lTcpClient);
                  lClient.OnShutDown += OnClientShutDown;
                  lock (_AllClients) _AllClients.Add(lClient);
                  lClient.ListenAsync(); // starts a background thread
               }
            }
            catch (Exception ex) { Console.WriteLine(ex.Message); }
         }

         void OnClientShutDown(NetworkListener.Client xClient) {
            lock (_AllClients) { _AllClients.Remove(xClient); }
         } //         

      } // class

      public class NetworkClient {
         public int Port { get; private set; }
         public string IpAddress { get; private set; }
         public string ThreadName { get; private set; }

         private NetworkStream _NetworkStream = null;
         private bool _ExitLoop = true;
         private BlockingCollection<string> _Queue = new BlockingCollection<string>();

         public NetworkClient(string xIpAddress, int xPort, string xThreadName) {
            Port = xPort;
            IpAddress = xIpAddress;
            ThreadName = xThreadName;
         } //

         public void ConnectAsync() {
            if (!_ExitLoop) return; // running already
            _ExitLoop = false;

            Thread lThread = new Thread(new ThreadStart(Loop));
            lThread.IsBackground = true;
            lThread.Name = ThreadName;
            lThread.Start();
         } //

         public void Disconnect() {
            _ExitLoop = true;
            _Queue.Add(null);
            if (_NetworkStream != null) _NetworkStream.ReadTimeout = 100;
         } //

         public void Send(string xText) {
            if (string.IsNullOrEmpty(xText)) return;
            _Queue.Add(xText); // thread safety by context switching
         } //

         private void Loop() {
            try {
               using (TcpClient lClient = new TcpClient()) {
                  lClient.Connect(IpAddress, Port);
                  using (_NetworkStream = lClient.GetStream()) {
                     using (StreamWriter lStreamWriter = new StreamWriter(_NetworkStream)) {
                        while (!_ExitLoop) {
                           try {
                              string s = _Queue.Take();
                              if (string.IsNullOrEmpty(s)) break;
                              lStreamWriter.WriteLine(s);
                           }
                           catch (System.IO.IOException) {
                              if (_ExitLoop) Console.WriteLine("CLIENT -> User requested shutdown.");
                              else Console.WriteLine("CLIENT -> disconnected");
                           }
                           catch (Exception ex) { Console.WriteLine(ex.Message); }
                        }
                     }
                     _ExitLoop = true;
                     Console.WriteLine(Environment.NewLine + "CLIENT -> shutting down");
                  }
               }
            }
            catch (Exception ex) { Console.WriteLine(ex.Message); }
         } //

      } // class


      public static void Test() {
         NetworkListener lServer = new NetworkListener("127.0.0.1", 65432, "Server");
         NetworkClient lClient = new NetworkClient("127.0.0.1", 65432, "Client");

         lServer.WaitForClientsAsync();

         lClient.ConnectAsync();
         lClient.Send("Hello World!"); // send a text message across the network

         System.Threading.Thread.Sleep(1000);

         lClient.Disconnect();
         lServer.Disconnect();

         Console.ReadLine();
      } //

   } // class
} // namespace

example output:
SERVER MESSAGE -> Waiting for client number 0
SERVER MESSAGE -> 127.0.0.1:65432 new client connected.
SERVER MESSAGE -> Waiting for client number 1
SERVER MESSAGE -> 127.0.0.1:65432 Client said: Hello World!

CLIENT -> shutting down
SERVER MESSAGE -> 127.0.0.1:65432 Listener is shutting down
A blocking operation was interrupted by a call to WSACancelBlockingCall

#pragma once

#include <iostream>
#include <QtNetwork>
#include <QObject>
#include <QString>
#include <QTcpSocket>
#include <string>

using namespace std;

class Client : QObject {
    Q_OBJECT

private:
  int _Port = 0;
  QString _IpAddress;
  QTcpSocket _TcpSocket;

private slots:
    void Connected() { cout << "CLIENT MESSAGE -> connection established " << _IpAddress.toStdString() << ":" << _Port << endl; }

public:
   inline int getPort() { return _Port; }
   inline QString getIpAddress() { return _IpAddress; }

   Client(QString xIpAddress, int xPort) : QObject() {
    _IpAddress = xIpAddress;
    _Port = xPort;
   }
   ~Client() { cout << "deconstructing Client class " << _IpAddress.toStdString() << endl; }
   Client(QObject* lParent): QObject(lParent) { cout << "Client :)" << endl; }

  void Connect() {
      connect(&_TcpSocket, SIGNAL(connected()), this, SLOT(Connected()));
      QHostAddress lHostAddress(_IpAddress);
      _TcpSocket.connectToHost(lHostAddress, _Port);
      while (!_TcpSocket.waitForConnected( 2000 )) {
        cout << "CLIENT MESSAGE -> waiting for a feedback" << endl;
      }
  }

  void Disconnect() {
    _TcpSocket.close();
  } //

  void Send(const string &xText) {
    if (xText == "") return;
    const char *lText = xText.c_str();
    _TcpSocket.write(lText, xText.length() + 1);
  } //

};

//----------------------------------------------------------------------------------------------------------------------

#pragma once

#include <QTcpSocket>
#include <QHostAddress>
#include <thread>
#include <set>
#include <iostream>
#include <QtNetwork>

using namespace std;

class Connection : QObject {
    Q_OBJECT

  private:
    QTcpSocket *_TcpSocket;
    string _IpAddress;

    void Read() {
      try {
        char lBuffer[1024] = {0};
        _TcpSocket->read(lBuffer, _TcpSocket->bytesAvailable());
        cout << "SERVER MESSAGE -> " << _IpAddress << " Client said: " << lBuffer << endl;
      }
        catch (exception &ex) { cout << ex.what() << endl; }
    } //


  public:
    const string &IpAddress() const { return _IpAddress; }

    void Disconnect() {
      _TcpSocket->close();
      delete _TcpSocket;
    } //

    Connection(QTcpSocket *xTcpSocket) : _TcpSocket(xTcpSocket) {
      QHostAddress lAddress = xTcpSocket->localAddress();
      QString s = lAddress.toString();
      _IpAddress = s.toLocal8Bit().constData();
      connect(xTcpSocket, SIGNAL(readyRead()), this, SLOT(Read()));  // setup event
      cout << "SERVER MESSAGE -> " + _IpAddress << " new client connected." << endl;
    } //
    Connection(QObject* lParent): QObject(lParent) { cout << "Connection :)"; }
    ~Connection() { Disconnect(); cout << "deconstructing Connection class " << _IpAddress << endl; }

 }; // class

//----------------------------------------------------------------------------------------------------------------------

#pragma once

#include <iostream>
#include <QtNetwork>
#include <QObject>
#include <QString>
#include <QTcpSocket>
#include <QTcpServer>
#include <string>
#include "Connection.cpp"
#include <mutex>


using namespace std;

class Server : QObject {
    Q_OBJECT

private:
    list<Connection*> _AllClients;
    mutex mutex_AllClients;
    int _Port;
    string _IpAddress;
    QTcpServer _TcpServer;

    void OnClientShutDown(Connection *xClient) {
        cout << "SERVER MESSAGE -> Connection shutting down" << endl;
        mutex_AllClients.lock();
        _AllClients.remove(xClient);
        mutex_AllClients.unlock();
        xClient->Disconnect();
    } //

private slots:
    void NewConnection() {
        cout << "SERVER MESSAGE -> new connection" << endl;
        QTcpSocket *lSocket = _TcpServer.nextPendingConnection();
        Connection *lClient = new Connection(lSocket);
        mutex_AllClients.lock();
        _AllClients.push_back(lClient);
        mutex_AllClients.unlock();
    } //

public:    
    Server(QObject* lParent): QObject(lParent) { cout << "Server :)"; }
    ~Server() { cout << "deconstructing Server class " << endl; }
    inline int getPort() { return _Port; }
    inline string getIpAddress() { return _IpAddress; }    

    Server(string xIpAddress, int xPort) : QObject() {
      _Port = xPort;
      _IpAddress = xIpAddress;
      connect(&_TcpServer, SIGNAL(newConnection()), this, SLOT(NewConnection())); // setup event
      QString lIpAddress(xIpAddress.c_str());
      //QHostAddress lHostAddress(lIpAddress);
      if (_TcpServer.listen(QHostAddress::Any, xPort)) {
      //if(_TcpServer.listen(lHostAddress, xPort)) {
        cout << "SERVER MESSAGE -> listener up and running" << endl;
      }
    } //

    void Disconnect() {
      mutex_AllClients.lock();
      list<Connection*> lClone;
      list<Connection*>::const_iterator lIterator;
      for (lIterator = _AllClients.begin(); lIterator != _AllClients.end(); ++lIterator) {
        lClone.push_back(*lIterator);  // we need a clone, because the _AllClients collection will be manipulated during the foreach loop
      }
      mutex_AllClients.unlock();

      for (Connection *lClient : lClone) lClient->Disconnect();
      _TcpServer.close();
    } //

}; // class

//----------------------------------------------------------------------------------------------------------------------

#include <QApplication>
#include <iostream>
#include <thread>
#include "Server.h"
#include "Client.h"

int main(int argc, char** argv)
{
  QApplication app(argc, argv);
  try {

    Server lServer("127.0.0.1", 65432);
    Client lClient("127.0.0.1", 65432);

    lClient.Connect();    

    this_thread::sleep_for(chrono::milliseconds(1000)) ;
    lClient.Send("Hello World!"); // send a text message across the network
    this_thread::sleep_for(chrono::milliseconds(1000)) ;

    lClient.Disconnect();
    lServer.Disconnect();

    cout << "press enter to exit" << endl;
    cin.get();
  }
  catch (exception &ex) { cerr << ex.what() << endl; }
  return app.exec();
} //
package DemoApp;

import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;
import java.util.ArrayList;

public final class Server extends Thread {

  private final ArrayList<Client> _Clients = new ArrayList<>();
  public final int port;
  ServerSocket _ServerSocket;

  public Server(String xName, int xPort) throws IOException {
    super(xName);
    port = xPort;
    setDaemon(true);
    _ServerSocket = new ServerSocket(xPort);
  } // constructor

  public void close() throws IOException {
    this.interrupt(); // exit loop
    synchronized (_Clients) {
      for (Client lClient : _Clients) lClient.close();
      _Clients.clear();
    }

    _ServerSocket.close();
    System.out.println(_ServerSocket.getLocalSocketAddress().toString() + " server closed down");
  } //

  public void broadcast(String xMessage) {
    synchronized (_Clients) {
      for (Client lClient : _Clients)
        lClient.getWriter().send(xMessage);
    }
  } //

  @Override
  public void run() {
    System.out.println(_ServerSocket.getLocalSocketAddress().toString() + " server is waiting for clients");

    try {
      while (true) {
        Socket lSocket = _ServerSocket.accept();  // wait for a client to connect
        Client lClient = new Client(lSocket);
        synchronized (_Clients) {
          _Clients.add(lClient);
        }
        lClient.connect();
      }
    } catch (IOException ex) {
      //System.out.println(ex.getMessage());
    }
  } //

} // class

// -----------------------------------------------------------------------------------------------------------------------

package DemoApp;

import java.io.IOException;
import java.net.Socket;

public class Client {

  private Socket _Socket = null;
  public final String IpAddressHost;
  public final String IpAddressLocal;
  public final String IpConnectionString;
  private Reader _Reader = null;
  private Writer _Writer = null;

  public Socket getSocket() {
    return _Socket;
  } //

  public Reader getReader() {
    return _Reader;
  } //

  public Writer getWriter() {
    return _Writer;
  } //

  public void close() throws IOException {
    synchronized (this) {
      if (_Reader != null) _Reader.close(true);
      _Reader = null;
      if (_Writer != null) _Writer.close(true);
      _Writer = null;
    }

    System.out.println(IpConnectionString + " client closed down");
  } //

  public Client(Socket xSocket) {
    _Socket = xSocket;
    IpAddressHost = xSocket.getInetAddress().getHostAddress() + ":" + _Socket.getPort();
    IpAddressLocal = xSocket.getLocalAddress().getHostAddress() + ":" + _Socket.getLocalPort();
    IpConnectionString = "(Local " + IpAddressLocal + ",  Host " + IpAddressHost + ")";
  } //

  public void connect() throws IOException {
    _Reader = new Reader(this);
    _Writer = new Writer(this);
    _Reader.start();  // start listening
    _Writer.start();  // start write loop        
  } //  

} // class

// -----------------------------------------------------------------------------------------------------------------------

package DemoApp;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.Socket;

public class Reader extends Thread {

  private final Client _Client;
  private BufferedReader _StreamReader;

  public Reader(Client xClient) throws IOException {
    _Client = xClient;
    Socket lSocket = xClient.getSocket();
    _StreamReader = new BufferedReader(new InputStreamReader(lSocket.getInputStream()));
  } // constructor

  public void close(boolean xExitLoop) throws IOException {
    synchronized (this) {
      if (xExitLoop) interrupt();
      if (_StreamReader == null) return;
      _StreamReader.close();
      _StreamReader = null;
    }

    System.out.println(_Client.IpConnectionString + " closed reader connection");
  } //

  @Override
  public void run() {
    System.out.println(_Client.IpConnectionString + " start new reader connection");

    while (true)
      try {
        String lInput = _StreamReader.readLine();
        if (lInput == null) break;
        if (lInput.isEmpty()) continue;

        System.out.println(_Client.IpConnectionString + " message received: " + lInput);
        if ("EXIT".equals(lInput)) break;

      } catch (IOException | RuntimeException ex) {
        System.out.println(ex);
      }
    try {
      close(false);
    } catch (IOException ex) {
    }
  } //

} // class

// -----------------------------------------------------------------------------------------------------------------------

package DemoApp;

import java.io.IOException;
import java.io.PrintWriter;
import java.net.Socket;
import java.util.LinkedList;

public class Writer extends Thread {

  private final Client _Client;
  private PrintWriter _PrintWriter;

  private final LinkedList<String> _WriteQueue = new LinkedList<>();

  public Writer(Client xClient) throws IOException {
    setDaemon(true);
    _Client = xClient;
    Socket lSocket = xClient.getSocket();
    _PrintWriter = new PrintWriter(lSocket.getOutputStream(), true);
  } // constructor

  public void close(boolean xExitLoop) throws IOException {
    synchronized (this) {
      if (xExitLoop) interrupt();
      if (_PrintWriter == null) return;
      _PrintWriter.close();
      _PrintWriter = null;
    }
    System.out.println(_Client.IpConnectionString + " closed writer connection");
  } //

  public void send(String xText) {
    if (xText == null) return;

    synchronized (_WriteQueue) {
      _WriteQueue.addLast(xText);
      _WriteQueue.notify();
    }
  } //

  @Override
  public void run() {
    System.out.println(_Client.IpConnectionString + " start new writer connection");

    try {
      while (true) {
        String lText = null;
        synchronized (_WriteQueue) {
          if (_WriteQueue.size() > 0) lText = _WriteQueue.removeFirst();
          else _WriteQueue.wait();
        }
        if (lText == null) continue;
        if (_PrintWriter == null) continue;
        _PrintWriter.println(lText);
      }
    } catch (InterruptedException ex) {
      if (ex.getMessage() != null) System.out.println(ex.getMessage());
    }
    try {
      close(false);
    } catch (IOException ex) {
    }
  } //

} // class

// -----------------------------------------------------------------------------------------------------------------------

package DemoApp;

import java.io.IOException;
import java.net.Socket;

public class Test {

  private static void Sleep() {
    try {
      Thread.sleep(2000);
    } catch (InterruptedException ex) {
      System.out.println(ex.getMessage());
    }
  } //

  public static void main(String[] args) throws IOException {
    Server lServer = new Server("MyServer", 65432);
    lServer.start();
    Sleep();

    Socket lClientSocket1 = new Socket("localhost", 65432);
    Client lClient1 = new Client(lClientSocket1);
    lClient1.connect();
    Sleep();

    Socket lClientSocket2 = new Socket("localhost", 65432);
    Client lClient2 = new Client(lClientSocket2);
    lClient2.connect();
    Sleep();

    lClient1.getWriter().send("client->server: hello server!");
    Sleep();

    lServer.broadcast("server->client: hello client!");
    Sleep();

    lClient1.getWriter().send("EXIT");
    lClient1.close();

    lClient2.getWriter().send("EXIT");
    lClient2.close();

    Sleep();

    lServer.close();
    //System.in.read();
    Sleep();
  }
} // class

example output:
0.0.0.0/0.0.0.0:65432 server is waiting for clients
(Local 127.0.0.1:65432, Host 127.0.0.1:54059) start new reader connection
(Local 127.0.0.1:54059, Host 127.0.0.1:65432) start new reader connection
(Local 127.0.0.1:65432, Host 127.0.0.1:54059) start new writer connection
(Local 127.0.0.1:54059, Host 127.0.0.1:65432) start new writer connection
(Local 127.0.0.1:54060, Host 127.0.0.1:65432) start new reader connection
(Local 127.0.0.1:54060, Host 127.0.0.1:65432) start new writer connection
(Local 127.0.0.1:65432, Host 127.0.0.1:54060) start new reader connection
(Local 127.0.0.1:65432, Host 127.0.0.1:54060) start new writer connection
(Local 127.0.0.1:65432, Host 127.0.0.1:54059) message received: client->server: hello server!
(Local 127.0.0.1:54059, Host 127.0.0.1:65432) message received: server->client: hello client!
(Local 127.0.0.1:54060, Host 127.0.0.1:65432) message received: server->client: hello client!
(Local 127.0.0.1:65432, Host 127.0.0.1:54059) message received: EXIT
(Local 127.0.0.1:65432, Host 127.0.0.1:54059) closed reader connection
(Local 127.0.0.1:54059, Host 127.0.0.1:65432) closed reader connection
(Local 127.0.0.1:54059, Host 127.0.0.1:65432) closed writer connection
(Local 127.0.0.1:54059, Host 127.0.0.1:65432) client closed down
(Local 127.0.0.1:65432, Host 127.0.0.1:54060) message received: EXIT
(Local 127.0.0.1:65432, Host 127.0.0.1:54060) closed reader connection
(Local 127.0.0.1:54060, Host 127.0.0.1:65432) closed reader connection
(Local 127.0.0.1:54060, Host 127.0.0.1:65432) closed writer connection
(Local 127.0.0.1:54060, Host 127.0.0.1:65432) client closed down
(Local 127.0.0.1:65432, Host 127.0.0.1:54059) closed writer connection
(Local 127.0.0.1:65432, Host 127.0.0.1:54059) client closed down
(Local 127.0.0.1:65432, Host 127.0.0.1:54060) closed writer connection
(Local 127.0.0.1:65432, Host 127.0.0.1:54060) client closed down
0.0.0.0/0.0.0.0:65432 server closed down

using System;
using System.Collections.Generic;
using System.Net;
using System.Net.Sockets;
using System.Threading;

namespace Tcp {
   public class Server {

      private readonly List<Client> _Clients = new List<Client>();
      public readonly int Port;
      private TcpListener _TcpListener = null;
      private Thread _Thread;
      private readonly string Name;

      public Server(String xName, int xPort) {
         Name = xName;
         Port = xPort;
      } // constructor

      public bool start() {
         try {
            _TcpListener = new TcpListener(IPAddress.Parse("127.0.0.1"), Port);
            _TcpListener.Start();

            _Thread = new Thread(loop);
            _Thread.IsBackground = true;
            _Thread.Name = Name;
            _Thread.Start();
         }
         catch (Exception ex) { Console.WriteLine(ex.StackTrace + Environment.NewLine + ex.Message); return false; }
         return true;
      } //

      public void Close() {
         if (_Thread != null) { _Thread.Interrupt(); _Thread = null; }
         lock (_Clients) {
            foreach (Client lClient in _Clients) lClient.close();
            _Clients.Clear();
         }

         _TcpListener.Stop();
         Console.WriteLine(_TcpListener.LocalEndpoint.ToString() + " server closed down");
         _TcpListener = null;
      } //

      public void broadcast(String xMessage) {
         lock (_Clients) {
            foreach (Client lClient in _Clients) lClient.Writer.Send(xMessage);
         }
      } //

      private void loop() {
         Console.WriteLine(_TcpListener.LocalEndpoint.ToString() + " server is waiting for clients");

         try {
            while (true) {
               int lClientCount;
               lock (_Clients) lClientCount = _Clients.Count;
               TcpClient lTcpClient = _TcpListener.AcceptTcpClient();   // wait for a client to connect
               Client lClient = new Client(lTcpClient);
               lock (_Clients) _Clients.Add(lClient);
               lClient.Connect();
            }
         }
         catch (ThreadInterruptedException) { } // Thread interruption
         catch (SocketException) { } // Thread interruption => SocketException
         catch (Exception ex) { Console.WriteLine(ex.StackTrace + Environment.NewLine + ex.Message); }
      } //

   } // class
} // namespace

// -----------------------------------------------------------------------------------------------------------------------

using System;
using System.Net.Sockets;

namespace Tcp {
   public class Client {
      public TcpClient TcpClient { get; private set; }
      public readonly String IpAddressHost;
      public readonly String IpAddressLocal;
      public readonly String IpConnectionString;
      public Reader Reader { get; private set; }
      public Writer Writer { get; private set; }

      public void close() {
         lock (this) {
            if (Reader != null) Reader.Close(true);
            Reader = null;
            if (Writer != null) Writer.Close(true);
            Writer = null;
            if (TcpClient != null) {
               TcpClient.Close();
               Console.WriteLine(IpConnectionString + " TcpClient read/write closed");
            }
            TcpClient = null;
         }

         Console.WriteLine(IpConnectionString + " client closed down");
      } //

      public Client(string xHostname, int xPort) : this(new TcpClient(xHostname, xPort)) { }   // create and connect

      public Client(TcpClient xTcpClient) {
         TcpClient = xTcpClient;
         IpAddressHost = TcpClient.Client.RemoteEndPoint.ToString();
         IpAddressLocal = TcpClient.Client.LocalEndPoint.ToString();
         IpConnectionString = "(Local " + IpAddressLocal + ",  Host " + IpAddressHost + ")";
      } //

      public void Connect() {
         Reader = new Reader(this);
         Writer = new Writer(this);
         Reader.Start();  // start listening
         Writer.Start();  // start write loop        
      } //  

   } // class
} // namespace

// -----------------------------------------------------------------------------------------------------------------------

using System;
using System.IO;
using System.Net.Sockets;
using System.Threading;

namespace Tcp {
   public class Reader {

      private readonly Client _Client;
      private StreamReader _StreamReader = null;
      private Thread _Thread = null;

      public Reader(Client xClient) {
         _Client = xClient;
         NetworkStream lNetworkStream = xClient.TcpClient.GetStream();
         _StreamReader = new StreamReader(lNetworkStream);
      } // constructor

      public void Start() {
         _Thread = new Thread(loop);
         _Thread.IsBackground = true;
         _Thread.Start();
      } //

      public void Close(bool xExitLoop) {
         lock (this) {
            if (xExitLoop && (_Thread != null)) { _Thread.Interrupt(); _Thread = null; }

            // _Client.TcpClient.Close() in Client object
            //if (_StreamReader == null) return;
            //_StreamReader.Close();
            //_StreamReader = null;
         }
         
         //Console.WriteLine(_Client.IpConnectionString + " closed reader connection");
      } //

      private void loop() {
         Console.WriteLine(_Client.IpConnectionString + " start new reader connection");

         while (true) {
            try {
               String lInput = _StreamReader.ReadLine();
               if (lInput == null) break;
               if (lInput == "") continue;

               Console.WriteLine(_Client.IpConnectionString + " message received: " + lInput);
               if ("EXIT".Equals(lInput)) break;

            }
            catch (ThreadInterruptedException) { break; } // Thread interruption
            catch (IOException) { break; } // StreamReader disposed
            catch (Exception ex) { Console.WriteLine(ex.StackTrace + Environment.NewLine + ex.Message); }
         }
         Close(false);
      } //

   } // class
} // namespace

// -----------------------------------------------------------------------------------------------------------------------

using System;
using System.Collections.Concurrent;
using System.IO;
using System.Net.Sockets;
using System.Threading;

namespace Tcp {
   public class Writer {

      private readonly Client _Client;
      private StreamWriter _StreamWriter = null;
      private Thread _Thread = null;

      private readonly BlockingCollection<String> _WriteQueue = new BlockingCollection<string>();

      public Writer(Client xClient) {
         _Client = xClient;
         NetworkStream lNetworkStream = xClient.TcpClient.GetStream();
         _StreamWriter = new StreamWriter(lNetworkStream);
      } // constructor

      public void Start() {
         _Thread = new Thread(Loop);
         _Thread.IsBackground = true;
         _Thread.Start();
      } //

      public void Close(bool xExitLoop) {
         lock (this) {
            if (xExitLoop && (_Thread != null)) { _Thread.Interrupt(); _Thread = null; }

            // _Client.TcpClient.Close() in Client object
            //if (_StreamWriter == null) return;
            //_StreamWriter.Close();
            //_StreamWriter = null;
         }
         //Console.WriteLine(_Client.IpConnectionString + " closed writer connection");
      } //

      public void Send(string xText) {
         if (string.IsNullOrEmpty(xText)) return;
         _WriteQueue.Add(xText);
      } //


      private void Loop() {
         Console.WriteLine(_Client.IpConnectionString + " start new writer connection");

         while (true) {
            try {
               string lText = null;
               lText = _WriteQueue.Take();
               if (string.IsNullOrEmpty(lText)) continue;
               if (_StreamWriter == null) continue;
               _StreamWriter.WriteLine(lText);
               _StreamWriter.Flush();
            }
            catch (ObjectDisposedException) { break; } 
            catch (ThreadInterruptedException) { break; } // Thread interruption
            catch (Exception ex) { Console.WriteLine(ex.StackTrace + Environment.NewLine + ex.Message); }
         }

         Close(false);
      } //


   } // class
} // namespace

// -----------------------------------------------------------------------------------------------------------------------

using System.Net.Sockets;
using System.Threading;

namespace Tcp {
   class Program {
      static void Main(string[] args) {

         Server lServer = new Server("MyServer", 65432);
         lServer.start();
         Thread.Sleep(2000);

         TcpClient lClientSocket1 = new TcpClient("localhost", 65432);
         Client lClient1 = new Client(lClientSocket1);
         lClient1.Connect();
         Thread.Sleep(2000);

         TcpClient lClientSocket2 = new TcpClient("localhost", 65432);
         Client lClient2 = new Client(lClientSocket2);
         lClient2.Connect();
         Thread.Sleep(2000);

         lClient1.Writer.Send("client->server: hello server!");
         Thread.Sleep(2000);

         lServer.broadcast("server->client: hello client!");
         Thread.Sleep(2000);

         lClient1.Writer.Send("EXIT");
         Thread.Sleep(2000);
         lClient1.close();

         lClient2.Writer.Send("EXIT");
         Thread.Sleep(2000);
         lClient2.close();

         Thread.Sleep(1000);

         lServer.Close();
         //Console.ReadLine();
         Thread.Sleep(5000);

      } // main
   } // class
} // namespace

example output:
127.0.0.1:65432 server is waiting for clients
(Local 127.0.0.1:65432, Host 127.0.0.1:58161) start new reader connection
(Local 127.0.0.1:58161, Host 127.0.0.1:65432) start new reader connection
(Local 127.0.0.1:58161, Host 127.0.0.1:65432) start new writer connection
(Local 127.0.0.1:65432, Host 127.0.0.1:58161) start new writer connection
(Local 127.0.0.1:65432, Host 127.0.0.1:58163) start new reader connection
(Local 127.0.0.1:58163, Host 127.0.0.1:65432) start new reader connection
(Local 127.0.0.1:65432, Host 127.0.0.1:58163) start new writer connection
(Local 127.0.0.1:58163, Host 127.0.0.1:65432) start new writer connection
(Local 127.0.0.1:65432, Host 127.0.0.1:58161) message received: client->server: hello server!
(Local 127.0.0.1:58161, Host 127.0.0.1:65432) message received: server->client: hello client!
(Local 127.0.0.1:58163, Host 127.0.0.1:65432) message received: server->client: hello client!
(Local 127.0.0.1:65432, Host 127.0.0.1:58161) message received: EXIT
(Local 127.0.0.1:58161, Host 127.0.0.1:65432) TcpClient read/write closed
(Local 127.0.0.1:58161, Host 127.0.0.1:65432) client closed down
(Local 127.0.0.1:65432, Host 127.0.0.1:58163) message received: EXIT
(Local 127.0.0.1:58163, Host 127.0.0.1:65432) TcpClient read/write closed
(Local 127.0.0.1:58163, Host 127.0.0.1:65432) client closed down
(Local 127.0.0.1:65432, Host 127.0.0.1:58161) TcpClient read/write closed
(Local 127.0.0.1:65432, Host 127.0.0.1:58161) client closed down
(Local 127.0.0.1:65432, Host 127.0.0.1:58163) TcpClient read/write closed
(Local 127.0.0.1:65432, Host 127.0.0.1:58163) client closed down
127.0.0.1:65432 server closed down

Remoting (part 2, advanced), WCF

WCF improves code concepts substantially. There are templates like “WCF Service Application” in Visual Studio making programming extremely easy. You get results with a few clicks.
I am concentrating on an independent code solution today. This basically is a similar solution to my remoting example in part 1. You can clearly see the differences rather than starting an entirely new project type.

WcfSolution

We need two App.config files.
One on the client side

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
	<system.serviceModel>

    <client>
      <endpoint name="ServerInstance" address="net.tcp://localhost/ServerInstance" binding="netTcpBinding" bindingConfiguration="tcp_Unsecured" contract="Shared.MyInterface"/>
		</client>
    <bindings>
      <netTcpBinding>
        <binding name="tcp_Unsecured">
          <security mode="None" />
        </binding>
      </netTcpBinding>
    </bindings>
    
  </system.serviceModel>
</configuration>

and one on the server-side.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>

  <system.serviceModel>

    <services>
      <service name="Server.ServerInstance">
        <endpoint address="net.tcp://localhost/ServerInstance" binding="netTcpBinding" bindingConfiguration="tcp_Unsecured" contract="Shared.MyInterface" />
      </service>
    </services>
    <bindings>
      <netTcpBinding>
        <binding name="tcp_Unsecured">
          <security mode="None" />
        </binding>
      </netTcpBinding>
    </bindings>
  </system.serviceModel>
</configuration>

I guess you have already noticed the Shared.MyInterface. This is the same approach as in the classical Remoting. We share data definitions.

using System;
using System.Collections.Generic;
using System.ServiceModel;

namespace Shared {
   [ServiceContract(SessionMode = SessionMode.Allowed)]
   public interface MyInterface {
      [OperationContract]
      List<TradeData> GetTrades();
      [OperationContract]
      string HelloWorld(string xName);
      [OperationContract]
      DateTime Ping();     
   } // interface

} // namespace

The shared area also needs our TradeData class, which is used inside the Interface. Do not forget the Serializable attribute. You do not necessarily get an error message. The program can simply refuse proper execution and only provide mind-boggling error messages.

using System;

namespace Shared {

   [Serializable]
   public class TradeData {
      public DateTime tradeTime;
      public string ticker;
      public double price;
      public int quantity;

      public override string ToString() {
         return tradeTime.ToString("dd.MMM.yy HH:mm:ss ") + ticker + " " + quantity + " @ " + price + " EUR";
      } //

      public TradeData(DateTime xTradeTime, string xTicker, double xPrice, int xQuantity) {
         tradeTime = xTradeTime;
         ticker = xTicker;
         price = xPrice;
         quantity = xQuantity;
      } // constructor

   } // class
} // namespace

The class ServerInstance is part of the server-side. It inherits MyInterface and adds the required statements of MyInterface.

using System;
using System.Collections.Generic;
using System.ServiceModel;
using Shared;

namespace Server {
   [ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
   internal class ServerInstance : MyInterface {

      public List<TradeData> GetTrades() {
         DateTime lDummyTime = new DateTime(2014, 02, 12, 16, 30, 10);
         TradeData lTrade1 = new TradeData(lDummyTime, "DTE", 11.83, 100);
         TradeData lTrade2 = new TradeData(lDummyTime.AddSeconds(2), "DAI", 66.45, 300);
         TradeData lTrade3 = new TradeData(lDummyTime.AddSeconds(5), "DBK", 35.91, 100);
         return new List<TradeData>() { lTrade1, lTrade2, lTrade3 };
      } //

      public string HelloWorld(string xName) {
         return "Hello " + xName;
      } //

      public DateTime Ping() {
         return DateTime.Now;
      } //

   } // class  
} // namespace

The server side code is relatively short. It is more or less a two liner to instantiate a ServiceHost and open it.

using System;
using System.ServiceModel;
using System.Threading;

namespace Server {
   public class Program {
      
      public static void Main(string[] arguments) {

         Thread.Sleep(10000); // we let the client wait 10 seconds to simulate a server that is down

         using (ServiceHost lServiceHost = new ServiceHost(typeof(ServerInstance))) {
            lServiceHost.Open();

            Console.WriteLine("Service is available.");
            Console.ReadKey();
         }
      } // main

   } // class
} // namespace

And the client side is not much longer. Have a look for yourself. Most of the lines are taking care of printing the data to the console window.

using System;
using System.ServiceModel;
using System.ServiceModel.Channels;
using System.Threading;
using Shared;

namespace Client {
   public class Program {

      public static void Main(string[] args) {

         ChannelFactory<MyInterface> lFactory = new ChannelFactory<MyInterface>("ServerInstance");

         MyInterface lShared = null;
         while (true) {
            try {
               lShared = lFactory.CreateChannel();                              

               var o = lShared.GetTrades();
               var s = lShared.HelloWorld("Mr. Ohta");
               var p = lShared.Ping();

               Console.WriteLine("GetTrades():");
               foreach (var lTrade in o) Console.WriteLine(lTrade.ToString());
               Console.WriteLine();
               Console.WriteLine("HelloWorld(): " + s);
               Console.WriteLine();
               Console.WriteLine("Ping(): " + p);

               break;
            }
            catch (Exception ex) {
               Console.WriteLine(ex.Message);
               Thread.Sleep(2000);
            }
            finally {
               IChannel lChannel = lShared as IChannel;
               if (lChannel.State == CommunicationState.Opened) lChannel.Close();
            }
         }

         Console.ReadKey();
      } // main

   } // class
} // namespace

After starting the client and server concurrently,

MultipleProject

you should get output results like these:

SERVER example output:
Service is available.

CLIENT example output:
Could not connect to net.tcp://localhost/ServerInstance. The connection attempt
lasted for a time span of 00:00:02.0521174. TCP error code 10061: No connection
could be made because the target machine actively refused it 127.0.0.1:808.
Could not connect to net.tcp://localhost/ServerInstance. The connection attempt
lasted for a time span of 00:00:02.0391166. TCP error code 10061: No connection
could be made because the target machine actively refused it 127.0.0.1:808.
GetTrades():
12.Feb.14 16:30:10 DTE 100 @ 11.83 EUR
12.Feb.14 16:30:12 DAI 300 @ 66.45 EUR
12.Feb.14 16:30:15 DBK 100 @ 35.91 EUR

HelloWorld(): Hello Mr. Ohta

Ping(): 17/02/2014 12:30:59

The error messages on the client side are as expected. They are caused by “Thread.Sleep(10000);” on the server side. I thought it would be interesting to show a server which is not running at the time when starting the client.

Put aside all the complex and confusing examples on the internet. This one here is spot on. It provides a nice and comfortable entry point into WCF. The code does not have to be complex.
I recommend YouTube for further WCF explanations.

Remoting (part 1, basics), old school

Before I come to the new WCF style Remoting, I would like to introduce the good old System.Runtime.Remoting namespace.
Remoting is a convenient way to call methods across the network. You don’t have to write messages to trigger methods on the server, wait for results and then analyse feedback messages on the client side. What could be easier than calling server methods directly?

The downside is the slowness and the missing encryption. Nevertheless, if you don’t have many server requests then Remoting is probably the right solution for you. The ease is hard to beat.

We need a library to share the syntax between the server and client. In practice you would compile the following code into a library and implement it on both sides. Don’t be lazy and only share the source code. This cannot work, because two compilers generate two different system object IDs.

In today’s example we are running the code in one Visual Studio process. We are also using the localhost. This is why we do not need an external library on both sides. We are using the same compiler object ID.
The class MarshalByRefObject enables Remoting access to objects across application domain boundaries. We need to inherit this class.

public abstract class RemotingShared : MarshalByRefObject {

   public const string RemotingName = "MyRemotingName";
   public const string ServerIpAddress = "127.0.0.1";
   public const int Port = 65432;


   [Serializable]
   public class TradeData {
      public DateTime tradeTime;
      public string ticker;
      public double price;
      public int quantity;
         
      public override string ToString() {
         return tradeTime.ToString("dd.MMM.yy HH:mm:ss ") + ticker + " " + quantity + " @ " + price + " EUR";
      } //

      public TradeData(DateTime xTradeTime, string xTicker, double xPrice, int xQuantity) {
         tradeTime = xTradeTime;
         ticker = xTicker;
         price = xPrice;
         quantity = xQuantity;
      } // constructor

   } // class

   public abstract List<TradeData> GetTrades();
   public abstract string HelloWorld(string xName);
   public abstract DateTime Ping();      
} // class

The server overrides methods of the abstract class RemotingShared by inheriting it. We do not need to create any instance. This is done in the background by the Remoting process. The option WellKnownObjectMode.Singleton makes sure that only one instance will be created and reused. WellKnownObjectMode.SingleCall would create new instances for each incoming message.

public class RemotingSharedDerived : RemotingShared {

   public override List<RemotingShared.TradeData> GetTrades() {
      DateTime lDummyTime = new DateTime(2014, 02, 12, 16, 30, 10) ;
      RemotingShared.TradeData lTrade1 = new TradeData(lDummyTime, "DTE", 11.83, 100);
      RemotingShared.TradeData lTrade2 = new TradeData(lDummyTime.AddSeconds(2), "DAI", 66.45, 300);
      RemotingShared.TradeData lTrade3 = new TradeData(lDummyTime.AddSeconds(5), "DBK", 35.91, 100);
      return new List<TradeData>() { lTrade1, lTrade2, lTrade3 };
   } //

   public override string HelloWorld(string xName) {
      return "Hello " + xName;
   } //

   public override DateTime Ping() { 
      return DateTime.Now; 
   } //
} // class

public static void StartServer() {
   TcpChannel lTcpChannel = new TcpChannel(RemotingShared.Port);
   ChannelServices.RegisterChannel(lTcpChannel, true);
   Type lRemotingSharedType = typeof(RemotingSharedDerived);
   RemotingConfiguration.ApplicationName = RemotingShared.RemotingName + "App";
   RemotingConfiguration.RegisterWellKnownServiceType(lRemotingSharedType, RemotingShared.RemotingName, WellKnownObjectMode.Singleton);         
} //

Let’s have a look at the client now. As usual I kept the code as short as possible. Personally I do not like example programs that include too much redundant information, they can be quite confusing sometimes.
As we are on the localhost, the server has registered a ‘tcp’ channel already. We check if it exists although we already know it does. But the example program would throw an exception otherwise. Keep it in the code, it does make sense when you are remoting between two different IP addresses.

public static void StartClient() {
   string lPath = "tcp://" + RemotingShared.ServerIpAddress + ":" + RemotingShared.Port + "/" + RemotingShared.RemotingName;
   TcpChannel lTcpChannel = new TcpChannel();

   if (!ChannelServices.RegisteredChannels.Any(lChannel => lChannel.ChannelName == lTcpChannel.ChannelName)) {
      ChannelServices.RegisterChannel(lTcpChannel, true);
   }

   RemotingShared lShared = (RemotingShared)Activator.GetObject(typeof(RemotingShared), lPath);
   var o = lShared.GetTrades();
   var s = lShared.HelloWorld("Mr. Ohta");
   var p = lShared.Ping();

   Console.WriteLine("GetTrades():");
   foreach (var lTrade in o) Console.WriteLine(lTrade.ToString());
   Console.WriteLine();
   Console.WriteLine("HelloWorld(): " + s);
   Console.WriteLine();
   Console.WriteLine("Ping(): " + p);
} //

Let’s run the code now.

public static void Test() {
   RemotingServer.StartServer();
   RemotingClient.StartClient();
   Console.WriteLine("\nPress any key to exit.");
   Console.ReadKey();
} //

example output:
GetTrades():
12.Feb.14 16:30:10 DTE 100 @ 11.83 EUR
12.Feb.14 16:30:12 DAI 300 @ 66.45 EUR
12.Feb.14 16:30:15 DBK 100 @ 35.91 EUR

HelloWorld(): Hello Mr. Ohta

Ping(): 12/02/2014 20:36:55

Press any key to exit.

Compression (part 1, basics), GZip

We have done all sorts of things with data now. Compression is next on the agenda. The consideration to apply compression deals with quantity and speed. The more data you have, or the less network/hard disk speed you have, the more attracting compression becomes.
Streaming media would still be in the stone age without complex compression. And what about backup software?

For many local network messages we see a different picture. It might take longer to compress and then send data rather than to send the corresponding raw data. For small packages it is definitely not worth it. You need larger data to make compression efficient.

Some files like MP3 or JPG should not be compressed. They are compressed already. Theses files do not get smaller. You spend a lot of time compressing data just to have a larger file afterwards.

The .Net Framework supports GZip and Zip.
Today I will show you, how GZIP could be used to compress/decompress data easily with some built-in methods of the .Net Framework. I added a StopWatch to show how much time you would spend packing data. Do the maths and see how reasonable compression is compared to your device or network speed.

You obviously would not use a MemoryStream to generate the byte data. You can replace it by eg. a NetworkStream or a FileStream.

//byte[] lData = new byte[cDataSize];
//lDecompressorStream.Read(lData, 0, cDataSize);

The above lines are commented out in the following source code example. The reason for this is that the program does not use any data header to know the raw data size in advance. Therefore the final decompressed size cannot be allocated. The example would run smoothly, because the size is known in this case, it would a questionable example though.

using System;
using System.Diagnostics;
using System.IO;
using System.IO.Compression;
using System.Linq;
const int cDataSize = 10000;
const int cIterations = 1000;

public static void Test() {
   Stopwatch lWatch = new Stopwatch();
   lWatch.Start();

   byte[] lSource = GenerateRandomData(cDataSize);
   for (int i = 0, n = cIterations; i < n; i++) {
      //byte[] lSource = GenerateRandomData(cDataSize);
      byte[] lCompressed = Compress(lSource);
      byte[] lTarget = Decompress(lCompressed);

      if (i < n - 5) continue; // we just print the last 5 results
      // compare the result with the original data
      Console.WriteLine("before: " + string.Join(" ", lSource.Take(20)) + " ...  ,length: " + lSource.Length);
      Console.WriteLine("after:  " + string.Join(" ", lTarget.Take(20)) + " ...  ,length: " + lTarget.Length);
      Console.WriteLine("compressed size was: " + lCompressed.Length + " = " + (lCompressed.Length/(double)lSource.Length).ToString("0.00%"));
      Console.WriteLine();
   }

   lWatch.Stop();
   Console.WriteLine();
   Console.WriteLine("time elapsed: " + lWatch.ElapsedMilliseconds.ToString("#,##0") + " ms");
   Console.WriteLine("iterations: " + cIterations.ToString("#,##0"));

   Console.ReadLine();
} //


// creates random data with frequent repetitions
private static byte[] GenerateRandomData(int xDataSize) {
   byte[] lData = new byte[xDataSize];
   Random lRandom = new Random(DateTime.Now.Millisecond);
   for (int i = 0, n = lData.Length; i < n; i++) {
      lData[i] = (byte)lRandom.Next(0, 10);
   }

   return lData;
} //


private static byte[] Compress(byte[] xData) {
   MemoryStream lTargetStream = new MemoryStream();
   using (GZipStream lCompressorStream = new GZipStream(lTargetStream, CompressionMode.Compress)) {
      lCompressorStream.Write(xData, 0, xData.Length);
   }

   return lTargetStream.ToArray();
} //

private static byte[] Decompress(byte[] xCompressedData) {
   GZipStream lDecompressorStream = new GZipStream(new MemoryStream(xCompressedData), CompressionMode.Decompress);
   //byte[] lData = new byte[cDataSize];
   //lDecompressorStream.Read(lData, 0, cDataSize);
   MemoryStream lTargetStream = new MemoryStream();
   lDecompressorStream.CopyTo(lTargetStream);

   int lLength = (int)lTargetStream.Length;
   byte[] lDecompressedData = new byte[lLength];
   lTargetStream.Seek(0, SeekOrigin.Begin);
   lTargetStream.Read(lDecompressedData, 0, lLength);
         
   return lDecompressedData;
} //      

example output:
before: 8 5 6 1 2 3 0 8 8 6 5 5 0 4 4 8 3 2 9 0 … ,length: 10000
after: 8 5 6 1 2 3 0 8 8 6 5 5 0 4 4 8 3 2 9 0 … ,length: 10000
compressed size was: 5089 = 50.89%

before: 8 5 6 1 2 3 0 8 8 6 5 5 0 4 4 8 3 2 9 0 … ,length: 10000
after: 8 5 6 1 2 3 0 8 8 6 5 5 0 4 4 8 3 2 9 0 … ,length: 10000
compressed size was: 5089 = 50.89%

before: 8 5 6 1 2 3 0 8 8 6 5 5 0 4 4 8 3 2 9 0 … ,length: 10000
after: 8 5 6 1 2 3 0 8 8 6 5 5 0 4 4 8 3 2 9 0 … ,length: 10000
compressed size was: 5089 = 50.89%

before: 8 5 6 1 2 3 0 8 8 6 5 5 0 4 4 8 3 2 9 0 … ,length: 10000
after: 8 5 6 1 2 3 0 8 8 6 5 5 0 4 4 8 3 2 9 0 … ,length: 10000
compressed size was: 5089 = 50.89%

before: 8 5 6 1 2 3 0 8 8 6 5 5 0 4 4 8 3 2 9 0 … ,length: 10000
after: 8 5 6 1 2 3 0 8 8 6 5 5 0 4 4 8 3 2 9 0 … ,length: 10000
compressed size was: 5089 = 50.89%

time elapsed: 1,274 ms
iterations: 1,000

Encryption (part 2, basics, advanced), RSA

The last post dealt with symmetric encryption. Today we are going to deal with asymmetric encryption. This involves two keys; a public and a private key.
The .NET Framework provides the RSACryptoServiceProvider and DSACryptoServiceProvider classes for asymmetric encryption.

The public key must be known to encrypt data. The private key can be used to decrypt data. You cannot use a public key to decrypt data. A good YouTube introduction about private and public keys can be found here.
The two keys are mathematically related with each other. But you cannot determine the private key by knowing the public key and vice versa.
RSA (RSACryptoServiceProvider class) is a public key algorithm with key sizes ranging from 1024 to 4096. The key sizes are obviously much larger than for the AES algorithm. They did not contribute to increased security though.

We start with key generations, which are returned in the XML format.

public static void Test4() {
   string lPublicKey, lPrivateKey;
   GetKeys(out lPublicKey, out lPrivateKey);

   Console.WriteLine(Environment.NewLine + " P U B L I C  K E Y : " + Environment.NewLine);
   Console.WriteLine(lPublicKey);
   Console.WriteLine(Environment.NewLine + " P R I V A T E  K E Y : " + Environment.NewLine);
   Console.WriteLine(lPrivateKey);
   Console.ReadLine();
} //

private static void GetKeys(out string xPublicKey, out string xPrivateKey) {
   using (RSACryptoServiceProvider lRSA = new RSACryptoServiceProvider()) {
      xPublicKey = lRSA.ToXmlString(false);
      xPrivateKey = lRSA.ToXmlString(true);
   }
} //

edited/formatted example output:


<!--P U B L I C  K E Y :-->

<RSAKeyValue>
  <Modulus>
    qfwkAyqbHnI+a2f/xh9+3zufUE6drTrT7i83t5lFwvVQp1QGxSp0wrkNl3
    alkuUw48N4vfn3vbjpViB3fb0ndmpyRKx1Tffa2gYmYe2DNZw79kAR6mcJLIsedOluon93etw3cHtTef
    zXtj9aefTtAS6R/VNkYBKyPiwz4JbR7nM=
  </Modulus>
  <Exponent>AQAB</Exponent>
</RSAKeyValue>

<!--P R I V A T E  K E Y :-->

<RSAKeyValue>
  <Modulus>
    qfwkAyqbHnI+a2f/xh9+3zufUE6drTrT7i83t5lFwvVQp1QGxSp0wrkNl3
    alkuUw48N4vfn3vbjpViB3fb0ndmpyRKx1Tffa2gYmYe2DNZw79kAR6mcJLIsedOluon93etw3cHtTef
    zXtj9aefTtAS6R/VNkYBKyPiwz4JbR7nM=
  </Modulus>
  <Exponent>AQAB</Exponent>
  <P> 0K3ebgo/zcAZ5JHM2s7O78SdiLnXthO0IGOKYHfkZeegXMb8AzKpj38DwkNigkQ+rptFmwQuMTl9Eura+BtTGQ== </P>
  <Q> 0IgDlR6wS5TqnJmCpqb70fqMRhrMLEwFAvJlS3SFxLd+TiZSX+up68s+mg8BhhcsgkSnaSUxkzm/JYfZhxkraw== </Q>
  <DP> ibEf4kXbAbzumNXncL0i6Cw4sh3GCrsHkJN9m9egGely86TMZqPIJAnwBf+GgWPcZEPJ8tYYUJyZPaE/qJQHCQ== </DP>
  <DQ> MsHonVNq9fq5YIS9GHNsuB+UJTxAlkeqsJzvqv4h0VAYnk0Vn+Ns6Mf/5N/iLxFU9CBh32X+OyfDLw9yE0A9IQ== </DQ>
  <InverseQ> CgD4wQfd6bfcJgHKmmkXmoWTkz3VT722jiQ5mwSIjbGo6sZ0zBBF8qUJNzsybpg+ilqzStcOQqwO2lqwHqnw8g== </InverseQ>
  <D>
    TiN3snTtZWuCwgDGlJ55xcg0jcf1t2Hpdf4CkMVGSj5WWvTHP+8qSTCjzNJffk0Y0jpS0JGNjor
    nyA2YoBZJgufWeEv2rNfTkblVaLx+1nWlFJ2hWz80XbaBeK4zpA1sf8SwnUefdxnqmjDs0Jc0vUwnhFP
    HfXZ6hD02uTwJxSE=
  </D>
</RSAKeyValue>

You do not have to work with XML. You can also access binary data directly by using the methods ExportParameters or ExportCspBlob. A definition for blob can be found here.

private static void GetKeys() {
   using (RSACryptoServiceProvider lRSA = new RSACryptoServiceProvider()) {
      RSAParameters xPublicKey = lRSA.ExportParameters(false);
      byte[] lP = xPublicKey.P;
      byte[] lQ = xPublicKey.Q;
      byte[] lModulus = xPublicKey.Modulus;
      byte[] lExponent = xPublicKey.Exponent;
      ...

      byte[] xPrivateKey = lRSA.ExportCspBlob(true);          
   }
} //

We have two keys now. Let’s put that into practice.

private static void GetKeys(out string xPublicKey, out string xPrivateKey) {
   using (RSACryptoServiceProvider lRSA = new RSACryptoServiceProvider()) {
      xPublicKey = lRSA.ToXmlString(false);
      xPrivateKey = lRSA.ToXmlString(true);
   }
} //

public static void HowToUseAsymmetricEncryption() {
   string lText = "Let's encrypt and decrypt this text :)";
   byte[] lData = UnicodeEncoding.Unicode.GetBytes(lText);

   string lPublicKey, lPrivateKey;
   GetKeys(out lPublicKey, out lPrivateKey);

   byte[] lEncrypted;
   using (RSACryptoServiceProvider lRSA = new RSACryptoServiceProvider()) {
      lRSA.FromXmlString(lPublicKey);
      lEncrypted = lRSA.Encrypt(lData, false);
   }

   byte[] lDecrypted;
   using (RSACryptoServiceProvider lRSA = new RSACryptoServiceProvider()) {
      lRSA.FromXmlString(lPrivateKey);
      lDecrypted = lRSA.Decrypt(lEncrypted, false);
   }

   string lResult = UnicodeEncoding.Unicode.GetString(lDecrypted);
   Console.WriteLine(lResult);
   Console.ReadLine();
} //

There are many ways to store keys. One is to store them on the computer, so they can be accessed from anywhere. This can be achieved by setting “RSACryptoServiceProvider.UseMachineKeyStore” to true. Or you can instantiate CspParameters and set the UseMachineKeyStore in Flags (see following example).
The UseMachineKeyStore property applies to all code in the current application domain, whereas a CspParameters object applies only to classes that explicitly reference it. By instanciating RSACryptoServiceProvider with CspParameters as the constructor parameter, the keys get stored automatically. There is no explicit Store() method.
To delete the keys simply use the RSACryptoServiceProvider instance, set PersistKeyInCsp to false and then call the method Clear().
Uncomment these two lines in the code and you will see the exception “Key not valid for use in specified state”. The keys of the new instance are not the same anymore.

private static void GetKeys(out string xPublicKey, out string xPrivateKey) {
   using (RSACryptoServiceProvider lRSA = new RSACryptoServiceProvider()) {
      xPublicKey = lRSA.ToXmlString(false);
      xPrivateKey = lRSA.ToXmlString(true);
   }
} //

private static CspParameters GetCspParameters() {
   CspParameters lCspParams = new CspParameters();
   lCspParams.Flags |= CspProviderFlags.UseMachineKeyStore;   // all users of this computer have access
   //lCspParams.Flags |= CspProviderFlags.UseUserProtectedKey;  // a popup window will ask for confirmation
   //lCspParams.Flags |= CspProviderFlags.UseNonExportableKey;  // you can use the key, but not read or export it
   lCspParams.KeyContainerName = "MySecretKeyContainer";
   return lCspParams;
} //

public static void HowToUseAsymmetricEncryption2() {
   CspParameters lCspParams;
   string lText = "Let's encrypt and decrypt this text :)";
   byte[] lData = UnicodeEncoding.Unicode.GetBytes(lText);

   string lPublicKey, lPrivateKey;
   GetKeys(out lPublicKey, out lPrivateKey);

   try {
      lCspParams = GetCspParameters(); // new instance         

      byte[] lEncrypted;
      using (RSACryptoServiceProvider lRSA = new RSACryptoServiceProvider(lCspParams)) {
         lRSA.FromXmlString(lPublicKey);
         lRSA.FromXmlString(lPrivateKey); // assigned here to show that we assigned it to a different RSACryptoServiceProvider instance
         lEncrypted = lRSA.Encrypt(lData, false);

         byte[] x = lRSA.ExportCspBlob(true);
         
         // how to delete a KeyContainer
         // lRSA.PersistKeyInCsp = false; 
         // lRSA.Clear();
      }


      lCspParams = GetCspParameters(); // new instance to demonstrate the independence

      using (RSACryptoServiceProvider lRSA = new RSACryptoServiceProvider(lCspParams)) {
         byte[] lDecrypted = lRSA.Decrypt(lEncrypted, false);
         string lResult = UnicodeEncoding.Unicode.GetString(lDecrypted);
         Console.WriteLine(lResult);
      }
   }
   catch (Exception ex) {
      Console.WriteLine(ex.Message + Environment.NewLine);
      Console.WriteLine(ex.StackTrace);
   }

   Console.ReadLine();
} //

I would recommend playing with the “CspProviderFlags.UseUserProtectedKey” flag. A pop-up should show up when you uncomment the line in method GetCspParameters().

Key

Also test the “spProviderFlags.UseNonExportableKey” flag. I added the nondescript line byte[] x = lRSA.ExportCspBlob(true) to demonstrate the effect .
The nicest exception message wins the contest 😉

Encryption (part 1, basics), AES

After serializing, storing and sending data across the network, we are still working on the same subject. Cryptography does belong into that area as well.
The simplified process is:

data => password => encryption => cipher => password => decryption => data

I am going to concentrate on the well-known AES algorithm. It has a good and reliable history. You can hardly hack it with eg. three allowable attempts per five minutes. As usual I have added links to this post for further explanations. AES (Advanced Encryption Standard) is reportedly used by the U.S. government and was recommended by the U.S. National Institute of Standards and Technology (NIST).

The .Net Framework supports cryptography in the System.Security.Cryptography namespace. The AES algorithm can be found in the AesManaged class. It uses symmetric encryption; there is only one password.

Besides the password there is a so-called Initialization Vector (IV). It adds more randomness. The IV ensures that data does not result in the same cipher data. You could pretty much say that each result is unique. Repetitive patterns are much harder to find.

Users enter passwords, but the AES algorithm does not use these directly. It requires keys instead. Therefore we must translate passwords into keys.

Key sizes of 128, 160, 192, 224, and 256 bits are supported by the algorithm, but only the 128, 192, and 256-bit key sizes are specified in the AES standard.
Block sizes of 128, 160, 192, 224, and 256 bits are supported by the algorithm, but only the 128-bit block size is specified in the AES standard.

using System;
using System.IO;
using System.Linq;
using System.Security.Cryptography;
using System.Text;

Some extension methods can be found in System.Linq. We use string.Join().

private static AesManaged GetAesAlgo() {
   AesManaged lAlgo = new AesManaged();
   lAlgo.BlockSize = 128;
   lAlgo.KeySize = 128;
   lAlgo.Key = ASCIIEncoding.ASCII.GetBytes("Bastian/M/K/Ohta"); // 128 bits = 16 bytes = 16 ASCII characters
   lAlgo.IV = ASCIIEncoding.ASCII.GetBytes("1234567890123456"); // 128 bits = 16 bytes = 16 ASCII characters
   Console.WriteLine("Aes block size is " + lAlgo.BlockSize + " bits");
   return lAlgo;
} //

static byte[] Encrypt(SymmetricAlgorithm xAlgo, byte[] xData) {
   ICryptoTransform lEncryptor = xAlgo.CreateEncryptor(xAlgo.Key, xAlgo.IV);
   return lEncryptor.TransformFinalBlock(xData, 0, xData.Length);
} //

static byte[] Decrypt(SymmetricAlgorithm xAlgo, byte[] xCipher) {
   ICryptoTransform lDecryptor = xAlgo.CreateDecryptor(xAlgo.Key, xAlgo.IV);
   return lDecryptor.TransformFinalBlock(xCipher, 0, xCipher.Length);
} //

public static void Test1() {
   string lText = "Let's encrypt and decrypt this text :)";
   byte[] lTextAsBytes = UnicodeEncoding.Unicode.GetBytes(lText);

   Console.WriteLine("text length in characters " + lText.Length);
   Console.WriteLine("text length in bytes " + lTextAsBytes.Length);

   using (SymmetricAlgorithm lAlgo = GetAesAlgo()) {
      byte[] lEncrypted = Encrypt(lAlgo, lTextAsBytes);
      Console.WriteLine("encrypted data size in bytes " + lTextAsBytes.Length);

      byte[] lDecrypted = Decrypt(lAlgo, lEncrypted);
      Console.WriteLine("decrypted text length in bytes " + lDecrypted.Length);

      string lResult = UnicodeEncoding.Unicode.GetString(lDecrypted);
      Console.WriteLine("text length in characters now " + lResult.Length);
      Console.WriteLine();
      Console.WriteLine("the text was: \"" + lResult + "\"");
   }
   Console.ReadLine();
} //

example output:
text length in characters 38
text length in bytes 76
Aes block size is 128 bits
encrypted data size in bytes 76
decrypted text length in bytes 76
text length in characters now 38

the text was: “Let’s encrypt and decrypt this text :)”

The above code was for warming up. It encrypts/decrypts all data in memory without the use of any stream. The next example uses the FileStream class to store a file to the desktop. Some people prefer to use the memory stream in their examples. I personally do not get the point of this as it is not a requirement. I did show above that you can convert something in memory without using the MemoryStream class.

private static AesManaged GetAesAlgo() {
   AesManaged lAlgo = new AesManaged();
   lAlgo.BlockSize = 128;
   lAlgo.KeySize = 128;
   lAlgo.Key = ASCIIEncoding.ASCII.GetBytes("Bastian/M/K/Ohta"); // 128 bits = 16 bytes = 16 ASCII characters
   lAlgo.IV = ASCIIEncoding.ASCII.GetBytes("1234567890123456"); // 128 bits = 16 bytes = 16 ASCII characters
   Console.WriteLine("Aes block size is " + lAlgo.BlockSize + " bits");
   return lAlgo;
} //

public static void Test2() {
   string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
   string lFileName = lDesktopPath + "Encrypted.bin";
   string lText = "Let's encrypt and decrypt this text :)";
   byte[] lTextAsBytes = UnicodeEncoding.Unicode.GetBytes(lText);

   Console.WriteLine("text length in characters " + lText.Length);
   Console.WriteLine("text length in bytes " + lTextAsBytes.Length);

   try {
      using (SymmetricAlgorithm lAlgo = GetAesAlgo()) {
         EncryptToStream(lAlgo, lFileName, lTextAsBytes);

         FileInfo lFileInfo = new FileInfo(lFileName);
         Console.WriteLine("encrypted file size in bytes " + lFileInfo.Length);

         int lLength;
         byte[] lDecrypted = DecryptFromStream(lAlgo, lFileName, out lLength);
         Console.WriteLine("decrypted text length in bytes " + lDecrypted.Length);
         //Console.WriteLine("decrypted text length in bytes " + lLength);

         string lResult = UnicodeEncoding.Unicode.GetString(lDecrypted);
         //string lResult = UnicodeEncoding.Unicode.GetString(lDecrypted, 0, lLength);

         Console.WriteLine("text length in characters now " + lResult.Length);
         Console.WriteLine();
         Console.WriteLine("text before encryption: \"" + lText + "\"");
         Console.WriteLine("text after decryption: \"" + lResult + "\"");
      }
   }
   catch (Exception ex) { Console.WriteLine(ex.Message); }
   Console.ReadLine();
} //

static void EncryptToStream(SymmetricAlgorithm xAlgo, string xFileName, byte[] xData) {
   ICryptoTransform lEncryptor = xAlgo.CreateEncryptor(xAlgo.Key, xAlgo.IV);
   System.IO.File.Delete(xFileName);
   using (FileStream lFileStream = new FileStream(xFileName, FileMode.CreateNew)) {
      using (CryptoStream lCryptoStream = new CryptoStream(lFileStream, lEncryptor, CryptoStreamMode.Write)) {
         lCryptoStream.Write(xData, 0, xData.Length);
      }
   }
} //

static byte[] DecryptFromStream(SymmetricAlgorithm xAlgo, string xFileName, out int xLength) {
   ICryptoTransform lDecryptor = xAlgo.CreateDecryptor(xAlgo.Key, xAlgo.IV);
   using (FileStream lFileStream = new FileStream(xFileName, FileMode.Open)) {
      using (CryptoStream lCryptoStream = new CryptoStream(lFileStream, lDecryptor, CryptoStreamMode.Read)) {
         int lLength = (int)lFileStream.Length;
         byte[] lResult = new byte[lLength];
         xLength = lCryptoStream.Read(lResult, 0, lLength);
         return lResult;
      }
   }
} //

example output:
text length in characters 38
text length in bytes 76
Aes block size is 128 bits
encrypted file size in bytes 80
decrypted text length in bytes 80
text length in characters now 40

text before encryption: “Let’s encrypt and decrypt this text :)”
text after decryption: “Let’s encrypt and decrypt this text 🙂 ”

I added a typical beginner’s mistake. And to point it out I printed a lot of text messages about the data sizes. Comment and uncomment the following lines to correct the mistake:

Console.WriteLine("decrypted text length in bytes " + lDecrypted.Length);
//Console.WriteLine("decrypted text length in bytes " + lLength);

string lResult = UnicodeEncoding.Unicode.GetString(lDecrypted);
//string lResult = UnicodeEncoding.Unicode.GetString(lDecrypted, 0, lLength);

CHANGE THE ABOVE LINES TO:

//Console.WriteLine("decrypted text length in bytes " + lDecrypted.Length);
Console.WriteLine("decrypted text length in bytes " + lLength);

//string lResult = UnicodeEncoding.Unicode.GetString(lDecrypted);
string lResult = UnicodeEncoding.Unicode.GetString(lDecrypted, 0, lLength);

example output:
text length in characters 38
text length in bytes 76
Aes block size is 128 bits
encrypted file size in bytes 80
decrypted text length in bytes 76
text length in characters now 38

text before encryption: “Let’s encrypt and decrypt this text :)”
text after decryption: “Let’s encrypt and decrypt this text :)”

The block size is 128 bits (=16 bytes). 76 bytes are not a multiple of 16, but 80 bytes are. This results in a larger file, which uses padding at the end. Taking the wrong size did result in a wrong decrypted text.

Usually you do not store passwords. Passwords are used as keys to decrypt data. If the password is wrong, then the decoded data is wrong. Some programmers hardcode passwords to lazily compare them against password entries. This is really bad practice!
Anyhow, some applications do store passwords to make logins easier. Eg. your internet browser has such functionality. In this case you should salt the result. Salt makes cracking passwords more time-consuming, it doesn’t stop criminals though. Nevertheless, every little helps. John Doe will not be able to use your salted password.

The good old hardcore hackers did not care much about passwords. There was no need to crack them. Hackers had freezer modules that could freeze the computer and gave access to all memory including the running assembler code. You only needed to freeze the PC when it was asking for a password. You checked where your program counter was and then did the same right after you had entered the password.
All you had to do was to add an unconditional branch command to bypass this password request. Hackers were then looking for the same code sequence on the disk and changed it accordingly. These days work a bit different.
But why am I telling this? I want to emphasize that you have to encrypt executable code as well to protect yourself efficiently. It is not enough to only encrypt data. The best code decrypts itself bit by bit at runtime.

Back to salt.
Sometimes users use the same password. When you store these passwords with individual salt values in your database, then they do look entirely different. Again, this makes John Doe’s criminal life or his simple temptation harder. A static salt value is not a good idea. You’d better store the salt value right next to your password. You can even encrypt the salt value with hardcoded algorithm parameters. This makes it a bit safer again.

public static void Test3() {

   // variable salt
   Console.WriteLine("random salt");
   string lPassword = "MySooperDooperSekr1tPa$$wörd";
   byte[] lKey, lSalt;
   GenerateSaltedKey(lPassword, out lKey, out lSalt, 128);
   Console.WriteLine("password: " + lPassword);
   Console.WriteLine("key: " + string.Join(" ", lKey));
   Console.WriteLine("salt: " + string.Join(" ", lSalt));
   Console.WriteLine();
   Console.WriteLine("saving to database");
   Console.WriteLine("[Base64String] key: " + Convert.ToBase64String(lKey));
   Console.WriteLine("[Base64String] salt: " + Convert.ToBase64String(lSalt));
   Console.WriteLine();

   // backtest
   byte[] lKey2 = GenerateKey128(lPassword, lSalt); // do we get the same key?
   Console.WriteLine("key: " + string.Join(" ", lKey));
   Console.WriteLine("key: " + string.Join(" ", lKey2));
   if (lKey.SequenceEqual(lKey2)) Console.WriteLine("keys are equal");
   else Console.WriteLine("Oh dear, something went wrong!");
   Console.WriteLine();


   // static salt
   Console.WriteLine("static salt");
   for (int i = 0; i < 10; i++) {
      byte[] lStaticKey = GenerateKey128(lPassword);
      Console.WriteLine(i + " static key " + string.Join(" ", lStaticKey));
   }

   Console.ReadLine();
} //

private const int _Iterations = 2500;

private static void GenerateSaltedKey(string xPassword, out byte[] xKey, out byte[] xSalt, int xKeySize = 256) {
   xKeySize /= 8; // now in bytes
   var keyGenerator = new Rfc2898DeriveBytes(xPassword, xKeySize, _Iterations);
   xKey = keyGenerator.GetBytes(xKeySize);
   xSalt = keyGenerator.Salt;
} //

private static byte[] GenerateKey128(string xPassword) {
   byte[] lSalt = { 252, 132, 52, 13, 64, 158, 12, 10, 50, 80, 74, 63, 15, 54, 76, 246 }; // 16 bytes == 128 bits
   return new Rfc2898DeriveBytes(xPassword, lSalt, _Iterations).GetBytes(16);
} //

private static byte[] GenerateKey128(string xPassword, byte[] xSalt) {
   return new Rfc2898DeriveBytes(xPassword, xSalt, _Iterations).GetBytes(16);
} //

example output:
random salt
password: MySooperDooperSekr1tPa$$wörd
key: 139 209 30 237 82 66 102 245 193 89 175 218 62 190 7 8
salt: 58 247 144 225 92 236 82 167 255 185 6 135 45 104 86 98

saving to database
[Base64String] key: i9Ee7VJCZvXBWa/aPr4HCA==
[Base64String] salt: OveQ4VzsUqf/uQaHLWhWYg==

key: 139 209 30 237 82 66 102 245 193 89 175 218 62 190 7 8
key: 139 209 30 237 82 66 102 245 193 89 175 218 62 190 7 8
keys are equal

static salt
0 static key 103 183 126 80 87 205 118 92 45 195 66 134 124 104 32 206
1 static key 103 183 126 80 87 205 118 92 45 195 66 134 124 104 32 206
2 static key 103 183 126 80 87 205 118 92 45 195 66 134 124 104 32 206
3 static key 103 183 126 80 87 205 118 92 45 195 66 134 124 104 32 206
4 static key 103 183 126 80 87 205 118 92 45 195 66 134 124 104 32 206
5 static key 103 183 126 80 87 205 118 92 45 195 66 134 124 104 32 206
6 static key 103 183 126 80 87 205 118 92 45 195 66 134 124 104 32 206
7 static key 103 183 126 80 87 205 118 92 45 195 66 134 124 104 32 206
8 static key 103 183 126 80 87 205 118 92 45 195 66 134 124 104 32 206
9 static key 103 183 126 80 87 205 118 92 45 195 66 134 124 104 32 206

I did set the number of iterations to 2500 (private const int _Iterations = 2500). This is an arbitrary number. A value of above 1000 is generally recommended. It tells the key/IV generator how many times it should run. Think of recursion to get an idea. Make sure you use the same number of iterations whenever you want the same results.

Protocol Buffers (part 3, advanced), Tcp Networking

Ok guys, hardcore. Unfortunately in a different way than I thought 😉
Once again it took a lot of time to prepare the next source code example. It extends the code of my last post.

I removed the ProtoType class, because inheritance is not a real strength of Protobuf-Net. It is possible, but the ease of code maintenance is a definite argument against it. The complexity increases slowly, we cannot afford code that is difficult to change.

In theory we could write a Serialize() method for each class, the result would be extreme performance. You could use the BitConverter class to convert and then merge values without causing too much overhead. And by using Assembler I guess you could even make it 10 times faster again.

I think that all modern protocols lose a lot of time due to the required reflection, which is mainly the mapping of methods and properties. And unlike Assembler the modern languages do not allow to eg. simply write an integer to a memory address and only read the first byte of it.
You can rotate bits or multiply and divide, what is much less efficient.

In C# you can use the Parallel class to boost your performance. Of course the processing time of each code piece has to be substantial, otherwise it takes longer to create tasks than to solve the problems. In slow networks it might even be worth considering packing data before it is sent.

I replaced the ProtoType class by the Header class, which tells the type of the succeeding data block and also adds a serial identifier. This identifier can be used to receive vital feedbacks. For instance the server tells the client if the data transmission was successful or not. It can also send calculation results or error messages.

The data stream has the following order: header, data, header, data, header, data … except for feedback headers; they do not need data blocks. The enum eType inside the Header class defines all possible data types: ErrorMessage, Feedback, Book or Fable.
As said before I did not use inheritance for the Header class. The code remains legible and there is no spaghetti code to achieve indirect multiple inheritance.

In my last post, the client was sending and the server was receiving data. Now the communication is bidirectional. The client uses two threads, one to send and one to receive. The sending method is still using the BlockingCollection construction and an endless loop on a separate thread.
The server can be connected to several clients simultaneously. To keep it simple I decided sending data without any context switching. This usually blocks threads during the IO operation. I have added tasks inside the event OnMessageBook to give an example on how to avoid this. Nevertheless the send method uses a lock on the client preventing simultaneous write actions on the socket. This is bad practice; you should not apply locks on objects that might be locked in other areas as well. This is out of your scope, you don’t know if the .Net framework or any other code uses a lock on the same object, which could cause undesired behavior. In the below example it would have been better to wrap the client object into another class and apply the lock on that outer shell object. But I guess this is ok in our shorter example. The code was long enough already.

public static void Send(TcpClient xClient, ProtoBufExample.Header xHeader) {
  if (xHeader == null) return;
  if (xClient == null) return;

  lock (xClient) {
     NetworkStream lNetworkStream = xClient.GetStream();
  ....

The above lock problem could be avoided with this:

public class ClientData {
   public TcpClient Client { get; private set; }
   private DateTime _ConnectedSince;
   private string _UserName;
   ....

public static void Send(ClientData xClient, ProtoBufExample.Header xHeader) {
  if (xHeader == null) return;
  if (xClient == null) return;

  lock (xClient) {
     NetworkStream lNetworkStream = xClient.Client.GetStream();
  ....

And here is the entire source code:

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using ProtoBuf;

namespace DemoApp {

   public class ProtoBufExample {

      public enum eType : byte { eError = 0, eFeedback, eBook, eFable };

      [ProtoContract]
      public class Header {
         [ProtoMember(1)] public eType objectType;
         [ProtoMember(2)] public readonly int serialMessageId;

         public object data;
         private static int _HeaderSerialId = 0;

         public Header(object xData, eType xObjectType, int xSerialMessageId = 0) {
            data = xData;
            serialMessageId = (xSerialMessageId == 0) ? Interlocked.Increment(ref _HeaderSerialId) : xSerialMessageId;
            objectType = xObjectType; // we could use "if typeof(T) ...", but it would be slower, harder to maintain and less legible
         } // constructor

         // parameterless constructor needed for Protobuf-net
         public Header() {
         } // constructor

      } // class

      [ProtoContract]
      public class ErrorMessage {
         [ProtoMember(1)]
         public string Text;
      } // class

      [ProtoContract]
      public class Book {
         [ProtoMember(1)] public string author;
         [ProtoMember(2, DataFormat = DataFormat.Group)] public List<Fable> stories;
         [ProtoMember(3)] public DateTime edition;
         [ProtoMember(4)] public int pages;
         [ProtoMember(5)] public double price;
         [ProtoMember(6)] public bool isEbook;

         public override string ToString() {
            StringBuilder s = new StringBuilder();
            s.Append("by "); s.Append(author);
            s.Append(", edition "); s.Append(edition.ToString("dd MMM yyyy"));
            s.Append(", pages "); s.Append(pages);
            s.Append(", price "); s.Append(price);
            s.Append(", isEbook "); s.Append(isEbook);
            s.AppendLine();
            if (stories != null) foreach (Fable lFable in stories) {
                  s.Append("title "); s.Append(lFable.title);
                  s.Append(", rating "); s.Append(lFable.customerRatings.Average()); // Average() is an extension method of "using System.Linq;"
                  s.AppendLine();
               }

            return s.ToString();
         } //
      } // class

      [ProtoContract]
      public class Fable {
         [ProtoMember(1)] public string title;
         [ProtoMember(2, DataFormat = DataFormat.Group)]
         public double[] customerRatings;

         public override string ToString() {
            return "title " + title + ", rating " + customerRatings.Average();
         } //
      } // class

      public static Book GetData() {
         return new Book {
            author = "Aesop",
            price = 1.99,
            isEbook = false,
            edition = new DateTime(1975, 03, 13),
            pages = 203,
            stories = new List<Fable>(new Fable[] {
                new Fable{ title = "The Fox & the Grapes", customerRatings = new double[]{ 0.7, 0.7, 0.8} },
                new Fable{ title = "The Goose that Laid the Golden Eggs", customerRatings = new double[]{ 0.6, 0.75, 0.5, 1.0} },
                new Fable{ title = "The Cat & the Mice", customerRatings = new double[]{ 0.1, 0.0, 0.3} },
                new Fable{ title = "The Mischievous Dog", customerRatings = new double[]{ 0.45, 0.5, 0.4, 0.0, 0.5} }
            })
         };
      } //
   } // class

   public class PendingFeedbacks {
      private readonly ConcurrentDictionary<int, ProtoBufExample.Header> _Messages = new ConcurrentDictionary<int, ProtoBufExample.Header>();

      public int Count { get { return _Messages.Count; } }

      public void Add(ProtoBufExample.Header xHeader) {
         if (xHeader == null) throw new Exception("cannot add a null header");

         if (!_Messages.TryAdd(xHeader.serialMessageId, xHeader)) {
            throw new Exception("there must be a programming error somewhere");
         }
      } //

      public void Remove(ProtoBufExample.Header xHeader) {
         ProtoBufExample.Header lHeader;
         if (!_Messages.TryRemove(xHeader.serialMessageId, out lHeader)) {
            throw new Exception("there must be a programming error somewhere");
         }

         switch (xHeader.objectType) {
            case ProtoBufExample.eType.eError:
               Console.WriteLine("error: " + ((ProtoBufExample.ErrorMessage)xHeader.data).Text);
               Console.WriteLine("the message that was sent out was: " + lHeader.objectType + " with serial id " + lHeader.serialMessageId);
               Console.WriteLine("please check the log files" + Environment.NewLine);
               break;
            case ProtoBufExample.eType.eFeedback:
               // all ok !
               break;
            default:
               Console.WriteLine("warning: This message type was not expected.");
               break;
         }
      } //
   } // class

   public static class NetworkTest {
      public static void Test() {
         NetworkListener lServer = new NetworkListener("127.0.0.1", 65432, "Server");
         NetworkClient lClient = new NetworkClient("127.0.0.1", 65432, "Client");

         lServer.Connect();
         lServer.OnMessageBook += new NetworkListener.dOnMessageBook(OnMessageBook);
         lServer.OnMessageFable += new NetworkListener.dOnMessageFable(OnMessageFable);

         lClient.Connect();

         ProtoBufExample.Header lHeader;

         // send a book across the network
         ProtoBufExample.Book lBook = ProtoBufExample.GetData();
         lHeader = new ProtoBufExample.Header(lBook, ProtoBufExample.eType.eBook);
         lClient.Send(lHeader);

         System.Threading.Thread.Sleep(1000);  // remove this to see the asynchonous processing (the output will look terrible)

         // send a fable across the network
         lHeader = new ProtoBufExample.Header(lBook.stories[1], ProtoBufExample.eType.eFable);
         lClient.Send(lHeader);

         System.Threading.Thread.Sleep(1000);

         lClient.Disconnect();
         lServer.Disconnect();

         Console.ReadLine();
      }

      // demo: synchronous processing
      static void OnMessageFable(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Fable xFable) {
         Console.WriteLine(Environment.NewLine + "received a fable: ");
         Console.WriteLine(xFable.ToString());

         // demo: we tell the server that something went wrong
         ProtoBufExample.ErrorMessage lErrorMessage = new ProtoBufExample.ErrorMessage() { Text = "The fable was rejected. It is far too short." };
         ProtoBufExample.Header lErrorHeader = new ProtoBufExample.Header(lErrorMessage, ProtoBufExample.eType.eError, xHeader.serialMessageId);
         NetworkListener.Send(xSender, lErrorHeader);
      } //

      // demo: asynchronous processing
      static void OnMessageBook(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Book xBook) {
         Task.Factory.StartNew(() => {
            Console.WriteLine(Environment.NewLine + "received a book: ");
            Console.WriteLine(xBook.ToString());

            // send a feedback without any body to signal all was ok
            ProtoBufExample.Header lFeedback = new ProtoBufExample.Header(null, ProtoBufExample.eType.eFeedback, xHeader.serialMessageId);
            NetworkListener.Send(xSender, lFeedback);
            return;
         });

         Console.WriteLine("Book event was raised");
      } //

   } // class

   public class NetworkListener {

      private bool _ExitLoop = true;
      private TcpListener _Listener;

      public delegate void dOnMessageBook(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Book xBook);
      public event dOnMessageBook OnMessageBook;

      public delegate void dOnMessageFable(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Fable xFable);
      public event dOnMessageFable OnMessageFable;

      private List<TcpClient> _Clients = new List<TcpClient>();

      public int Port { get; private set; }
      public string IpAddress { get; private set; }
      public string ThreadName { get; private set; }

      public NetworkListener(string xIpAddress, int xPort, string xThreadName) {
         Port = xPort;
         IpAddress = xIpAddress;
         ThreadName = xThreadName;
      } //

      public bool Connect() {
         if (!_ExitLoop) {
            Console.WriteLine("Listener running already");
            return false;
         }
         _ExitLoop = false;

         try {
            _Listener = new TcpListener(IPAddress.Parse(IpAddress), Port);
            _Listener.Start();

            Thread lThread = new Thread(new ThreadStart(LoopWaitingForClientsToConnect));
            lThread.IsBackground = true;
            lThread.Name = ThreadName + "WaitingForClients";
            lThread.Start();

            return true;
         }
         catch (Exception ex) { Console.WriteLine(ex.Message); }
         return false;
      } //

      public void Disconnect() {
         _ExitLoop = true;
         lock (_Clients) {
            foreach (TcpClient lClient in _Clients) lClient.Close();
            _Clients.Clear();
         }
      } //        

      private void LoopWaitingForClientsToConnect() {
         try {
            while (!_ExitLoop) {
               Console.WriteLine("waiting for a client");
               TcpClient lClient = _Listener.AcceptTcpClient();
               string lClientIpAddress = lClient.Client.LocalEndPoint.ToString();
               Console.WriteLine("new client connecting: " + lClientIpAddress);
               if (_ExitLoop) break;
               lock (_Clients) _Clients.Add(lClient);

               Thread lThread = new Thread(new ParameterizedThreadStart(LoopRead));
               lThread.IsBackground = true;
               lThread.Name = ThreadName + "CommunicatingWithClient";
               lThread.Start(lClient);
            }
         }
         catch (Exception ex) { Console.WriteLine(ex.Message); }
         finally {
            _ExitLoop = true;
            if (_Listener != null) _Listener.Stop();
         }
      } // 

      private void LoopRead(object xClient) {
         TcpClient lClient = xClient as TcpClient;
         NetworkStream lNetworkStream = lClient.GetStream();

         while (!_ExitLoop) {
            try {
               ProtoBufExample.Header lHeader = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Header>(lNetworkStream, ProtoBuf.PrefixStyle.Fixed32);
               if (lHeader == null) break; // happens during shutdown process
               switch (lHeader.objectType) {

                  case ProtoBufExample.eType.eBook:
                     ProtoBufExample.Book lBook = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Book>(lNetworkStream, ProtoBuf.PrefixStyle.Fixed32);
                     if (lBook == null) break;
                     lHeader.data = lBook; // not necessary, but nicer                            

                     dOnMessageBook lEventBook = OnMessageBook;
                     if (lEventBook == null) continue;
                     lEventBook(lClient, lHeader, lBook);
                     break;

                  case ProtoBufExample.eType.eFable:
                     ProtoBufExample.Fable lFable = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Fable>(lNetworkStream, ProtoBuf.PrefixStyle.Fixed32);
                     if (lFable == null) break;
                     lHeader.data = lFable; // not necessary, but nicer                            

                     dOnMessageFable lEventFable = OnMessageFable;
                     if (lEventFable == null) continue;
                     lEventFable(lClient, lHeader, lFable);
                     break;

                  default:
                     Console.WriteLine("Mayday, mayday, we are in big trouble.");
                     break;
               }
            }
            catch (System.IO.IOException) {
               if (_ExitLoop) Console.WriteLine("user requested client shutdown");
               else Console.WriteLine("disconnected");
            }
            catch (Exception ex) { Console.WriteLine(ex.Message); }
         }
         Console.WriteLine("server: listener is shutting down");
      } //

      public static void Send(TcpClient xClient, ProtoBufExample.Header xHeader) {
         if (xHeader == null) return;
         if (xClient == null) return;

         lock (xClient) {
            try {
               NetworkStream lNetworkStream = xClient.GetStream();

               // send header (most likely a simple feedback)
               ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Header>(lNetworkStream, xHeader, ProtoBuf.PrefixStyle.Fixed32);

               // send errors
               if (xHeader.objectType != ProtoBufExample.eType.eError) return;
               ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.ErrorMessage>(lNetworkStream, (ProtoBufExample.ErrorMessage)xHeader.data, ProtoBuf.PrefixStyle.Fixed32);
            }
            catch (Exception ex) { Console.WriteLine(ex.Message); }
         }

      } //

   } // class

   public class NetworkClient {
      public int Port { get; private set; }
      public string IpAddress { get; private set; }
      public string ThreadName { get; private set; }
      private NetworkStream _NetworkStream = null;
      private TcpClient _Client = null;
      private bool _ExitLoop = true;
      private BlockingCollection<ProtoBufExample.Header> _Queue = new BlockingCollection<ProtoBufExample.Header>();
      private readonly PendingFeedbacks _PendingFeedbacks = new PendingFeedbacks();

      public NetworkClient(string xIpAddress, int xPort, string xThreadName) {
         Port = xPort;
         IpAddress = xIpAddress;
         ThreadName = xThreadName;
      } //

      public void Connect() {
         if (!_ExitLoop) return; // running already
         _ExitLoop = false;

         _Client = new TcpClient();
         _Client.Connect(IpAddress, Port);
         _NetworkStream = _Client.GetStream();

         Thread lLoopWrite = new Thread(new ThreadStart(LoopWrite));
         lLoopWrite.IsBackground = true;
         lLoopWrite.Name = ThreadName + "Write";
         lLoopWrite.Start();

         Thread lLoopRead = new Thread(new ThreadStart(LoopRead));
         lLoopRead.IsBackground = true;
         lLoopRead.Name = ThreadName + "Read";
         lLoopRead.Start();
      } //

      public void Disconnect() {
         _ExitLoop = true;
         _Queue.Add(null);
         if (_Client != null) _Client.Close();
         //if (_NetworkStream != null) _NetworkStream.Close();
      } //

      public void Send(ProtoBufExample.Header xHeader) {
         if (xHeader == null) return;
         _PendingFeedbacks.Add(xHeader);
         _Queue.Add(xHeader);
      } //


      private void LoopWrite() {
         while (!_ExitLoop) {
            try {
               ProtoBufExample.Header lHeader = _Queue.Take();
               if (lHeader == null) break;

               // send header
               ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Header>(_NetworkStream, lHeader, ProtoBuf.PrefixStyle.Fixed32);

               // send data
               switch (lHeader.objectType) {
                  case ProtoBufExample.eType.eBook:
                     ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Book>(_NetworkStream, (ProtoBufExample.Book)lHeader.data, ProtoBuf.PrefixStyle.Fixed32);
                     break;
                  case ProtoBufExample.eType.eFable:
                     ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Fable>(_NetworkStream, (ProtoBufExample.Fable)lHeader.data, ProtoBuf.PrefixStyle.Fixed32);
                     break;
                  default:
                     break;
               }
            }
            catch (System.IO.IOException) {
               if (_ExitLoop) Console.WriteLine("user requested client shutdown.");
               else Console.WriteLine("disconnected");
            }
            catch (Exception ex) { Console.WriteLine(ex.Message); }
         }
         _ExitLoop = true;
         Console.WriteLine("client: writer is shutting down");
      } //


      private void LoopRead() {
         while (!_ExitLoop) {
            try {
               ProtoBufExample.Header lHeader = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Header>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
               if (lHeader == null) break;
               if (lHeader.objectType == ProtoBufExample.eType.eError) {
                  ProtoBufExample.ErrorMessage lErrorMessage = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.ErrorMessage>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
                  lHeader.data = lErrorMessage;
               }
               _PendingFeedbacks.Remove(lHeader);
            }
            catch (System.IO.IOException) {
               if (_ExitLoop) Console.WriteLine("user requested client shutdown");
               else Console.WriteLine("disconnected");
            }
            catch (Exception ex) { Console.WriteLine(ex.Message); }
         }
         Console.WriteLine("client: reader is shutting down");
      } //

   } // class

} // namespace

example output:
waiting for a client
new client connecting: 127.0.0.1:65432
waiting for a client
Book event was raised

received a book:
by Aesop, edition 13 Mar 1975, pages 203, price 1.99, isEbook False
title The Fox & the Grapes, rating 0.733333333333333
title The Goose that Laid the Golden Eggs, rating 0.7125
title The Cat & the Mice, rating 0.133333333333333
title The Mischievous Dog, rating 0.37

received a fable:
title The Goose that Laid the Golden Eggs, rating 0.7125
error: The fable was rejected. It is far too short.
the message that was sent out was: eFable with serial id 2
please check the log files

client: writer is shutting down
server: listener is shutting down
user requested client shutdown
client: reader is shutting down

Protocol Buffers (part 2, advanced), Tcp Networking

There has been no post yesterday. As usual I am preparing the source code for you during my spare time. This is not easy in my current environment. I will have to slow down the post frequency as the source codes are getting longer and more complex.

So, what is on the agenda now? Today we are going to send serialized binary data across the network. I am using the localhost to have a running example. Simply amend the local IP-addresses “127.0.0.1” to whatever suits you.

I wrote a small class (“ProtoType”) to identify objects quickly during the serialization and deserialization process. It is abstract and forms our base class. The class “Book” and “Fable” are nearly unchanged from the previous posts. I just added an override “ToString” in “Fable” to have a nicer output. Both classes must inherit from our abstract base class called “ProtoType”, which avoids implementing an interface and impacting the code legibility more than needed.

public class ProtoBufExample {

    public enum eType { Unknown = 0, eBook = 1, eFable };
    public abstract class ProtoType {
        public readonly eType objectId;
        public readonly byte[] objectIdAsBytes;

        public ProtoType() {
            Type t = this.GetType();
            if (t == typeof(ProtoBufExample.Book)) objectId = eType.eBook; // to identify the object before deserialization
            else if (t == typeof(ProtoBufExample.Fable)) objectId = eType.eFable; // to identify the object before deserialization
            else throw new Exception("object type unknown");
            objectIdAsBytes = BitConverter.GetBytes((Int16)objectId);
        } // constructor
    } // class 

    [ProtoContract]
    public class Book : ProtoType {            
        [ProtoMember(1)]
        public string author;
        [ProtoMember(2)]
        public List<Fable> stories;
        [ProtoMember(3)]
        public DateTime edition;
        [ProtoMember(4)]
        public int pages;
        [ProtoMember(5)]
        public double price;
        [ProtoMember(6)]
        public bool isEbook;

        public override string ToString() {
            StringBuilder s = new StringBuilder();
            s.Append("by "); s.Append(author);
            s.Append(", edition "); s.Append(edition.ToString("dd MMM yyyy"));
            s.Append(", pages "); s.Append(pages);
            s.Append(", price "); s.Append(price);
            s.Append(", isEbook "); s.Append(isEbook);
            s.AppendLine();
            if (stories != null) foreach (Fable lFable in stories) {
                    s.Append("title "); s.Append(lFable.title);
                    s.Append(", rating "); s.Append(lFable.customerRatings.Average()); // Average() is an extension method of "using System.Linq;"
                    s.AppendLine();
                }

            return s.ToString();
        } //
    } // class

    [ProtoContract]
    public class Fable : ProtoType {
        [ProtoMember(1)]
        public string title;
        [ProtoMember(2)]
        public double[] customerRatings;

        public override string ToString() {
            return "title " + title + ", rating " + customerRatings.Average();
        } //
    } // class

    public static Book GetData() {
        return new Book {
            author = "Aesop",
            price = 1.99,
            isEbook = false,
            edition = new DateTime(1975, 03, 13),
            pages = 203,
            stories = new List<Fable>(new Fable[] {
                new Fable{ title = "The Fox & the Grapes", customerRatings = new double[]{ 0.7, 0.7, 0.8} },
                new Fable{ title = "The Goose that Laid the Golden Eggs", customerRatings = new double[]{ 0.6, 0.75, 0.5, 1.0} },
                new Fable{ title = "The Cat & the Mice", customerRatings = new double[]{ 0.1, 0.0, 0.3} },
                new Fable{ title = "The Mischievous Dog", customerRatings = new double[]{ 0.45, 0.5, 0.4, 0.0, 0.5} }
            })
        };
    } //
} // class

The server-side always needs to be started before the client. Otherwise the client would not receive an immediate response from the server. We set up a listener on localhost port 65432 (arbitrary choice) and wait for a client connection. I chose the TCP protocol. It is easy to use and very reliable. We do not need to care about transmission problems. The downside is slowness. I will cover that one in comming posts.
The deserializer on the server-side cannot be run right away. We need to determine the object type first. And here we make use of our “ProtoType” class. The first two bytes of the transmission tell us the type. This is quite important for the speed of the deserialization. You see that the ProtoBuf.Serializer.DeserializeWithLengthPrefix method is generic. The type casting is minimized that way.

public class NetworkListener {
                
    private bool _ExitLoop = true;
    private TcpListener _Listener;
    public delegate void dOnMessage(object xSender, ProtoBufExample.Book xBook);
    public event dOnMessage OnMessage;
    private NetworkStream _NetworkStream = null;

    public int Port { get; private set; }
    public string IpAddress { get; private set; }
    public string ThreadName { get; private set; }

    public NetworkListener(string xIpAddress, int xPort, string xThreadName) {
        Port = xPort;
        IpAddress = xIpAddress;
        ThreadName = xThreadName;
    } //

    public bool Connect() {
        if (!_ExitLoop) {
            Console.WriteLine("Listener running already");
            return false;
        }
        _ExitLoop = false;

        try {
            _Listener = new TcpListener(IPAddress.Parse(IpAddress), Port);
            _Listener.Start();

            Thread lThread = new Thread(new ThreadStart(Loop));
            lThread.IsBackground = true;
            lThread.Name = ThreadName;
            lThread.Start();

            return true;
        }
        catch (Exception ex) { Console.WriteLine(ex.Message); }
        return false;
    } //

    public void Disconnect() {
        _ExitLoop = true;
        _NetworkStream.WriteTimeout = 5; // immediate timeout
    } //

    private void Loop() {
        try {
            while (!_ExitLoop) {
                Console.WriteLine("Waiting for a client");
                using (TcpClient lClient = _Listener.AcceptTcpClient()) {
                    string lClientIpAddress = lClient.Client.LocalEndPoint.ToString();
                    Console.WriteLine("New client connecting: " + lClientIpAddress);
                    using (_NetworkStream = lClient.GetStream()) {

                        while (!_ExitLoop) {
                            try {
                                byte[] lHeader = new byte[2];    // to indentify the object
                                if (_NetworkStream.Read(lHeader, 0, 2) != 2) break;
                                int lObjectType = BitConverter.ToInt16(lHeader, 0);
                                ProtoBufExample.eType lType = (ProtoBufExample.eType)lObjectType;
                                switch (lType) {
                                    case ProtoBufExample.eType.Unknown:
                                        break;
                                    case ProtoBufExample.eType.eBook:
                                        ProtoBufExample.Book lBook = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Book>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
                                        Console.WriteLine(Environment.NewLine + "received a book: ");
                                        Console.WriteLine(lBook.ToString());

                                        // raise an event
                                        dOnMessage lEvent = OnMessage;
                                        if (lEvent == null) continue;
                                        lEvent(lClient, lBook);

                                        break;
                                    case ProtoBufExample.eType.eFable:
                                        ProtoBufExample.Fable lFable = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Fable>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
                                        Console.WriteLine(Environment.NewLine + "received a fable: ");
                                        Console.WriteLine(lFable.ToString());
                                        break;
                                    default:
                                        Console.WriteLine("Mayday, mayday, we are in big trouble.");
                                        break;
                                }
                            }
                            catch (System.IO.IOException) {
                                if (_ExitLoop) Console.WriteLine("user requested TcpClient shutdown");
                                else Console.WriteLine("disconnected");
                            }
                            catch (Exception ex) { Console.WriteLine(ex.Message); }
                        }
                        Console.WriteLine(Environment.NewLine + "server/listener: shutting down");
                    }
                }
            }
        }
        catch (Exception ex) {
            Console.WriteLine(ex.Message);
        }
        finally {
            _ExitLoop = true;
            if (_Listener != null) _Listener.Stop();
        }
    } // 

} // class

The client side is straight forward. There is not much processing besides choosing the correct generic type for the serialization process. I used a BlockingCollection to make the context switching easier to read. It is not the fastest solution, but it definitely makes passing objects into a thread loop easy. I am personally no fan of the concurrent collections. They are not as predictable as any custom solution. _Queue.Take(); blocks and waits for data to arrive. It is thread-safe and does not require any object locking.

public class NetworkClient {
    public int Port { get; private set; }
    public string IpAddress { get; private set; }
    public string ThreadName { get; private set; }
    private NetworkStream _NetworkStream = null;
    private bool _ExitLoop = true;
    private BlockingCollection<ProtoBufExample.ProtoType> _Queue = new BlockingCollection<ProtoBufExample.ProtoType>();

    public NetworkClient(string xIpAddress, int xPort, string xThreadName) {
        Port = xPort;
        IpAddress = xIpAddress;
        ThreadName = xThreadName;
    } //

    public void Connect() {
        if (!_ExitLoop) return; // running already
        _ExitLoop = false;

        Thread lThread = new Thread(new ThreadStart(Loop));
        lThread.IsBackground = true;
        lThread.Name = ThreadName;
        lThread.Start();
    } //

    public void Disconnect() {
        _ExitLoop = true;
        _Queue.Add(null);
        if (_NetworkStream != null) _NetworkStream.ReadTimeout = 100;
    } //

    public void Send(ProtoBufExample.ProtoType xObject) {
        if (xObject == null) return;
        _Queue.Add(xObject);
    } //

    private void Loop() {
        try {
            using (TcpClient lClient = new TcpClient()) {
                lClient.Connect(IpAddress, Port);
                using (_NetworkStream = lClient.GetStream()) {

                    while (!_ExitLoop) {
                        try {
                            ProtoBufExample.ProtoType lObject = _Queue.Take();
                            if (lObject == null) break;

                            switch (lObject.objectId) {
                                case ProtoBufExample.eType.eBook:
                                    _NetworkStream.Write(lObject.objectIdAsBytes, 0, 2);
                                    ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Book>(_NetworkStream, (ProtoBufExample.Book)lObject, ProtoBuf.PrefixStyle.Fixed32);
                                    break;
                                case ProtoBufExample.eType.eFable:
                                    _NetworkStream.Write(lObject.objectIdAsBytes, 0, 2);
                                    ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Fable>(_NetworkStream, (ProtoBufExample.Fable)lObject, ProtoBuf.PrefixStyle.Fixed32);
                                    break;
                                default:
                                    break;
                            }
                        }
                        catch (System.IO.IOException) {
                            if (_ExitLoop) Console.WriteLine("user requested TcpClient shutdown.");
                            else Console.WriteLine("disconnected");
                        }
                        catch (Exception ex) { Console.WriteLine(ex.Message); }
                    }
                    _ExitLoop = true;
                    Console.WriteLine(Environment.NewLine + "client: shutting down");
                }
            }
        }
        catch (Exception ex) { Console.WriteLine(ex.Message); }
    } // 

} // class

The main program is pretty neat. We get the “Book” data and send it across the localhost. Then we take a single story of the same book and send it again to see if the program deals with the different types properly. Et voilà, it does.
The framing of the serialized data is very important. If you do not frame it, then you cannot determine the type. There are weird examples on the internet. My quick google research showed that people generally do not implement that centerpiece, leaving beginners in the middle of nowhere.

public static class NetworkTest {

    public static void Test() {
        NetworkListener lServer = new NetworkListener("127.0.0.1", 65432, "Server");
        NetworkClient lClient = new NetworkClient("127.0.0.1", 65432, "Client");

        lServer.Connect();
        lServer.OnMessage += new NetworkListener.dOnMessage(OnBook);

        lClient.Connect();

        // send a book across the network
        ProtoBufExample.Book lBook = ProtoBufExample.GetData();
        lClient.Send(lBook);

        // send a fable across the network
        lClient.Send(lBook.stories[1]);

        System.Threading.Thread.Sleep(1000);

        lClient.Disconnect();
        lServer.Disconnect();

        Console.ReadLine();
    }

    static void OnBook(object xSender, ProtoBufExample.Book xBook) {
        Console.WriteLine("Book event was raised");
    } //
} //

example output:
Waiting for a client
New client connecting: 127.0.0.1:65432

received a book:
by Aesop, edition 13 Mar 1975, pages 203, price 1.99, isEbook False
title The Fox & the Grapes, rating 0.733333333333333
title The Goose that Laid the Golden Eggs, rating 0.7125
title The Cat & the Mice, rating 0.133333333333333
title The Mischievous Dog, rating 0.37

Book event was raised

received a fable:
title The Goose that Laid the Golden Eggs, rating 0.7125

client: shutting down

server/listener: shutting down

There is one tiny issue with the OnBook event. We receive data on a thread and block the thread as long as it takes to process the event. To avoid thread bottlenecks you should think about forwarding “Book” events to tasks and let them deal with whatever needs to be done asynchronously. I posted about tasks eg. here. ThreadPools and the Parallel class might also be interesting posts in this context.

Protocol Buffers (part 1, basics), follow-up to JSON & XML

I quickly introduced Protocol Buffers in my last post. It is an amazing tool that every programmer should know. The .Net version protobuf-net can be downloaded from here.

On its homepage Protocol Buffers is described with the following words:

Protocol buffers is the name of the binary serialization format used by Google for much of their data communications. It is designed to be:

small in size – efficient data storage (far smaller than xml)
cheap to process – both at the client and server
platform independent – portable between different programming architectures
extensible – to add new data to old messages

This is succinct. Protobuf-net also supports the WCF.

Add the protobuf-net.dll, which is located in the net30 installation directory, to your solution references.
We need some data structure to start with.

public class Book {
    public string author;
    public List<Fable> stories;
    public DateTime edition;
    public int pages;
    public double price;
    public bool isEbook;
} // class

public class Fable {
    public string title;
    public double[] customerRatings;
} // class

Protocol Buffers uses attributes to identify types to serialize. The ProtoMember attribute needs a positive integer. This can be painful, because you have to avoid overlapping when using inheritance. But integers have a distinct advantage as well. They are much faster than strings. As you are already aware, Protocol Buffers is about speed.

[ProtoContract]
public class Book {
    [ProtoMember(1)]
    public string author;
    [ProtoMember(2)]
    public List stories;
    [ProtoMember(3)]
    public DateTime edition;
    [ProtoMember(4)]
    public int pages;
    [ProtoMember(5)]
    public double price;
    [ProtoMember(6)]
    public bool isEbook;

    public override string ToString() {
        StringBuilder s = new StringBuilder();
        s.Append("by "); s.Append(author);
        s.Append(", edition "); s.Append(edition.ToString("dd MMM yyyy"));
        s.Append(", pages "); s.Append(pages);
        s.Append(", price "); s.Append(price);
        s.Append(", isEbook "); s.Append(isEbook);
        s.AppendLine();
        if (stories != null) foreach (Fable lFable in stories) {
                s.Append("title "); s.Append(lFable.title);
                s.Append(", rating "); s.Append(lFable.customerRatings.Average());
                s.AppendLine();
            }

        return s.ToString();
    } //
} // class

[ProtoContract]
public class Fable {
    [ProtoMember(1)]
    public string title;
    [ProtoMember(2)]
    public double[] customerRatings;
} // class

public static Book GetData() {
    return new Book {
        author = "Aesop",
        price = 1.99,
        isEbook = false,
        edition = new DateTime(1975, 03, 13),
        pages = 203,
        stories = new List<Fable>(new Fable[] {
            new Fable{ title = "The Fox & the Grapes", customerRatings = new double[]{ 0.7, 0.7, 0.8} },
            new Fable{ title = "The Goose that Laid the Golden Eggs", customerRatings = new double[]{ 0.6, 0.75, 0.5, 1.0} },
            new Fable{ title = "The Cat & the Mice", customerRatings = new double[]{ 0.1, 0.0, 0.3} },
            new Fable{ title = "The Mischievous Dog", customerRatings = new double[]{ 0.45, 0.5, 0.4, 0.0, 0.5} }
    })
    };
} //

Let’s serialize the data now.

public static void SerializeData() {
    MemoryStream lStream = new MemoryStream();
    BinaryWriter lWriter = new BinaryWriter(lStream); // no "using", because it would close the MemoryStream automatically
    Book lBook = GetData();
    ProtoBuf.Serializer.Serialize<Book>(lStream, lBook);
    lWriter.Flush();
    lStream.Position = 0;

    using (BinaryReader lReader = new BinaryReader(lStream)) {
        for (long i = 0, n = lStream.Length; i < n; i++) {
            byte b = lReader.ReadByte();
            Console.Write(string.Format("{0:X2} ", b));
            if ((i+1) % 20 == 0) Console.WriteLine();
        }
        Console.WriteLine();
        Console.WriteLine();
        Console.WriteLine("number of bytes: " + lStream.Length);
    }
    Console.ReadLine();
} //

example output:
0A 05 41 65 73 6F 70 12 31 0A 14 54 68 65 20 46 6F 78 20 26
20 74 68 65 20 47 72 61 70 65 73 11 66 66 66 66 66 66 E6 3F
11 66 66 66 66 66 66 E6 3F 11 9A 99 99 99 99 99 E9 3F 12 49
0A 23 54 68 65 20 47 6F 6F 73 65 20 74 68 61 74 20 4C 61 69
64 20 74 68 65 20 47 6F 6C 64 65 6E 20 45 67 67 73 11 33 33
33 33 33 33 E3 3F 11 00 00 00 00 00 00 E8 3F 11 00 00 00 00
00 00 E0 3F 11 00 00 00 00 00 00 F0 3F 12 2F 0A 12 54 68 65
20 43 61 74 20 26 20 74 68 65 20 4D 69 63 65 11 9A 99 99 99
99 99 B9 3F 11 00 00 00 00 00 00 00 00 11 33 33 33 33 33 33
D3 3F 12 42 0A 13 54 68 65 20 4D 69 73 63 68 69 65 76 6F 75
73 20 44 6F 67 11 CD CC CC CC CC CC DC 3F 11 00 00 00 00 00
00 E0 3F 11 9A 99 99 99 99 99 D9 3F 11 00 00 00 00 00 00 00
00 11 00 00 00 00 00 00 E0 3F 1A 03 08 D2 1D 20 CB 01 29 D7
A3 70 3D 0A D7 FF 3F

number of bytes: 267

And back again: deserialize data.

public static void ToAndFro() {
    using (MemoryStream lStream = new MemoryStream()) {
        BinaryWriter lWriter = new BinaryWriter(lStream);
        Book lBook = GetData();
        ProtoBuf.Serializer.Serialize<Book>(lStream, lBook);
        lWriter.Flush();
        lStream.Position = 0;

        Book lCopy = ProtoBuf.Serializer.Deserialize<Book>(lStream);
        Console.WriteLine(lCopy.ToString());
    }

    Console.ReadLine();
} //

example output:
by Aesop, edition 13 Mar 1975, pages 203, price 1.99, isEbook False
title The Fox & the Grapes, rating 0.733333333333333
title The Goose that Laid the Golden Eggs, rating 0.7125
title The Cat & the Mice, rating 0.133333333333333
title The Mischievous Dog, rating 0.37