Monthly Archives: January 2014
Protocol Buffers (part 3, advanced), Tcp Networking
Ok guys, hardcore. Unfortunately in a different way than I thought 😉
Once again it took a lot of time to prepare the next source code example. It extends the code of my last post.
I removed the ProtoType class, because inheritance is not a real strength of Protobuf-Net. It is possible, but the ease of code maintenance is a definite argument against it. The complexity increases slowly, we cannot afford code that is difficult to change.
In theory we could write a Serialize() method for each class, the result would be extreme performance. You could use the BitConverter class to convert and then merge values without causing too much overhead. And by using Assembler I guess you could even make it 10 times faster again.
I think that all modern protocols lose a lot of time due to the required reflection, which is mainly the mapping of methods and properties. And unlike Assembler the modern languages do not allow to eg. simply write an integer to a memory address and only read the first byte of it.
You can rotate bits or multiply and divide, what is much less efficient.
In C# you can use the Parallel class to boost your performance. Of course the processing time of each code piece has to be substantial, otherwise it takes longer to create tasks than to solve the problems. In slow networks it might even be worth considering packing data before it is sent.
I replaced the ProtoType class by the Header class, which tells the type of the succeeding data block and also adds a serial identifier. This identifier can be used to receive vital feedbacks. For instance the server tells the client if the data transmission was successful or not. It can also send calculation results or error messages.
The data stream has the following order: header, data, header, data, header, data … except for feedback headers; they do not need data blocks. The enum eType inside the Header class defines all possible data types: ErrorMessage, Feedback, Book or Fable.
As said before I did not use inheritance for the Header class. The code remains legible and there is no spaghetti code to achieve indirect multiple inheritance.
In my last post, the client was sending and the server was receiving data. Now the communication is bidirectional. The client uses two threads, one to send and one to receive. The sending method is still using the BlockingCollection construction and an endless loop on a separate thread.
The server can be connected to several clients simultaneously. To keep it simple I decided sending data without any context switching. This usually blocks threads during the IO operation. I have added tasks inside the event OnMessageBook to give an example on how to avoid this. Nevertheless the send method uses a lock on the client preventing simultaneous write actions on the socket. This is bad practice; you should not apply locks on objects that might be locked in other areas as well. This is out of your scope, you don’t know if the .Net framework or any other code uses a lock on the same object, which could cause undesired behavior. In the below example it would have been better to wrap the client object into another class and apply the lock on that outer shell object. But I guess this is ok in our shorter example. The code was long enough already.
public static void Send(TcpClient xClient, ProtoBufExample.Header xHeader) {
if (xHeader == null) return;
if (xClient == null) return;
lock (xClient) {
NetworkStream lNetworkStream = xClient.GetStream();
....
The above lock problem could be avoided with this:
public class ClientData {
public TcpClient Client { get; private set; }
private DateTime _ConnectedSince;
private string _UserName;
....
public static void Send(ClientData xClient, ProtoBufExample.Header xHeader) {
if (xHeader == null) return;
if (xClient == null) return;
lock (xClient) {
NetworkStream lNetworkStream = xClient.Client.GetStream();
....
And here is the entire source code:
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using ProtoBuf;
namespace DemoApp {
public class ProtoBufExample {
public enum eType : byte { eError = 0, eFeedback, eBook, eFable };
[ProtoContract]
public class Header {
[ProtoMember(1)] public eType objectType;
[ProtoMember(2)] public readonly int serialMessageId;
public object data;
private static int _HeaderSerialId = 0;
public Header(object xData, eType xObjectType, int xSerialMessageId = 0) {
data = xData;
serialMessageId = (xSerialMessageId == 0) ? Interlocked.Increment(ref _HeaderSerialId) : xSerialMessageId;
objectType = xObjectType; // we could use "if typeof(T) ...", but it would be slower, harder to maintain and less legible
} // constructor
// parameterless constructor needed for Protobuf-net
public Header() {
} // constructor
} // class
[ProtoContract]
public class ErrorMessage {
[ProtoMember(1)]
public string Text;
} // class
[ProtoContract]
public class Book {
[ProtoMember(1)] public string author;
[ProtoMember(2, DataFormat = DataFormat.Group)] public List<Fable> stories;
[ProtoMember(3)] public DateTime edition;
[ProtoMember(4)] public int pages;
[ProtoMember(5)] public double price;
[ProtoMember(6)] public bool isEbook;
public override string ToString() {
StringBuilder s = new StringBuilder();
s.Append("by "); s.Append(author);
s.Append(", edition "); s.Append(edition.ToString("dd MMM yyyy"));
s.Append(", pages "); s.Append(pages);
s.Append(", price "); s.Append(price);
s.Append(", isEbook "); s.Append(isEbook);
s.AppendLine();
if (stories != null) foreach (Fable lFable in stories) {
s.Append("title "); s.Append(lFable.title);
s.Append(", rating "); s.Append(lFable.customerRatings.Average()); // Average() is an extension method of "using System.Linq;"
s.AppendLine();
}
return s.ToString();
} //
} // class
[ProtoContract]
public class Fable {
[ProtoMember(1)] public string title;
[ProtoMember(2, DataFormat = DataFormat.Group)]
public double[] customerRatings;
public override string ToString() {
return "title " + title + ", rating " + customerRatings.Average();
} //
} // class
public static Book GetData() {
return new Book {
author = "Aesop",
price = 1.99,
isEbook = false,
edition = new DateTime(1975, 03, 13),
pages = 203,
stories = new List<Fable>(new Fable[] {
new Fable{ title = "The Fox & the Grapes", customerRatings = new double[]{ 0.7, 0.7, 0.8} },
new Fable{ title = "The Goose that Laid the Golden Eggs", customerRatings = new double[]{ 0.6, 0.75, 0.5, 1.0} },
new Fable{ title = "The Cat & the Mice", customerRatings = new double[]{ 0.1, 0.0, 0.3} },
new Fable{ title = "The Mischievous Dog", customerRatings = new double[]{ 0.45, 0.5, 0.4, 0.0, 0.5} }
})
};
} //
} // class
public class PendingFeedbacks {
private readonly ConcurrentDictionary<int, ProtoBufExample.Header> _Messages = new ConcurrentDictionary<int, ProtoBufExample.Header>();
public int Count { get { return _Messages.Count; } }
public void Add(ProtoBufExample.Header xHeader) {
if (xHeader == null) throw new Exception("cannot add a null header");
if (!_Messages.TryAdd(xHeader.serialMessageId, xHeader)) {
throw new Exception("there must be a programming error somewhere");
}
} //
public void Remove(ProtoBufExample.Header xHeader) {
ProtoBufExample.Header lHeader;
if (!_Messages.TryRemove(xHeader.serialMessageId, out lHeader)) {
throw new Exception("there must be a programming error somewhere");
}
switch (xHeader.objectType) {
case ProtoBufExample.eType.eError:
Console.WriteLine("error: " + ((ProtoBufExample.ErrorMessage)xHeader.data).Text);
Console.WriteLine("the message that was sent out was: " + lHeader.objectType + " with serial id " + lHeader.serialMessageId);
Console.WriteLine("please check the log files" + Environment.NewLine);
break;
case ProtoBufExample.eType.eFeedback:
// all ok !
break;
default:
Console.WriteLine("warning: This message type was not expected.");
break;
}
} //
} // class
public static class NetworkTest {
public static void Test() {
NetworkListener lServer = new NetworkListener("127.0.0.1", 65432, "Server");
NetworkClient lClient = new NetworkClient("127.0.0.1", 65432, "Client");
lServer.Connect();
lServer.OnMessageBook += new NetworkListener.dOnMessageBook(OnMessageBook);
lServer.OnMessageFable += new NetworkListener.dOnMessageFable(OnMessageFable);
lClient.Connect();
ProtoBufExample.Header lHeader;
// send a book across the network
ProtoBufExample.Book lBook = ProtoBufExample.GetData();
lHeader = new ProtoBufExample.Header(lBook, ProtoBufExample.eType.eBook);
lClient.Send(lHeader);
System.Threading.Thread.Sleep(1000); // remove this to see the asynchonous processing (the output will look terrible)
// send a fable across the network
lHeader = new ProtoBufExample.Header(lBook.stories[1], ProtoBufExample.eType.eFable);
lClient.Send(lHeader);
System.Threading.Thread.Sleep(1000);
lClient.Disconnect();
lServer.Disconnect();
Console.ReadLine();
}
// demo: synchronous processing
static void OnMessageFable(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Fable xFable) {
Console.WriteLine(Environment.NewLine + "received a fable: ");
Console.WriteLine(xFable.ToString());
// demo: we tell the server that something went wrong
ProtoBufExample.ErrorMessage lErrorMessage = new ProtoBufExample.ErrorMessage() { Text = "The fable was rejected. It is far too short." };
ProtoBufExample.Header lErrorHeader = new ProtoBufExample.Header(lErrorMessage, ProtoBufExample.eType.eError, xHeader.serialMessageId);
NetworkListener.Send(xSender, lErrorHeader);
} //
// demo: asynchronous processing
static void OnMessageBook(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Book xBook) {
Task.Factory.StartNew(() => {
Console.WriteLine(Environment.NewLine + "received a book: ");
Console.WriteLine(xBook.ToString());
// send a feedback without any body to signal all was ok
ProtoBufExample.Header lFeedback = new ProtoBufExample.Header(null, ProtoBufExample.eType.eFeedback, xHeader.serialMessageId);
NetworkListener.Send(xSender, lFeedback);
return;
});
Console.WriteLine("Book event was raised");
} //
} // class
public class NetworkListener {
private bool _ExitLoop = true;
private TcpListener _Listener;
public delegate void dOnMessageBook(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Book xBook);
public event dOnMessageBook OnMessageBook;
public delegate void dOnMessageFable(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Fable xFable);
public event dOnMessageFable OnMessageFable;
private List<TcpClient> _Clients = new List<TcpClient>();
public int Port { get; private set; }
public string IpAddress { get; private set; }
public string ThreadName { get; private set; }
public NetworkListener(string xIpAddress, int xPort, string xThreadName) {
Port = xPort;
IpAddress = xIpAddress;
ThreadName = xThreadName;
} //
public bool Connect() {
if (!_ExitLoop) {
Console.WriteLine("Listener running already");
return false;
}
_ExitLoop = false;
try {
_Listener = new TcpListener(IPAddress.Parse(IpAddress), Port);
_Listener.Start();
Thread lThread = new Thread(new ThreadStart(LoopWaitingForClientsToConnect));
lThread.IsBackground = true;
lThread.Name = ThreadName + "WaitingForClients";
lThread.Start();
return true;
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
return false;
} //
public void Disconnect() {
_ExitLoop = true;
lock (_Clients) {
foreach (TcpClient lClient in _Clients) lClient.Close();
_Clients.Clear();
}
} //
private void LoopWaitingForClientsToConnect() {
try {
while (!_ExitLoop) {
Console.WriteLine("waiting for a client");
TcpClient lClient = _Listener.AcceptTcpClient();
string lClientIpAddress = lClient.Client.LocalEndPoint.ToString();
Console.WriteLine("new client connecting: " + lClientIpAddress);
if (_ExitLoop) break;
lock (_Clients) _Clients.Add(lClient);
Thread lThread = new Thread(new ParameterizedThreadStart(LoopRead));
lThread.IsBackground = true;
lThread.Name = ThreadName + "CommunicatingWithClient";
lThread.Start(lClient);
}
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
finally {
_ExitLoop = true;
if (_Listener != null) _Listener.Stop();
}
} //
private void LoopRead(object xClient) {
TcpClient lClient = xClient as TcpClient;
NetworkStream lNetworkStream = lClient.GetStream();
while (!_ExitLoop) {
try {
ProtoBufExample.Header lHeader = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Header>(lNetworkStream, ProtoBuf.PrefixStyle.Fixed32);
if (lHeader == null) break; // happens during shutdown process
switch (lHeader.objectType) {
case ProtoBufExample.eType.eBook:
ProtoBufExample.Book lBook = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Book>(lNetworkStream, ProtoBuf.PrefixStyle.Fixed32);
if (lBook == null) break;
lHeader.data = lBook; // not necessary, but nicer
dOnMessageBook lEventBook = OnMessageBook;
if (lEventBook == null) continue;
lEventBook(lClient, lHeader, lBook);
break;
case ProtoBufExample.eType.eFable:
ProtoBufExample.Fable lFable = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Fable>(lNetworkStream, ProtoBuf.PrefixStyle.Fixed32);
if (lFable == null) break;
lHeader.data = lFable; // not necessary, but nicer
dOnMessageFable lEventFable = OnMessageFable;
if (lEventFable == null) continue;
lEventFable(lClient, lHeader, lFable);
break;
default:
Console.WriteLine("Mayday, mayday, we are in big trouble.");
break;
}
}
catch (System.IO.IOException) {
if (_ExitLoop) Console.WriteLine("user requested client shutdown");
else Console.WriteLine("disconnected");
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
Console.WriteLine("server: listener is shutting down");
} //
public static void Send(TcpClient xClient, ProtoBufExample.Header xHeader) {
if (xHeader == null) return;
if (xClient == null) return;
lock (xClient) {
try {
NetworkStream lNetworkStream = xClient.GetStream();
// send header (most likely a simple feedback)
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Header>(lNetworkStream, xHeader, ProtoBuf.PrefixStyle.Fixed32);
// send errors
if (xHeader.objectType != ProtoBufExample.eType.eError) return;
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.ErrorMessage>(lNetworkStream, (ProtoBufExample.ErrorMessage)xHeader.data, ProtoBuf.PrefixStyle.Fixed32);
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
} //
} // class
public class NetworkClient {
public int Port { get; private set; }
public string IpAddress { get; private set; }
public string ThreadName { get; private set; }
private NetworkStream _NetworkStream = null;
private TcpClient _Client = null;
private bool _ExitLoop = true;
private BlockingCollection<ProtoBufExample.Header> _Queue = new BlockingCollection<ProtoBufExample.Header>();
private readonly PendingFeedbacks _PendingFeedbacks = new PendingFeedbacks();
public NetworkClient(string xIpAddress, int xPort, string xThreadName) {
Port = xPort;
IpAddress = xIpAddress;
ThreadName = xThreadName;
} //
public void Connect() {
if (!_ExitLoop) return; // running already
_ExitLoop = false;
_Client = new TcpClient();
_Client.Connect(IpAddress, Port);
_NetworkStream = _Client.GetStream();
Thread lLoopWrite = new Thread(new ThreadStart(LoopWrite));
lLoopWrite.IsBackground = true;
lLoopWrite.Name = ThreadName + "Write";
lLoopWrite.Start();
Thread lLoopRead = new Thread(new ThreadStart(LoopRead));
lLoopRead.IsBackground = true;
lLoopRead.Name = ThreadName + "Read";
lLoopRead.Start();
} //
public void Disconnect() {
_ExitLoop = true;
_Queue.Add(null);
if (_Client != null) _Client.Close();
//if (_NetworkStream != null) _NetworkStream.Close();
} //
public void Send(ProtoBufExample.Header xHeader) {
if (xHeader == null) return;
_PendingFeedbacks.Add(xHeader);
_Queue.Add(xHeader);
} //
private void LoopWrite() {
while (!_ExitLoop) {
try {
ProtoBufExample.Header lHeader = _Queue.Take();
if (lHeader == null) break;
// send header
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Header>(_NetworkStream, lHeader, ProtoBuf.PrefixStyle.Fixed32);
// send data
switch (lHeader.objectType) {
case ProtoBufExample.eType.eBook:
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Book>(_NetworkStream, (ProtoBufExample.Book)lHeader.data, ProtoBuf.PrefixStyle.Fixed32);
break;
case ProtoBufExample.eType.eFable:
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Fable>(_NetworkStream, (ProtoBufExample.Fable)lHeader.data, ProtoBuf.PrefixStyle.Fixed32);
break;
default:
break;
}
}
catch (System.IO.IOException) {
if (_ExitLoop) Console.WriteLine("user requested client shutdown.");
else Console.WriteLine("disconnected");
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
_ExitLoop = true;
Console.WriteLine("client: writer is shutting down");
} //
private void LoopRead() {
while (!_ExitLoop) {
try {
ProtoBufExample.Header lHeader = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Header>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
if (lHeader == null) break;
if (lHeader.objectType == ProtoBufExample.eType.eError) {
ProtoBufExample.ErrorMessage lErrorMessage = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.ErrorMessage>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
lHeader.data = lErrorMessage;
}
_PendingFeedbacks.Remove(lHeader);
}
catch (System.IO.IOException) {
if (_ExitLoop) Console.WriteLine("user requested client shutdown");
else Console.WriteLine("disconnected");
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
Console.WriteLine("client: reader is shutting down");
} //
} // class
} // namespace
example output:
waiting for a client
new client connecting: 127.0.0.1:65432
waiting for a client
Book event was raisedreceived a book:
by Aesop, edition 13 Mar 1975, pages 203, price 1.99, isEbook False
title The Fox & the Grapes, rating 0.733333333333333
title The Goose that Laid the Golden Eggs, rating 0.7125
title The Cat & the Mice, rating 0.133333333333333
title The Mischievous Dog, rating 0.37received a fable:
title The Goose that Laid the Golden Eggs, rating 0.7125
error: The fable was rejected. It is far too short.
the message that was sent out was: eFable with serial id 2
please check the log filesclient: writer is shutting down
server: listener is shutting down
user requested client shutdown
client: reader is shutting down
Protocol Buffers (part 2, advanced), Tcp Networking
There has been no post yesterday. As usual I am preparing the source code for you during my spare time. This is not easy in my current environment. I will have to slow down the post frequency as the source codes are getting longer and more complex.
So, what is on the agenda now? Today we are going to send serialized binary data across the network. I am using the localhost to have a running example. Simply amend the local IP-addresses “127.0.0.1” to whatever suits you.
I wrote a small class (“ProtoType”) to identify objects quickly during the serialization and deserialization process. It is abstract and forms our base class. The class “Book” and “Fable” are nearly unchanged from the previous posts. I just added an override “ToString” in “Fable” to have a nicer output. Both classes must inherit from our abstract base class called “ProtoType”, which avoids implementing an interface and impacting the code legibility more than needed.
public class ProtoBufExample {
public enum eType { Unknown = 0, eBook = 1, eFable };
public abstract class ProtoType {
public readonly eType objectId;
public readonly byte[] objectIdAsBytes;
public ProtoType() {
Type t = this.GetType();
if (t == typeof(ProtoBufExample.Book)) objectId = eType.eBook; // to identify the object before deserialization
else if (t == typeof(ProtoBufExample.Fable)) objectId = eType.eFable; // to identify the object before deserialization
else throw new Exception("object type unknown");
objectIdAsBytes = BitConverter.GetBytes((Int16)objectId);
} // constructor
} // class
[ProtoContract]
public class Book : ProtoType {
[ProtoMember(1)]
public string author;
[ProtoMember(2)]
public List<Fable> stories;
[ProtoMember(3)]
public DateTime edition;
[ProtoMember(4)]
public int pages;
[ProtoMember(5)]
public double price;
[ProtoMember(6)]
public bool isEbook;
public override string ToString() {
StringBuilder s = new StringBuilder();
s.Append("by "); s.Append(author);
s.Append(", edition "); s.Append(edition.ToString("dd MMM yyyy"));
s.Append(", pages "); s.Append(pages);
s.Append(", price "); s.Append(price);
s.Append(", isEbook "); s.Append(isEbook);
s.AppendLine();
if (stories != null) foreach (Fable lFable in stories) {
s.Append("title "); s.Append(lFable.title);
s.Append(", rating "); s.Append(lFable.customerRatings.Average()); // Average() is an extension method of "using System.Linq;"
s.AppendLine();
}
return s.ToString();
} //
} // class
[ProtoContract]
public class Fable : ProtoType {
[ProtoMember(1)]
public string title;
[ProtoMember(2)]
public double[] customerRatings;
public override string ToString() {
return "title " + title + ", rating " + customerRatings.Average();
} //
} // class
public static Book GetData() {
return new Book {
author = "Aesop",
price = 1.99,
isEbook = false,
edition = new DateTime(1975, 03, 13),
pages = 203,
stories = new List<Fable>(new Fable[] {
new Fable{ title = "The Fox & the Grapes", customerRatings = new double[]{ 0.7, 0.7, 0.8} },
new Fable{ title = "The Goose that Laid the Golden Eggs", customerRatings = new double[]{ 0.6, 0.75, 0.5, 1.0} },
new Fable{ title = "The Cat & the Mice", customerRatings = new double[]{ 0.1, 0.0, 0.3} },
new Fable{ title = "The Mischievous Dog", customerRatings = new double[]{ 0.45, 0.5, 0.4, 0.0, 0.5} }
})
};
} //
} // class
The server-side always needs to be started before the client. Otherwise the client would not receive an immediate response from the server. We set up a listener on localhost port 65432 (arbitrary choice) and wait for a client connection. I chose the TCP protocol. It is easy to use and very reliable. We do not need to care about transmission problems. The downside is slowness. I will cover that one in comming posts.
The deserializer on the server-side cannot be run right away. We need to determine the object type first. And here we make use of our “ProtoType” class. The first two bytes of the transmission tell us the type. This is quite important for the speed of the deserialization. You see that the ProtoBuf.Serializer.DeserializeWithLengthPrefix method is generic. The type casting is minimized that way.
public class NetworkListener {
private bool _ExitLoop = true;
private TcpListener _Listener;
public delegate void dOnMessage(object xSender, ProtoBufExample.Book xBook);
public event dOnMessage OnMessage;
private NetworkStream _NetworkStream = null;
public int Port { get; private set; }
public string IpAddress { get; private set; }
public string ThreadName { get; private set; }
public NetworkListener(string xIpAddress, int xPort, string xThreadName) {
Port = xPort;
IpAddress = xIpAddress;
ThreadName = xThreadName;
} //
public bool Connect() {
if (!_ExitLoop) {
Console.WriteLine("Listener running already");
return false;
}
_ExitLoop = false;
try {
_Listener = new TcpListener(IPAddress.Parse(IpAddress), Port);
_Listener.Start();
Thread lThread = new Thread(new ThreadStart(Loop));
lThread.IsBackground = true;
lThread.Name = ThreadName;
lThread.Start();
return true;
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
return false;
} //
public void Disconnect() {
_ExitLoop = true;
_NetworkStream.WriteTimeout = 5; // immediate timeout
} //
private void Loop() {
try {
while (!_ExitLoop) {
Console.WriteLine("Waiting for a client");
using (TcpClient lClient = _Listener.AcceptTcpClient()) {
string lClientIpAddress = lClient.Client.LocalEndPoint.ToString();
Console.WriteLine("New client connecting: " + lClientIpAddress);
using (_NetworkStream = lClient.GetStream()) {
while (!_ExitLoop) {
try {
byte[] lHeader = new byte[2]; // to indentify the object
if (_NetworkStream.Read(lHeader, 0, 2) != 2) break;
int lObjectType = BitConverter.ToInt16(lHeader, 0);
ProtoBufExample.eType lType = (ProtoBufExample.eType)lObjectType;
switch (lType) {
case ProtoBufExample.eType.Unknown:
break;
case ProtoBufExample.eType.eBook:
ProtoBufExample.Book lBook = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Book>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
Console.WriteLine(Environment.NewLine + "received a book: ");
Console.WriteLine(lBook.ToString());
// raise an event
dOnMessage lEvent = OnMessage;
if (lEvent == null) continue;
lEvent(lClient, lBook);
break;
case ProtoBufExample.eType.eFable:
ProtoBufExample.Fable lFable = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Fable>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
Console.WriteLine(Environment.NewLine + "received a fable: ");
Console.WriteLine(lFable.ToString());
break;
default:
Console.WriteLine("Mayday, mayday, we are in big trouble.");
break;
}
}
catch (System.IO.IOException) {
if (_ExitLoop) Console.WriteLine("user requested TcpClient shutdown");
else Console.WriteLine("disconnected");
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
Console.WriteLine(Environment.NewLine + "server/listener: shutting down");
}
}
}
}
catch (Exception ex) {
Console.WriteLine(ex.Message);
}
finally {
_ExitLoop = true;
if (_Listener != null) _Listener.Stop();
}
} //
} // class
The client side is straight forward. There is not much processing besides choosing the correct generic type for the serialization process. I used a BlockingCollection to make the context switching easier to read. It is not the fastest solution, but it definitely makes passing objects into a thread loop easy. I am personally no fan of the concurrent collections. They are not as predictable as any custom solution. _Queue.Take(); blocks and waits for data to arrive. It is thread-safe and does not require any object locking.
public class NetworkClient {
public int Port { get; private set; }
public string IpAddress { get; private set; }
public string ThreadName { get; private set; }
private NetworkStream _NetworkStream = null;
private bool _ExitLoop = true;
private BlockingCollection<ProtoBufExample.ProtoType> _Queue = new BlockingCollection<ProtoBufExample.ProtoType>();
public NetworkClient(string xIpAddress, int xPort, string xThreadName) {
Port = xPort;
IpAddress = xIpAddress;
ThreadName = xThreadName;
} //
public void Connect() {
if (!_ExitLoop) return; // running already
_ExitLoop = false;
Thread lThread = new Thread(new ThreadStart(Loop));
lThread.IsBackground = true;
lThread.Name = ThreadName;
lThread.Start();
} //
public void Disconnect() {
_ExitLoop = true;
_Queue.Add(null);
if (_NetworkStream != null) _NetworkStream.ReadTimeout = 100;
} //
public void Send(ProtoBufExample.ProtoType xObject) {
if (xObject == null) return;
_Queue.Add(xObject);
} //
private void Loop() {
try {
using (TcpClient lClient = new TcpClient()) {
lClient.Connect(IpAddress, Port);
using (_NetworkStream = lClient.GetStream()) {
while (!_ExitLoop) {
try {
ProtoBufExample.ProtoType lObject = _Queue.Take();
if (lObject == null) break;
switch (lObject.objectId) {
case ProtoBufExample.eType.eBook:
_NetworkStream.Write(lObject.objectIdAsBytes, 0, 2);
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Book>(_NetworkStream, (ProtoBufExample.Book)lObject, ProtoBuf.PrefixStyle.Fixed32);
break;
case ProtoBufExample.eType.eFable:
_NetworkStream.Write(lObject.objectIdAsBytes, 0, 2);
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Fable>(_NetworkStream, (ProtoBufExample.Fable)lObject, ProtoBuf.PrefixStyle.Fixed32);
break;
default:
break;
}
}
catch (System.IO.IOException) {
if (_ExitLoop) Console.WriteLine("user requested TcpClient shutdown.");
else Console.WriteLine("disconnected");
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
_ExitLoop = true;
Console.WriteLine(Environment.NewLine + "client: shutting down");
}
}
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
} //
} // class
The main program is pretty neat. We get the “Book” data and send it across the localhost. Then we take a single story of the same book and send it again to see if the program deals with the different types properly. Et voilà , it does.
The framing of the serialized data is very important. If you do not frame it, then you cannot determine the type. There are weird examples on the internet. My quick google research showed that people generally do not implement that centerpiece, leaving beginners in the middle of nowhere.
public static class NetworkTest {
public static void Test() {
NetworkListener lServer = new NetworkListener("127.0.0.1", 65432, "Server");
NetworkClient lClient = new NetworkClient("127.0.0.1", 65432, "Client");
lServer.Connect();
lServer.OnMessage += new NetworkListener.dOnMessage(OnBook);
lClient.Connect();
// send a book across the network
ProtoBufExample.Book lBook = ProtoBufExample.GetData();
lClient.Send(lBook);
// send a fable across the network
lClient.Send(lBook.stories[1]);
System.Threading.Thread.Sleep(1000);
lClient.Disconnect();
lServer.Disconnect();
Console.ReadLine();
}
static void OnBook(object xSender, ProtoBufExample.Book xBook) {
Console.WriteLine("Book event was raised");
} //
} //
example output:
Waiting for a client
New client connecting: 127.0.0.1:65432received a book:
by Aesop, edition 13 Mar 1975, pages 203, price 1.99, isEbook False
title The Fox & the Grapes, rating 0.733333333333333
title The Goose that Laid the Golden Eggs, rating 0.7125
title The Cat & the Mice, rating 0.133333333333333
title The Mischievous Dog, rating 0.37Book event was raised
received a fable:
title The Goose that Laid the Golden Eggs, rating 0.7125client: shutting down
server/listener: shutting down
There is one tiny issue with the OnBook event. We receive data on a thread and block the thread as long as it takes to process the event. To avoid thread bottlenecks you should think about forwarding “Book” events to tasks and let them deal with whatever needs to be done asynchronously. I posted about tasks eg. here. ThreadPools and the Parallel class might also be interesting posts in this context.
Protocol Buffers (part 1, basics), follow-up to JSON & XML
I quickly introduced Protocol Buffers in my last post. It is an amazing tool that every programmer should know. The .Net version protobuf-net can be downloaded from here.
On its homepage Protocol Buffers is described with the following words:
Protocol buffers is the name of the binary serialization format used by Google for much of their data communications. It is designed to be:
small in size – efficient data storage (far smaller than xml)
cheap to process – both at the client and server
platform independent – portable between different programming architectures
extensible – to add new data to old messages
This is succinct. Protobuf-net also supports the WCF.
Add the protobuf-net.dll, which is located in the net30 installation directory, to your solution references.
We need some data structure to start with.
public class Book {
public string author;
public List<Fable> stories;
public DateTime edition;
public int pages;
public double price;
public bool isEbook;
} // class
public class Fable {
public string title;
public double[] customerRatings;
} // class
Protocol Buffers uses attributes to identify types to serialize. The ProtoMember attribute needs a positive integer. This can be painful, because you have to avoid overlapping when using inheritance. But integers have a distinct advantage as well. They are much faster than strings. As you are already aware, Protocol Buffers is about speed.
[ProtoContract]
public class Book {
[ProtoMember(1)]
public string author;
[ProtoMember(2)]
public List stories;
[ProtoMember(3)]
public DateTime edition;
[ProtoMember(4)]
public int pages;
[ProtoMember(5)]
public double price;
[ProtoMember(6)]
public bool isEbook;
public override string ToString() {
StringBuilder s = new StringBuilder();
s.Append("by "); s.Append(author);
s.Append(", edition "); s.Append(edition.ToString("dd MMM yyyy"));
s.Append(", pages "); s.Append(pages);
s.Append(", price "); s.Append(price);
s.Append(", isEbook "); s.Append(isEbook);
s.AppendLine();
if (stories != null) foreach (Fable lFable in stories) {
s.Append("title "); s.Append(lFable.title);
s.Append(", rating "); s.Append(lFable.customerRatings.Average());
s.AppendLine();
}
return s.ToString();
} //
} // class
[ProtoContract]
public class Fable {
[ProtoMember(1)]
public string title;
[ProtoMember(2)]
public double[] customerRatings;
} // class
public static Book GetData() {
return new Book {
author = "Aesop",
price = 1.99,
isEbook = false,
edition = new DateTime(1975, 03, 13),
pages = 203,
stories = new List<Fable>(new Fable[] {
new Fable{ title = "The Fox & the Grapes", customerRatings = new double[]{ 0.7, 0.7, 0.8} },
new Fable{ title = "The Goose that Laid the Golden Eggs", customerRatings = new double[]{ 0.6, 0.75, 0.5, 1.0} },
new Fable{ title = "The Cat & the Mice", customerRatings = new double[]{ 0.1, 0.0, 0.3} },
new Fable{ title = "The Mischievous Dog", customerRatings = new double[]{ 0.45, 0.5, 0.4, 0.0, 0.5} }
})
};
} //
Let’s serialize the data now.
public static void SerializeData() {
MemoryStream lStream = new MemoryStream();
BinaryWriter lWriter = new BinaryWriter(lStream); // no "using", because it would close the MemoryStream automatically
Book lBook = GetData();
ProtoBuf.Serializer.Serialize<Book>(lStream, lBook);
lWriter.Flush();
lStream.Position = 0;
using (BinaryReader lReader = new BinaryReader(lStream)) {
for (long i = 0, n = lStream.Length; i < n; i++) {
byte b = lReader.ReadByte();
Console.Write(string.Format("{0:X2} ", b));
if ((i+1) % 20 == 0) Console.WriteLine();
}
Console.WriteLine();
Console.WriteLine();
Console.WriteLine("number of bytes: " + lStream.Length);
}
Console.ReadLine();
} //
example output:
0A 05 41 65 73 6F 70 12 31 0A 14 54 68 65 20 46 6F 78 20 26
20 74 68 65 20 47 72 61 70 65 73 11 66 66 66 66 66 66 E6 3F
11 66 66 66 66 66 66 E6 3F 11 9A 99 99 99 99 99 E9 3F 12 49
0A 23 54 68 65 20 47 6F 6F 73 65 20 74 68 61 74 20 4C 61 69
64 20 74 68 65 20 47 6F 6C 64 65 6E 20 45 67 67 73 11 33 33
33 33 33 33 E3 3F 11 00 00 00 00 00 00 E8 3F 11 00 00 00 00
00 00 E0 3F 11 00 00 00 00 00 00 F0 3F 12 2F 0A 12 54 68 65
20 43 61 74 20 26 20 74 68 65 20 4D 69 63 65 11 9A 99 99 99
99 99 B9 3F 11 00 00 00 00 00 00 00 00 11 33 33 33 33 33 33
D3 3F 12 42 0A 13 54 68 65 20 4D 69 73 63 68 69 65 76 6F 75
73 20 44 6F 67 11 CD CC CC CC CC CC DC 3F 11 00 00 00 00 00
00 E0 3F 11 9A 99 99 99 99 99 D9 3F 11 00 00 00 00 00 00 00
00 11 00 00 00 00 00 00 E0 3F 1A 03 08 D2 1D 20 CB 01 29 D7
A3 70 3D 0A D7 FF 3Fnumber of bytes: 267
And back again: deserialize data.
public static void ToAndFro() {
using (MemoryStream lStream = new MemoryStream()) {
BinaryWriter lWriter = new BinaryWriter(lStream);
Book lBook = GetData();
ProtoBuf.Serializer.Serialize<Book>(lStream, lBook);
lWriter.Flush();
lStream.Position = 0;
Book lCopy = ProtoBuf.Serializer.Deserialize<Book>(lStream);
Console.WriteLine(lCopy.ToString());
}
Console.ReadLine();
} //
example output:
by Aesop, edition 13 Mar 1975, pages 203, price 1.99, isEbook False
title The Fox & the Grapes, rating 0.733333333333333
title The Goose that Laid the Golden Eggs, rating 0.7125
title The Cat & the Mice, rating 0.133333333333333
title The Mischievous Dog, rating 0.37
JSON (basics), follow-up to XML
XML is quite verbose. This leads to higher memory usage and slowness during the (de-)serialization process. There are alternatives in the market that also support the cross-platform approach. The most basic one, which is used among web APIs, is JSON. It is well-integrated into JavaScript.
The .Net framework supports JSON. Thus JSON is a good start to slowly delve into posts about networking. There are more advanced cross-platform solutions. Jackson, GSON and BSON are some of the really good ones. Besides this you can always use the good old .Net data serialization (or remoting). If you need speed rather than flexibility, then I would probably recommend to use your own binary protocol. Unfortunately custom binary protocols are error-prone, debugging can be hell and the control factor for IT support people is close to zero in case the protocol is corrupt. Does transmitting data take longer than encoding/packing data? That is the fundamental question. Protocol Buffers will most likely be the “binary number cruncher’s” best friend. It is a free (Apache License) cross plattforms product. There is also a specific version for the .Net environment. Protocol Buffers is not the fastest tool in the market, but all faster tools I know are (like kryo) for Java only. Protocol Buffers is supported by many programming languages. Its speed is pretty good.
As usual plenty of posts lie ahead of us.
Back to Json.
Here is a quick example from wiki:
This is a follow-up on XML. So we have to start with an XML example somehow 🙂
<person>
<firstName>John</firstName>
<lastName>Smith</lastName>
<age>25</age>
<address>
<streetAddress>21 2nd Street</streetAddress>
<city>New York</city>
<state>NY</state>
<postalCode>10021</postalCode>
</address>
<phoneNumbers>
<phoneNumber type="home">212 555-1234</phoneNumber>
<phoneNumber type="fax">646 555-4567</phoneNumber>
</phoneNumbers>
</person>
or
<person firstName="John" lastName="Smith" age="25">
<address streetAddress="21 2nd Street" city="New York" state="NY" postalCode="10021" />
<phoneNumbers>
<phoneNumber type="home" number="212 555-1234"/>
<phoneNumber type="fax" number="646 555-4567"/>
</phoneNumbers>
</person>
And JSON looks more like C#, doesn’t it?
Btw. there is a really nice JSON online viewer here.
{
"firstName": "John",
"age": 25,
"address": {
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021"
},
"phoneNumber": [
{
"type": "home",
"number": "212 555-1234"
},
{
"type": "fax",
"number": "646 555-4567"
}
]
}
First of all we need to make JavaScriptSerializer available. Go to your Solution Explorer and right-click your project, then select “Properties”. Change your “Target Framework” on the “Application” page to “.Net Framework 4” or higher. Do not select “.Net Framework 4 Client Profile“.
Add the reference “System.Web.Extensions” to your project. The namespace “System.Web.Script.Serialization” is available now.
(In case it is still in your AssemblyInfo.cs, remove [assembly: AllowPartiallyTrustedCallersAttribute])
Let’s now convert our “Walmart.xml” example to JSON. The example shows how to serialize objetcs. Nothing was changed in the class WalmartWorld (see earlier post).
public static void SaveObjectsJSON(WalmartWorld xWalmartWorld) {
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
string lFile = lDesktopPath + "Walmart.json";
JavaScriptSerializer lSerializer = new JavaScriptSerializer();
string lSerialized = lSerializer.Serialize(xWalmartWorld);
File.WriteAllText(lFile, lSerialized);
} //
Formatted example output:
{
"Food": [
{
"Attribute": null,
"Name": "Banana",
"Price": 1.99,
"Description": "Mexican delicious",
"SomethingElse": null
},
{
"Attribute": null,
"Name": "Rice",
"Price": 0.79,
"Description": "the best you can get",
"SomethingElse": null
},
{
"Attribute": "What a wonderful world.",
"Name": "Cornflakes",
"Price": 3.85,
"Description": "buy some milk",
"SomethingElse": null
},
{
"Attribute": null,
"Name": "Milk",
"Price": 1.43,
"Description": "from happy cows",
"SomethingElse": null
},
{
"Attribute": null,
"Name": "baked beans",
"Price": 1.35,
"Description": "very British",
"SomethingElse": null
}
],
"Electronic": [
{
"Attribute": null,
"Name": "Kindle fire",
"Price": 100,
"Description": "Amazon loves you",
"SomethingElse": "the perfect Xmas gift for your kids"
}
],
"Text": "Luis Armstrong had a fabulous voice."
}
public static void LoadObjectJSON() {
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
string lFile = lDesktopPath + "Walmart.json";
JavaScriptSerializer lSerializer = new JavaScriptSerializer();
string lSerialized = File.ReadAllText(lFile);
WalmartWorld lWalmartWolrd = lSerializer.Deserialize<WalmartWorld>(lSerialized);
foreach (WalmartItem lItem in lWalmartWolrd.Electronic) Console.WriteLine(lItem);
foreach (WalmartItem lItem in lWalmartWolrd.Food) Console.WriteLine(lItem);
Console.ReadLine();
} //
example output:
Kindle fire 100.00 Amazon loves you !!! => the perfect Xmas gift for your kids Banana 1.99 Mexican delicious Rice 0.79 the best you can get Cornflakes 3.85 buy some milk Milk 1.43 from happy cows baked beans 1.35 very British
Unfortunately we have the same mess as with XML. There are several classes dealing with JSON in the .Net framework.
There is another serializer in the System.Runtime.Serialization.Json namespace called DataContractJsonSerializer. It is intended for use with WCF client applications. It does make use of DataContractAttribute and DataMemberAttribute to identify the data to serialize. If you do not add any of these attributes, then data is ignored.
Add the “System.Runtime.Serialization” reference to your project.
Let’s extend our Walmart classes and add the required attributes. You can remove the XML attributes if you do not want to work with XML anymore. You can leave them in the code to keep the back door open for quick and easy XML conversion. The XML attributes do not affect the program, there are just some extra bytes.
BUT beware of a little typo. If you use DataContractSerializer instead of DataContractJsonSerializer, then you will accidentally create XML and not Json files. Such little problems in life can drive you nuts.
Further reading for DataContractAttribute and DataMemberAttribute.
[Serializable]
[DataContract(Name="WalmartWorld")]
[XmlRoot("Walmart", Namespace = "")]
public class WalmartWorld {
[XmlElement("food")]
[DataMember]
public List<WalmartItem> Food { get; set; }
[XmlElement("electronic")]
[DataMember]
public List<WalmartItem> Electronic { get; set; }
[XmlText]
[DataMember]
public string Text { get; set; }
} // class
[Serializable]
[DataContract(Name = "WalmartItem")]
[XmlRoot("Walmart", Namespace = "")]
public class WalmartItem {
[XmlAttribute("attr")]
[DataMember(Name = "Attribute")] // just to demonstrate that you could define a name
public string Attribute { get; set; }
[XmlElement("name")]
[DataMember]
public string Name { get; set; }
[XmlElement("price")]
[DataMember]
public double Price { get; set; }
[XmlElement("description")]
[DataMember]
public string Description { get; set; }
[XmlElement("somethingElse")]
[DataMember]
public string SomethingElse { get; set; }
public override string ToString() {
return Name.PadRight(12) +
Price.ToString("#,##0.00").PadLeft(8) + " " +
Description +
(string.IsNullOrEmpty(SomethingElse) ? string.Empty : (" !!! => " + SomethingElse));
} //
} // class
public static void SaveObjectsJSON2(WalmartWorld xWalmartWorld) {
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
string lFile = lDesktopPath + "Walmart2.json";
using (FileStream lFileStream = new FileStream(lFile, FileMode.OpenOrCreate)) {
DataContractJsonSerializer lSerializer = new DataContractJsonSerializer(typeof(WalmartWorld));
lSerializer.WriteObject(lFileStream, xWalmartWorld);
}
} //
public static WalmartWorld LoadObjectJSON2() {
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
string lFile = lDesktopPath + "Walmart.json";
DataContractJsonSerializer lSerializer = new DataContractJsonSerializer(typeof(WalmartWorld));
using (FileStream lFileStream = new FileStream(lFile, FileMode.Open)) {
WalmartWorld lWalmartWolrd = lSerializer.ReadObject(lFileStream) as WalmartWorld;
foreach (WalmartItem lItem in lWalmartWolrd.Electronic) Console.WriteLine(lItem);
foreach (WalmartItem lItem in lWalmartWolrd.Food) Console.WriteLine(lItem);
return lWalmartWolrd;
}
} //
You could define your custom Name for each DataMember attribute. The JSON file would then address this name instead of the property name.
[DataMember(Name = "UseWhateverYouLike")]
public string Attribute { get; set; }
XML (part 4, advanced), XSD verification
Data integrity is a big issue in XML. You can generate XML Schemas to define your data structure and automate data integrity checks. As mentioned before, I am not going to explain the XSD structure itself. My purpose here is to explain the C# part.
Follow the link for a brilliant XSD introduction.
We are still using the embedded “Walmart.xml” file from the previous posts (1, 2, 3).
The easiest way to create an XML Schema is to use Visual Studio 2013. Open the XML file “Walmart.xml”. Then select “Create Schema” in the Window-Menu “XML” (between TEAM and TOOLS). The resulting XSD file looks like this:
<?xml version="1.0" encoding="utf-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="Walmart">
<xs:complexType>
<xs:sequence>
<xs:choice maxOccurs="unbounded">
<xs:element maxOccurs="unbounded" name="food">
<xs:complexType>
<xs:sequence>
<xs:element name="name" type="xs:string" />
<xs:element name="price" type="xs:decimal" />
<xs:element name="description" type="xs:string" />
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="electronic">
<xs:complexType>
<xs:sequence>
<xs:element name="name" type="xs:string" />
<xs:element name="price" type="xs:unsignedByte" />
<xs:element name="description" type="xs:string" />
<xs:element name="somethingElse" type="xs:string" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:choice>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
You can play with this file and modify it later on. Btw. one line should be amended from
<xs:element name="price" type="xs:unsignedByte" />
to
<xs:element name="price" type="xs:decimal" />
But for now let’s stick to the automatically generated XSD file as it is. Add the file to your Solution Explorer right next to “Walmart.xml” and set the property “Build Action” of “Walmart.xsd” to “Embedded Resource”.
private static void ValidationEvents(object sender, ValidationEventArgs e) {
switch (e.Severity) {
case XmlSeverityType.Error:
Console.WriteLine("Error: " + e.Message);
break;
case XmlSeverityType.Warning:
Console.WriteLine("Warning: " + e.Message);
break;
}
} //
public static void ValidateXML() {
Assembly lAssembly = Assembly.GetExecutingAssembly();
using (Stream lStreamXml = lAssembly.GetManifestResourceStream("DemoApp.XmlFiles.Walmart.xml")) {
using (Stream lStreamXsd = lAssembly.GetManifestResourceStream("DemoApp.XmlFiles.Walmart.xsd")) {
XmlDocument lXmlDocument = new XmlDocument();
XmlSchema x = XmlSchema.Read(lStreamXsd, new ValidationEventHandler(ValidationEvents));
lXmlDocument.Schemas.Add(x);
lXmlDocument.Load(lStreamXml);
lXmlDocument.Validate(ValidationEvents);
}
}
Console.WriteLine("press enter to exit program");
Console.ReadLine();
} //
Run ValidateXML(). There should be no error message. Now change the XSD or XML file and see what error messages you get.
XML (part 2, basics), embedded, XDocument, XmlDocument, XmlTextReader, XPathNavigator
We covered basic file IO for XML files yesterday. This is very useful for settings files. There are some XML files though that you want to hide from the user. I am not talking about encryption. Today we are going to embed the XML file the project itself.
We are still using yesterday’s “Walmart.xml” example. Create a folder “XmlFiles” in your Solution Explorer and drag the file into it.
Now right-click the file “Walmart.xml” and select “Properties”. Change the “Build Action” to “Embedded Resource”.
Use the following code to load the resource as an XDocument:
// since .Net 3.5
// namespace System.Xml.Linq
// xPath: namespace.folder.file.ext => here: "DemoApp.XmlFiles.Walmart.xml"
public static XDocument getAssemblyXDocument(string xPath) {
Assembly lAssembly = Assembly.GetExecutingAssembly();
using (Stream lStream = lAssembly.GetManifestResourceStream(xPath)) {
XDocument lXDocument = XDocument.Load(lStream);
return lXDocument;
}
} //
You can also load the document as an XmlDocument.
// since .Net 1.1
// namespace System.Xml
// xPath: namespace.folder.file.ext => here: "DemoApp.XmlFiles.Walmart.xml"
public static XmlDocument getAssemblyXml(string xPath) {
Assembly lAssembly = Assembly.GetExecutingAssembly();
using (Stream lStream = lAssembly.GetManifestResourceStream(xPath)) {
XmlDocument lXmlDocument = new XmlDocument();
lXmlDocument.Load(lStream);
return lXmlDocument;
}
} //
public static void LoadXmlExample() {
string lPath = "DemoApp.XmlFiles.Walmart.xml";
XmlDocument lXml = getAssemblyXml(lPath);
XDocument lXDoc = getAssemblyXDocument(lPath);
} //
Using an XmlDocument is similar to XDocument. XmlDocument does not support LINQ, which is a big disadvantage.
XmlDocument reads the entire document and initializes a tree structure, which needs a lot of memory. Processing bigger Xml files can cause issues. The tree can be accessed either by browsing through the nodes or by using XPath queries.
You can browse through documents forward/backward and carry out modifications.
XDocument (LINQ to XML) is easy to code and understand. Again the entire document is loaded into memory. To read Xml files we can use the XDocument or XElement class. Both classes support Load() methods. XElement represents an XML fragment while XDocument represents an entire XML document.
XmlTextReader is very fast and memory efficient. Unlike XmlDocument it can only read in forward direction. Searching for something specific is awkward. As the name says, there is no method to write or to modify data. And data validation is also left behind.
XPathNavigator is more complex and slower than XmlReader. XPathNavigator has to read enough data to be able to execute XPath queries. XmlDocument internally uses XPathNavigator. Using XPathNavigator directly is faster than using it via XmlDocument.
Using XmlTextReader:
public static void UseTextReader() {
string lPath = "DemoApp.XmlFiles.Walmart.xml";
Assembly lAssembly = Assembly.GetExecutingAssembly();
using (Stream lStream = lAssembly.GetManifestResourceStream(lPath)) {
using (XmlTextReader lXmlReader = new XmlTextReader(lStream)) {
while (lXmlReader.Read()) {
if (!lXmlReader.IsStartElement()) continue;
switch (lXmlReader.Name) {
case "name":
Console.WriteLine("name: " + lXmlReader.ReadString());
break;
case "price":
Console.WriteLine("price: " + lXmlReader.ReadString());
break;
case "description":
Console.WriteLine("description: " + lXmlReader.ReadString());
break;
case "somethingElse":
Console.WriteLine("somethingElse: " + lXmlReader.ReadString());
break;
default:
Console.WriteLine(Environment.NewLine + lXmlReader.Name);
break;
}
}
}
}
Console.ReadLine();
} //
example output:
Walmart
food
name: Banana
price: 1.99
description: Mexican deliciousfood
name: Rice
price: 0.79
description: the best you can getfood
name: Cornflakes
price: 3.85
description: buy some milkfood
name: Milk
price: 1.43
description: from happy cowselectronic
name: Kindle fire
price: 100
description: Amazon loves you
somethingElse: the perfect Xmas gift for your kidsfood
name: baked beans
price: 1.35
description: very British
Using XPathNavigator:
public static void UseXPathNavigator() {
string lPath = "DemoApp.XmlFiles.Walmart.xml";
Assembly lAssembly = Assembly.GetExecutingAssembly();
using (Stream lStream = lAssembly.GetManifestResourceStream(lPath)) {
XPathDocument lXPathDoc = new XPathDocument(lStream);
XPathNavigator lXNavi = lXPathDoc.CreateNavigator();
lXNavi.MoveToRoot();
lXNavi.MoveToFirstChild();
do {
if (lXNavi.MoveToFirstAttribute()) {
Console.WriteLine(lXNavi.Name + "=" + lXNavi.Value);
lXNavi.MoveToParent();
}
if (!lXNavi.MoveToFirstChild()) continue;
do {
if (!lXNavi.MoveToFirstChild()) break;
do {
Console.WriteLine(lXNavi.Name + "=" + lXNavi.Value);
} while (lXNavi.MoveToNext());
Console.WriteLine();
lXNavi.MoveToParent();
} while (lXNavi.MoveToNext());
lXNavi.MoveToParent();
} while (lXNavi.MoveToNext());
Console.ReadLine();
}
} //
example output:
name=Banana
price=1.99
description=Mexican deliciousname=Rice
price=0.79
description=the best you can getname=Cornflakes
price=3.85
description=buy some milkname=Milk
price=1.43
description=from happy cowsname=Kindle fire
price=100
description=Amazon loves you
somethingElse=the perfect Xmas gift for your kidsname=baked beans
price=1.35
description=very British
Using XPathNavigator with query:
public static void UseXPathNavigator(string xQuery) {
string lPath = "DemoApp.XmlFiles.Walmart.xml";
Assembly lAssembly = Assembly.GetExecutingAssembly();
using (Stream lStream = lAssembly.GetManifestResourceStream(lPath)) {
XPathDocument lXPathDoc = new XPathDocument(lStream);
XPathNavigator lXNavi = lXPathDoc.CreateNavigator();
XPathNodeIterator lIterator = lXNavi.Select(xQuery);
if (lIterator.Count <= 0) {
Console.WriteLine("Nothing found!");
return;
}
while (lIterator.MoveNext()) {
Console.WriteLine(lIterator.Current.Value);
}
}
} //
public static void TestQueries() {
UseXPathNavigator(@"/Walmart/electronic");
Console.WriteLine();
UseXPathNavigator(@"/Walmart/food[name='Banana']");
Console.ReadLine();
} //
example output:
Kindle fire100Amazon loves youthe perfect Xmas gift for your kidsBanana1.99Mexican delicious
My opinion is: Make use of LINQ to XML. It is very comfortable and can save you a lot of time. If you have a lot of data, then you should consider a database rather than XML files.
We will discuss XML object serialization tomorrow. Let C# save and load objects the nice way.
XML (part 1, basics), file IO
The last days were tough. Writing posts requires a lot of time. Explaining some basic know-how will give me some recreation today.
I will concentrate on the C# parts. There will be no explanations about XML, which is well documented across the internet.
XML stands for eXtensible Markup Language. XML is a versatile and flexible data structure. It is easy to read, learn and understand. The downside is the amount of bytes it needs. You can use XML to transport and/or store data, but it quickly becomes quite inefficient when a lot of datasets are involved.
XML is a well established industry standard. So there is no way around it, you have to know at least something about it.
Here is an XML file that we will use in the loading example:
<?xml version="1.0" encoding="UTF-8"?>
<!-- XML EXAMPLE -->
<Walmart>
<food>
<name>Banana</name>
<price>1.99</price>
<description>Mexican delicious</description>
</food>
<food>
<name>Rice</name>
<price>0.79</price>
<description>the best you can get</description>
</food>
<food>
<name>Cornflakes</name>
<price>3.85</price>
<description>buy some milk</description>
</food>
<food>
<name>Milk</name>
<price>1.43</price>
<description>from happy cows</description>
</food>
<electronic>
<name>Kindle fire</name>
<price>100</price>
<description>Amazon loves you</description>
<somethingElse>the perfect Xmas gift for your kids</somethingElse>
</electronic>
<food>
<name>baked beans</name>
<price>1.35</price>
<description>very British</description>
</food>
</Walmart>
Paste the XML structure into a text editor and save it as an XML file (“Walmart.xml”) onto your desktop.
The following class will be used to store datasets in memory.
public class WmItem {
public readonly string name;
public readonly double price;
public readonly string description;
public WmItem(string xName, double xPrice, string xDescription) {
name = xName;
price = xPrice;
description = xDescription;
} // constructor
public override string ToString() {
return name.PadRight(12) + price.ToString("#,##0.00").PadLeft(8) + " " + description;
} //
} // class
Loading XML files is easy. Processing files takes a bit longer. The centerpiece generally is the parsing algorithm.
There is a field called “somethingElse” in the XML file. It does not cause any trouble. The code simply disregards it. I added it to demonstrate the “eXtensible” in “eXtensible Markup Language”.
public static void LoadXml() {
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
string lFile = lDesktopPath + "Walmart.xml";
XDocument lXDocument = XDocument.Load(lFile);
// food (using WmItem)
WmItem[] lFood = (from lData in lXDocument.Descendants("Walmart").Descendants("food")
select new WmItem(
lData.Element("name").Value,
double.Parse(lData.Element("price").Value),
lData.Element("description").Value)
).ToArray();
foreach (WmItem lItem in lFood) Console.WriteLine(lItem);
Console.WriteLine();
// electronic (quick and dirty, using var)
var lElectronic = from lData in lXDocument.Descendants("Walmart").Descendants("electronic")
select lData;
foreach (var lItem in lElectronic) {
Console.WriteLine(lItem);
Console.WriteLine();
Console.WriteLine(lItem.Element("name").Value);
Console.WriteLine(lItem.Element("price").Value);
Console.WriteLine(lItem.Element("description").Value);
}
Console.ReadLine();
} //
example output:
Banana 1.99 Mexican delicious
Rice 0.79 the best you can get
Cornflakes 3.85 buy some milk
Milk 1.43 from happy cows
baked beans 1.35 very British<electronic>
<name>Kindle fire</name>
<price>100</price>
<description>Amazon loves you</description>
<somethingElse>the perfect Xmas gift for your kids</somethingElse>
</electronic>Kindle fire
100
Amazon loves you
Saving XML is also straight forward. I added a comment and attributes for demonstration purposes. The program generates and saves the XML file “Genesis.xml” on your desktop.
public static void SaveXml() {
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
string lFile = lDesktopPath + "Genesis.xml";
WmItem[] lMetals = {
new WmItem("Lead", 1.0, "here we go"),
new WmItem("Silver", 2.0, "cutlery"),
new WmItem("Gold", 3.0, "wife's best friend"),
new WmItem("Platinum", 4.0, "posh")
};
XDocument lXDocument = new XDocument();
lXDocument.Declaration = new XDeclaration("1.0", "utf-8", "yes");
lXDocument.Add(new XComment("copyfight by Bastian M.K. Ohta"));
XElement lLME = new XElement("London_Metal_Exchange", new XAttribute("attribute1", "buy here"), new XAttribute("AreYouSure", "yes"));
lXDocument.Add(lLME);
foreach (WmItem lMetal in lMetals) {
XElement lGroup = new XElement("metal");
lGroup.Add(new XElement("name", lMetal.name));
lGroup.Add(new XElement("price", lMetal.price));
lGroup.Add(new XElement("description", lMetal.description));
lLME.Add(lGroup);
}
lXDocument.Save(lFile);
//Console.ReadLine();
} //
Reflection (part 6, professional), Emit
Reflection can discover information about objects at runtime and execute against those objects. The namespace System.Reflection.Emit also allows you to build assemblies and create types at runtime. Only a few programmers will come across Emit. You have to be a kind of Rambo sewing yourself while fighting … and definitely not Alice in Wonderland.
Emit generates IL code in memory. The source code for such is quite complex and difficult. We are nearly back to Assembler style. And if you remember well, crashes and difficult debugging were the order of the day. There are tools like the EmitHelper class out there to make Emit easier. I am going to show the fundamentals today, this does not cover tools.
Today’s word list:
Domain
Application domains aid security, separating applications from each other and each other’s data. A single process can run several application domains, with the same level of isolation that would exist in separate processes. Running multiple applications within a single process increases server scalability.
Faults in one application domain cannot affect other code running in another application domain.
Module
A module is a portable executable file, such as type.dll or application.exe, consisting of one or more classes and interfaces. There may be multiple namespaces contained in a single module, and a namespace may span multiple modules.
One or more modules deployed as a unit compose an assembly.
The hierarchy is: domain => assemblies => modules => classes => functions
Ildasm.exe
A disassembler for MSIL code.
The first step is to create an executable file from:
using System;
using System.Reflection;
namespace HelloWorld {
public class Program {
static void Main(string[] args) {
} //
public string Test() {
return string.Format("DateTime is {0:dd MMM yyyy HH:mm:ss}", DateTime.Now);
} //
public string Test2() {
return DateTime.UtcNow.ToString();
} //
public string Test3() {
return Assembly.GetExecutingAssembly().ToString();
} //
public void Test4() {
Console.WriteLine("hello world !");
} //
} // class
} // namespace
On the windows taskbar, click Start, click All Programs, click Visual Studio, click Visual Studio Tools, and then click Visual Studio Command Prompt.
-or-
If you have the Windows SDK installed on your computer: On the taskbar, click Start, click All Programs, click the folder for the Windows SDK, and then click Command Prompt (or CMD Shell).
Change to the right folder and enter “ildasm HelloWorld.exe /output:HelloWorld.il”.
In your folder you should see something like this:
Open the HelloWorld.il file in a text editor and delve into the following sections:
method Test()
.method public hidebysig instance string
Test() cil managed
{
// Code size 21 (0x15)
.maxstack 8
IL_0000: ldstr "DateTime is {0:dd MMM yyyy HH:mm:ss}"
IL_0005: call valuetype [mscorlib]System.DateTime [mscorlib]System.DateTime::get_Now()
IL_000a: box [mscorlib]System.DateTime
IL_000f: call string [mscorlib]System.String::Format(string,
object)
IL_0014: ret
} // end of method Program::Test
method Test3()
.method public hidebysig instance string
Test2() cil managed
{
// Code size 20 (0x14)
.maxstack 1
.locals init ([0] valuetype [mscorlib]System.DateTime CS$0$0000)
IL_0000: call valuetype [mscorlib]System.DateTime [mscorlib]System.DateTime::get_UtcNow()
IL_0005: stloc.0
IL_0006: ldloca.s CS$0$0000
IL_0008: constrained. [mscorlib]System.DateTime
IL_000e: callvirt instance string [mscorlib]System.Object::ToString()
IL_0013: ret
} // end of method Program::Test2
method Test3()
.method public hidebysig instance string
Test3() cil managed
{
// Code size 11 (0xb)
.maxstack 8
IL_0000: call class [mscorlib]System.Reflection.Assembly [mscorlib]System.Reflection.Assembly::GetExecutingAssembly()
IL_0005: callvirt instance string [mscorlib]System.Object::ToString()
IL_000a: ret
} // end of method Program::Test3
method Test4()
.method public hidebysig instance void
Test4() cil managed
{
// Code size 11 (0xb)
.maxstack 8
IL_0000: ldstr "hello world !"
IL_0005: call void [mscorlib]System.Console::WriteLine(string)
IL_000a: ret
} // end of method Program::Test4
These code sections give you a rough idea of what we are going to code in runtime.
Start a new console project. You have to edit the file “AssemblyInfo.cs”. Add the assembly attribute AllowPartiallyTrustedCallersAttribute. The program won’t run otherwise. We are on a low coding level and Microsoft obviously tries to protect code especially on that level.
AssemblyInfo.cs
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using System.Security;
// General Information about an assembly is controlled through the following
// set of attributes. Change these attribute values to modify the information
// associated with an assembly.
[assembly: AssemblyTitle("oink")]
[assembly: AssemblyDescription("")]
[assembly: AssemblyConfiguration("")]
[assembly: AssemblyCompany("")]
[assembly: AssemblyProduct("blablabla")]
[assembly: AssemblyCopyright("Copyright © 2014")]
[assembly: AssemblyTrademark("")]
[assembly: AssemblyCulture("")]
[assembly: AllowPartiallyTrustedCallersAttribute]
// Setting ComVisible to false makes the types in this assembly not visible
// to COM components. If you need to access a type in this assembly from
// COM, set the ComVisible attribute to true on that type.
[assembly: ComVisible(false)]
// The following GUID is for the ID of the typelib if this project is exposed to COM
[assembly: Guid("0bf893aa-f3da-49ef-a2dc-a63d8ffc9ead")]
// Version information for an assembly consists of the following four values:
//
// Major Version
// Minor Version
// Build Number
// Revision
//
// You can specify all the values or you can default the Build and Revision Numbers
// by using the '*' as shown below:
// [assembly: AssemblyVersion("1.0.*")]
[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyFileVersion("1.0.0.0")]
The Print() method is not big, nevertheless you find something unusual here. The keyword __arglist is not official. It has been implemented by Microsoft, but is not documented at all. It is also not well supported in C#. For instance you cannot use a foreach loop. __arglist takes any number of optional arguments like object[]. In our example it does make a big difference to use __arglist. We do not have to create an array and fill it in. This would be much more code in Emit. On the contrary the __arglist algorithm is quite short and comfortable. Follow the link for more __arglist documentation.
public static void Print(string xHeader, __arglist) {
ArgIterator lIterator = new ArgIterator(__arglist);
Console.WriteLine(xHeader);
while (lIterator.GetRemainingCount() > 0) {
TypedReference r = lIterator.GetNextArg();
object o = TypedReference.ToObject(r);
Console.Write(o);
}
Console.WriteLine();
Console.WriteLine();
} //
EmitCode() is the actual code generation part. Let’s call the remaining source code “administration” to simplify the situation.
There are two ways to get the MethodInfo for a property. One is used for DateTime.Now, the other one is used for DateTime.UtcNow .
They can be used equally. I used both ways for demonstration purposes only.
I tried to keep the example simple. You don’t have to study the MSIL to understand the code. Some information about MSIL OpCodes can be found here.
static void EmitCode(ILGenerator xILGenerator) {
MethodInfo lMethodInfo_Print = typeof(EmitDemo).GetMethod("Print");
MethodInfo lDateTime_Now = typeof(DateTime).GetProperty("Now").GetGetMethod();
MethodInfo lFormat = typeof(string).GetMethod("Format", new Type[] { typeof(string), typeof(object) });
xILGenerator.Emit(OpCodes.Ldstr, "DateTime is {0:dd MMM yyyy HH:mm:ss}");
xILGenerator.Emit(OpCodes.Call, lDateTime_Now);
xILGenerator.Emit(OpCodes.Box, typeof(DateTime));
xILGenerator.Emit(OpCodes.Call, lFormat);
xILGenerator.EmitCall(OpCodes.Call, lMethodInfo_Print, new Type[] { });
xILGenerator.Emit(OpCodes.Ldstr, "This equals UTC: ");
xILGenerator.Emit(OpCodes.Call, typeof(DateTime).GetMethod("get_UtcNow")); // compare this with lDateTime_Now (== same, just another demo approach)
xILGenerator.EmitCall(OpCodes.Call, lMethodInfo_Print, new Type[] { typeof(DateTime) });
xILGenerator.Emit(OpCodes.Ldstr, "The assembly is: ");
xILGenerator.Emit(OpCodes.Call, typeof(Assembly).GetMethod("GetExecutingAssembly"));
xILGenerator.EmitCall(OpCodes.Call, lMethodInfo_Print, new Type[] { typeof(Assembly) });
xILGenerator.Emit(OpCodes.Ldstr, "Console.WriteLine() is old school.");
xILGenerator.Emit(OpCodes.Call, typeof(Console).GetMethod("WriteLine", new Type[] { typeof(object) }));
xILGenerator.EmitWriteLine("EmitWriteLine() is for lazy programmers.");
xILGenerator.Emit(OpCodes.Ret);
} //
Test1_ViaFullAssembly() shows you how to properly initialize and use an assembly. Remember the hierarchy from above: domain => assemblies => modules => classes => functions
The method is build like a chain. Each statement uses the result of the previous one.
public static void Test1_ViaFullAssembly() {
AssemblyName lAssemblyName = new AssemblyName("MyAssembly");
AssemblyBuilder lAssemblyBuilder = AppDomain.CurrentDomain.DefineDynamicAssembly(lAssemblyName, AssemblyBuilderAccess.Run);
ModuleBuilder lModuleBuilder = lAssemblyBuilder.DefineDynamicModule("MyModule");
TypeBuilder lTypeBuilder = lModuleBuilder.DefineType("MyType");
MethodBuilder lMethodBuilder = lTypeBuilder.DefineMethod("DoSomething", MethodAttributes.Public | MethodAttributes.Static, CallingConventions.Standard, typeof(void), Type.EmptyTypes);
EmitCode(lMethodBuilder.GetILGenerator());
Type lType = lTypeBuilder.CreateType();
lType.GetMethod("DoSomething").Invoke(null, null);
Console.ReadLine();
} //
example output:
DateTime is 15 Jan 2014 23:17:30This equals UTC:
15/01/2014 23:17:30The assembly is:
MyAssembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=nullConsole.WriteLine() is old school.
EmitWriteLine() is for lazy programmers.
You can avoid building a full assembly. DynamicMethod simplifies the dynamic creation of methods. Of course you won’t have any class structure. The method is super-public, it reminds me of VBA modules.
public static void ViaDynamicMethod() {
DynamicMethod lDynamicMethod = new DynamicMethod("DoSomething", typeof(void), Type.EmptyTypes, typeof(object));
EmitCode(lDynamicMethod.GetILGenerator());
lDynamicMethod.Invoke(null, null);
Console.ReadLine();
} //
example output:
DateTime is 15 Jan 2014 23:17:58This equals UTC:
15/01/2014 23:17:58The assembly is:
mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089Console.WriteLine() is old school.
EmitWriteLine() is for lazy programmers.
Reflection (part 5, professional), CodeDOM (& lambda expression tree)
Today’s post is about code generation at runtime using CodeDOM, which is the logical step before runtime code generation using Emit. The .Net framework has been built in an amazing way. We can easily neglect some flaws. Compared to Java there are no flaws at all. Jealousy needs to be deserved, sympathy is for free. Somehow Java is for free, and this is what you get.
Make sure, you are aware of the following differences. There are Expression and Statement. Please follow the links if you feel uncertain about the definitions.
Expression
An expression is a sequence of one or more operands and zero or more operators that can be evaluated to a single value, object, method, or namespace.
Statement
The actions that a program takes are expressed in statements. Common actions include declaring variables, assigning values, calling methods, looping through collections, and branching to one or another block of code, depending on a given condition. The order in which statements are executed in a program is called the flow of control or flow of execution.
A statement can consist of a single line of code that ends in a semicolon, or a series of single-line statements in a block. A statement block is enclosed in {} brackets and can contain nested blocks.
Succinct:
An Expression is a kind of mathematical or logical expression.
A Statement is a single code instruction or block of code to execute.
The first example code generates source code for C# and VB and saves the generated files on your desktop. Of course it has to be the classical “hello world” program.
CodeDOM stands for Code Document Object Model. It is located in the System.CodeDom namespace and provides methods to create code at runtime.
// what you need for the code example: using System; using System.CodeDom; using System.CodeDom.Compiler; using System.IO; using Microsoft.CSharp; using Microsoft.VisualBasic;
First of all you have to create a compile unit object CodeCompileUnit. It is the top container for namespaces, classes and members. Then add the components as follows:
public static void Test() {
CodeCompileUnit lUnit = new CodeCompileUnit();
CodeNamespace lNamespace = new CodeNamespace("MyNamespace");
lNamespace.Imports.Add(new CodeNamespaceImport("System"));
lNamespace.Imports.Add(new CodeNamespaceImport("System.IO")); // not used, just for demo
lUnit.Namespaces.Add(lNamespace);
CodeTypeDeclaration lClass = new CodeTypeDeclaration("MyClass");
CodeMethodInvokeExpression lExpression = new CodeMethodInvokeExpression(new CodeTypeReferenceExpression("Console"), "WriteLine", new CodePrimitiveExpression("hello world !"));
lNamespace.Types.Add(lClass);
// write Main entry point method
CodeEntryPointMethod lMain = new CodeEntryPointMethod();
lMain.Statements.Add(lExpression);
lClass.Members.Add(lMain);
// write another method
CodeMemberMethod lMethod = new CodeMemberMethod();
lMethod.Name = "DoSomething";
lClass.Members.Add(lMethod);
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
CodeGeneratorOptions lOptions = new CodeGeneratorOptions();
lOptions.IndentString = " "; // or "\t";
lOptions.BlankLinesBetweenMembers = true;
// generate a C# source code file
CSharpCodeProvider lCSharpCodeProvider = new CSharpCodeProvider();
using (StreamWriter lStreamWriter = new StreamWriter(lDesktopPath + "HelloWorld.cs", false)) {
lCSharpCodeProvider.GenerateCodeFromCompileUnit(lUnit, lStreamWriter, lOptions);
}
// generate a VB source code file
VBCodeProvider lVBCodeProvider = new VBCodeProvider();
using (StreamWriter lStreamWriter = new StreamWriter(lDesktopPath + "HelloWorld.vb", false)) {
lVBCodeProvider.GenerateCodeFromCompileUnit(lUnit, lStreamWriter, lOptions);
}
} //
example output file HelloWorld.cs:
//------------------------------------------------------------------------------ // // This code was generated by a tool. // Runtime Version:4.0.30319.18408 // // Changes to this file may cause incorrect behavior and will be lost if // the code is regenerated. // //------------------------------------------------------------------------------ namespace MyNamespace { using System; using System.IO; public class MyClass { public static void Main() { Console.WriteLine("hello world !"); } private void DoSomething() { } } }
example output file HelloWorld.vb:
'------------------------------------------------------------------------------ ' ' This code was generated by a tool. ' Runtime Version:4.0.30319.18408 ' ' Changes to this file may cause incorrect behavior and will be lost if ' the code is regenerated. ' '------------------------------------------------------------------------------ Option Strict Off Option Explicit On Imports System Imports System.IO Namespace MyNamespace Public Class [MyClass] Public Shared Sub Main() Console.WriteLine("hello world !") End Sub Private Sub DoSomething() End Sub End Class End Namespace
In the above example I have added an empty method. We will make it more complex now. The method will be called with parameters from Main() and return a value to a local variable. The local variable will be stored in a field via a property.
public static void Test2() {
CodeCompileUnit lUnit = new CodeCompileUnit();
CodeNamespace lNamespace = new CodeNamespace("MyNamespace");
lNamespace.Imports.Add(new CodeNamespaceImport("System"));
lUnit.Namespaces.Add(lNamespace);
CodeTypeDeclaration lClass = new CodeTypeDeclaration("MyClass");
lClass.IsClass = true;
lClass.Attributes = MemberAttributes.Public;
lNamespace.Types.Add(lClass);
// -----------------------------------------------
// method DoSomething()
// -----------------------------------------------
CodeTypeParameter lClassAsParameter = new CodeTypeParameter(lClass.Name);
CodeTypeReference lClassReference = new CodeTypeReference(lClassAsParameter);
CodeParameterDeclarationExpression lClassExpression = new CodeParameterDeclarationExpression(lClassReference, "xClass");
CodeMemberMethod lDoSomething = new CodeMemberMethod();
lDoSomething.Attributes = MemberAttributes.Public;
lDoSomething.Comments.Add(new CodeCommentStatement("This is an example comment for our method DoSomething()."));
lDoSomething.Name = "DoSomething";
lDoSomething.ReturnType = new CodeTypeReference(typeof(double));
lDoSomething.Parameters.Add(new CodeParameterDeclarationExpression(typeof(int), "xInteger"));
lDoSomething.Parameters.Add(new CodeParameterDeclarationExpression(typeof(string), "xString"));
lDoSomething.Parameters.Add(lClassExpression);
lDoSomething.Statements.Add(new CodeMethodReturnStatement(new CodePrimitiveExpression(1.0)));
lClass.Members.Add(lDoSomething);
// -----------------------------------------------
// private field _MyField
// -----------------------------------------------
CodeMemberField lField = new CodeMemberField(typeof(long), "_MyField");
lField.Attributes = MemberAttributes.Private;
lField.InitExpression = new CodePrimitiveExpression(10);
lClass.Members.Add(lField);
// -----------------------------------------------
// property MyProperty
// -----------------------------------------------
CodeMemberProperty lProperty = new CodeMemberProperty();
lProperty.Name = "MyProperty";
lProperty.Attributes = MemberAttributes.Public;
lProperty.Type = new CodeTypeReference(typeof(long));
lProperty.GetStatements.Add(new CodeMethodReturnStatement(new CodePrimitiveExpression(359)));
lProperty.SetStatements.Add(new CodeAssignStatement(new CodeFieldReferenceExpression(new CodeThisReferenceExpression(), lField.Name), new CodePropertySetValueReferenceExpression()));
lClass.Members.Add(lProperty);
// -----------------------------------------------
// Main()
// -----------------------------------------------
CodeEntryPointMethod lMain = new CodeEntryPointMethod();
lClass.Members.Add(lMain);
// local double variable d
CodeVariableReferenceExpression lLocalVariable_d = new CodeVariableReferenceExpression("d");
CodeVariableDeclarationStatement lLocalVariableStatement_d = new CodeVariableDeclarationStatement(typeof(double), lLocalVariable_d.VariableName);
lMain.Statements.Add(lLocalVariableStatement_d);
// local integer variable i
CodeVariableReferenceExpression lLocalVariable_i = new CodeVariableReferenceExpression("i");
CodeVariableDeclarationStatement lLocalVariableStatement_i = new CodeVariableDeclarationStatement(typeof(int), lLocalVariable_i.VariableName);
lLocalVariableStatement_i.InitExpression = new CodePrimitiveExpression(0);
lMain.Statements.Add(lLocalVariableStatement_i);
// local class variable MyClass
CodeVariableReferenceExpression lLocalVariable_Class = new CodeVariableReferenceExpression("lClass");
CodeVariableDeclarationStatement lLocalVariableStatement_Class = new CodeVariableDeclarationStatement(lClassReference, lLocalVariable_Class.VariableName);
lLocalVariableStatement_Class.InitExpression = new CodeObjectCreateExpression(lClass.Name, new CodeExpression[] { });
lMain.Statements.Add(lLocalVariableStatement_Class);
// DoSomething() method call
CodeMethodInvokeExpression lDoSomethingExpression = new CodeMethodInvokeExpression(lLocalVariable_Class, lDoSomething.Name, new CodeExpression[] { lLocalVariable_i, new CodePrimitiveExpression("hello"), lLocalVariable_Class});
CodeAssignStatement lAssignment = new CodeAssignStatement(lLocalVariable_d, lDoSomethingExpression);
lMain.Statements.Add(lAssignment);
//lMain.Statements.Add(lDoSomethingExpression); // without assignment
// cast the "double" result to type "long" and write it to field _MyField via property MyProperty
CodePropertyReferenceExpression lPropertyExpression = new CodePropertyReferenceExpression(lLocalVariable_Class, lProperty.Name);
CodeCastExpression lCastExpression = new CodeCastExpression(typeof(long), lLocalVariable_d);
lAssignment = new CodeAssignStatement(lPropertyExpression, lCastExpression);
lMain.Statements.Add(lAssignment);
// -----------------------------------------------
// create source code
// -----------------------------------------------
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
CodeGeneratorOptions lOptions = new CodeGeneratorOptions();
lOptions.IndentString = " "; // or "\t";
lOptions.BlankLinesBetweenMembers = true;
// generate a C# source code file
CSharpCodeProvider lCSharpCodeProvider = new CSharpCodeProvider();
using (StreamWriter lStreamWriter = new StreamWriter(lDesktopPath + "MoreComplex.cs", false)) {
lCSharpCodeProvider.GenerateCodeFromCompileUnit(lUnit, lStreamWriter, lOptions);
}
} //
example output:
//------------------------------------------------------------------------------ // // This code was generated by a tool. // Runtime Version:4.0.30319.18408 // // Changes to this file may cause incorrect behavior and will be lost if // the code is regenerated. // //------------------------------------------------------------------------------ namespace MyNamespace { using System; public class MyClass { private long _MyField = 10; public virtual long MyProperty { get { return 359; } set { this._MyField = value; } } // This is an example comment for our method DoSomething(). public virtual double DoSomething(int xInteger, string xString, MyClass xClass) { return 1D; } public static void Main() { double d; int i = 0; MyClass lClass = new MyClass(); d = lClass.DoSomething(i, "hello", lClass); lClass.MyProperty = ((long)(d)); } } }
Is your head burning already? This was quite complex. As usual there are easier ways for lazy people. I do not recommend any shortcuts. It undermines the idea of flexibility.
public static void Test3() {
CodeCompileUnit lUnit = new CodeCompileUnit();
CodeNamespace lNamespace = new CodeNamespace("MyNamespace");
lNamespace.Imports.Add(new CodeNamespaceImport("System"));
lUnit.Namespaces.Add(lNamespace);
CodeTypeDeclaration lClass = new CodeTypeDeclaration("MyClass");
lNamespace.Types.Add(lClass);
// write Main entry point method
CodeEntryPointMethod lMain = new CodeEntryPointMethod();
lMain.Statements.Add(new CodeSnippetExpression("Console.WriteLine(\"hello world !\")"));
lClass.Members.Add(lMain);
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
CodeGeneratorOptions lOptions = new CodeGeneratorOptions();
lOptions.IndentString = " "; // or "\t";
lOptions.BlankLinesBetweenMembers = true;
// generate a C# source code file
CSharpCodeProvider lCSharpCodeProvider = new CSharpCodeProvider();
using (StreamWriter lStreamWriter = new StreamWriter(lDesktopPath + "HelloWorldForLazyPeople.cs", false)) {
lCSharpCodeProvider.GenerateCodeFromCompileUnit(lUnit, lStreamWriter, lOptions);
}
} //
We created source code in the runtime environment. How can we execute it?
The next example compiles a class into an assembly and then executes the Main() method, which is automatically defined as the entry point. This happens in memory. If you want to write the executable code to a disk, then set GenerateInMemory to false and define a proper OutputAssembly path in the compiler parameters (variable lParameters). Use CompileAssemblyFromFile() instead of CompileAssemblyFromSource(). Btw. CompileAssemblyFromDom() would accept CodeCompileUnits (eg. used in Test2) as input data. Therefore you don’t have to generate a source code file to compile your code structure.
private static string[] _MyCode = new string[] {
"using System;" +
"public class MyClass {" +
"public static void Main() { Console.WriteLine(\"hello world !\"); }" +
"}" };
public static void Test4() {
CSharpCodeProvider lCodeProvider= new CSharpCodeProvider();
CompilerParameters lParameters = new CompilerParameters();
//lParameters.ReferencedAssemblies.Add("System.dll");
lParameters.GenerateInMemory = true;
lParameters.GenerateExecutable = true;
lParameters.OutputAssembly = "HelloWorld";
CompilerResults lCompilerResults = lCodeProvider.CompileAssemblyFromSource(lParameters, _MyCode);
if (lCompilerResults.Errors.Count > 0) {
foreach (CompilerError lError in lCompilerResults.Errors) Console.WriteLine(lError.ToString());
}
else {
Console.WriteLine("Compiled succesfully " + lCompilerResults.PathToAssembly);
Console.WriteLine("Executing code now");
MethodInfo m = lCompilerResults.CompiledAssembly.EntryPoint;
m.Invoke(null, new object[] {});
}
Console.ReadLine();
} //
example output:
Compiled succesfully
Executing code now
hello world !
Lambda expressions are using expression trees. The data-structure is like a tree. We are not necessarily talking about code here. Lambda expressions are not source code in the classical sense, because compilers translate expressions into code. In fact you don’t know the result, all you need to do is describing the code in an expression tree. For instance LINQ can interact between your code and a database. It can generate SQL queries. And these have nothing to do with the C# source code.
Let’s use an expression tree to generate another “Hello World” example at runtime. For this we need to implement the System.Linq.Expressions namespace. This is not really a part of reflection. I touch on the topic here to have it mentioned before we continue with Emit, which is far more on spot.
public static void Test5() {
Expression A = Expression.Call(null, typeof(Console).GetMethod("WriteLine", new Type[] { typeof(String) }), Expression.Constant("hello world !"));
Expression B = Expression.Call(null, typeof(Console).GetMethod("ReadLine", new Type[] { }), null);
Expression[] Expressions = new Expression[] { A, B };
BlockExpression lBlockExpression = Expression.Block(Expressions);
Action lAction = Expression.Lambda<Action>(lBlockExpression).Compile();
lAction();
} //
Let’s build a simple calculator now. Imagine someone enters a mathematical formula in a TextBox and wants to calculate it. I have seen so many solutions on the internet and I can tell you for sure they are all far too long! In the following code you just need to add some “nice features” like pre-parsing to allow more user flexibility. The entire Math library is yours! May the 4th be with you 😉
public static void Test6() {
string lTextBoxText = "Math.Floor(5.0 * (3.0 + 7.0) / 9.0 * Math.Exp(3.0))";
string[] lCode = new string[] {
"using System;" +
"public class MyClass {" +
"public static double Calc() { return (double)" + lTextBoxText + "; }" +
"}" };
CSharpCodeProvider lCodeProvider = new CSharpCodeProvider();
CompilerParameters lParameters = new CompilerParameters();
lParameters.GenerateInMemory = true;
lParameters.GenerateExecutable = false;
lParameters.OutputAssembly = "CalcAssembly";
CompilerResults lCompilerResults = lCodeProvider.CompileAssemblyFromSource(lParameters, lCode);
MethodInfo m = lCompilerResults.CompiledAssembly.GetType("MyClass").GetMethod("Calc");
double d = (double)m.Invoke(null, null);
Console.WriteLine("calculation result is " + d);
Console.WriteLine("cross check result is " + (Math.Floor(5.0 * (3.0 + 7.0) / 9.0 * Math.Exp(3.0))));
Console.ReadLine();
} //
example output:
calculation result is 111
cross check result is 111
Do you remember the post about extension methods? Unfortunately we cannot write any extension method for static classes like the .Net framework Math class. So I am going to use the string class instead. The calculator looks a little bit nicer then.
public static class MathExtension {
public static double Formula(this string xFormula) {
string[] lCode = new string[] {
"using System;" +
"public class MyClass {" +
"public static double Calc() { return (double)" + xFormula + "; }" +
"}" };
CSharpCodeProvider lCodeProvider = new CSharpCodeProvider();
CompilerParameters lParameters = new CompilerParameters();
lParameters.GenerateInMemory = true;
lParameters.GenerateExecutable = false;
lParameters.OutputAssembly = "CalcAssembly";
CompilerResults lCompilerResults = lCodeProvider.CompileAssemblyFromSource(lParameters, lCode);
MethodInfo m = lCompilerResults.CompiledAssembly.GetType("MyClass").GetMethod("Calc");
return (double)m.Invoke(null, null);
} //
} //
// how to call the extension method
public static void Test7() {
string lTextBoxText = "Math.Floor(5.0 * (3.0 + 7.0) / 9.0 * Math.Exp(3.0))";
double d = lTextBoxText.Formula();
Console.WriteLine(d);
Console.ReadLine();
} //
Epilogue
There are some stock market trading applications, which allow users to add their own algorithmic trading strategies. Wouldn’t it be great to enter your code in C# and use the .Net framework instead of reinventing the wheel and come up with your own scripting language that will never reach any widely recognized state of the art?
C# Hardcore Programming 