Blog Archives
Protocol Buffers (part 3, advanced), Tcp Networking
Ok guys, hardcore. Unfortunately in a different way than I thought 😉
Once again it took a lot of time to prepare the next source code example. It extends the code of my last post.
I removed the ProtoType class, because inheritance is not a real strength of Protobuf-Net. It is possible, but the ease of code maintenance is a definite argument against it. The complexity increases slowly, we cannot afford code that is difficult to change.
In theory we could write a Serialize() method for each class, the result would be extreme performance. You could use the BitConverter class to convert and then merge values without causing too much overhead. And by using Assembler I guess you could even make it 10 times faster again.
I think that all modern protocols lose a lot of time due to the required reflection, which is mainly the mapping of methods and properties. And unlike Assembler the modern languages do not allow to eg. simply write an integer to a memory address and only read the first byte of it.
You can rotate bits or multiply and divide, what is much less efficient.
In C# you can use the Parallel class to boost your performance. Of course the processing time of each code piece has to be substantial, otherwise it takes longer to create tasks than to solve the problems. In slow networks it might even be worth considering packing data before it is sent.
I replaced the ProtoType class by the Header class, which tells the type of the succeeding data block and also adds a serial identifier. This identifier can be used to receive vital feedbacks. For instance the server tells the client if the data transmission was successful or not. It can also send calculation results or error messages.
The data stream has the following order: header, data, header, data, header, data … except for feedback headers; they do not need data blocks. The enum eType inside the Header class defines all possible data types: ErrorMessage, Feedback, Book or Fable.
As said before I did not use inheritance for the Header class. The code remains legible and there is no spaghetti code to achieve indirect multiple inheritance.
In my last post, the client was sending and the server was receiving data. Now the communication is bidirectional. The client uses two threads, one to send and one to receive. The sending method is still using the BlockingCollection construction and an endless loop on a separate thread.
The server can be connected to several clients simultaneously. To keep it simple I decided sending data without any context switching. This usually blocks threads during the IO operation. I have added tasks inside the event OnMessageBook to give an example on how to avoid this. Nevertheless the send method uses a lock on the client preventing simultaneous write actions on the socket. This is bad practice; you should not apply locks on objects that might be locked in other areas as well. This is out of your scope, you don’t know if the .Net framework or any other code uses a lock on the same object, which could cause undesired behavior. In the below example it would have been better to wrap the client object into another class and apply the lock on that outer shell object. But I guess this is ok in our shorter example. The code was long enough already.
public static void Send(TcpClient xClient, ProtoBufExample.Header xHeader) {
if (xHeader == null) return;
if (xClient == null) return;
lock (xClient) {
NetworkStream lNetworkStream = xClient.GetStream();
....
The above lock problem could be avoided with this:
public class ClientData {
public TcpClient Client { get; private set; }
private DateTime _ConnectedSince;
private string _UserName;
....
public static void Send(ClientData xClient, ProtoBufExample.Header xHeader) {
if (xHeader == null) return;
if (xClient == null) return;
lock (xClient) {
NetworkStream lNetworkStream = xClient.Client.GetStream();
....
And here is the entire source code:
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using ProtoBuf;
namespace DemoApp {
public class ProtoBufExample {
public enum eType : byte { eError = 0, eFeedback, eBook, eFable };
[ProtoContract]
public class Header {
[ProtoMember(1)] public eType objectType;
[ProtoMember(2)] public readonly int serialMessageId;
public object data;
private static int _HeaderSerialId = 0;
public Header(object xData, eType xObjectType, int xSerialMessageId = 0) {
data = xData;
serialMessageId = (xSerialMessageId == 0) ? Interlocked.Increment(ref _HeaderSerialId) : xSerialMessageId;
objectType = xObjectType; // we could use "if typeof(T) ...", but it would be slower, harder to maintain and less legible
} // constructor
// parameterless constructor needed for Protobuf-net
public Header() {
} // constructor
} // class
[ProtoContract]
public class ErrorMessage {
[ProtoMember(1)]
public string Text;
} // class
[ProtoContract]
public class Book {
[ProtoMember(1)] public string author;
[ProtoMember(2, DataFormat = DataFormat.Group)] public List<Fable> stories;
[ProtoMember(3)] public DateTime edition;
[ProtoMember(4)] public int pages;
[ProtoMember(5)] public double price;
[ProtoMember(6)] public bool isEbook;
public override string ToString() {
StringBuilder s = new StringBuilder();
s.Append("by "); s.Append(author);
s.Append(", edition "); s.Append(edition.ToString("dd MMM yyyy"));
s.Append(", pages "); s.Append(pages);
s.Append(", price "); s.Append(price);
s.Append(", isEbook "); s.Append(isEbook);
s.AppendLine();
if (stories != null) foreach (Fable lFable in stories) {
s.Append("title "); s.Append(lFable.title);
s.Append(", rating "); s.Append(lFable.customerRatings.Average()); // Average() is an extension method of "using System.Linq;"
s.AppendLine();
}
return s.ToString();
} //
} // class
[ProtoContract]
public class Fable {
[ProtoMember(1)] public string title;
[ProtoMember(2, DataFormat = DataFormat.Group)]
public double[] customerRatings;
public override string ToString() {
return "title " + title + ", rating " + customerRatings.Average();
} //
} // class
public static Book GetData() {
return new Book {
author = "Aesop",
price = 1.99,
isEbook = false,
edition = new DateTime(1975, 03, 13),
pages = 203,
stories = new List<Fable>(new Fable[] {
new Fable{ title = "The Fox & the Grapes", customerRatings = new double[]{ 0.7, 0.7, 0.8} },
new Fable{ title = "The Goose that Laid the Golden Eggs", customerRatings = new double[]{ 0.6, 0.75, 0.5, 1.0} },
new Fable{ title = "The Cat & the Mice", customerRatings = new double[]{ 0.1, 0.0, 0.3} },
new Fable{ title = "The Mischievous Dog", customerRatings = new double[]{ 0.45, 0.5, 0.4, 0.0, 0.5} }
})
};
} //
} // class
public class PendingFeedbacks {
private readonly ConcurrentDictionary<int, ProtoBufExample.Header> _Messages = new ConcurrentDictionary<int, ProtoBufExample.Header>();
public int Count { get { return _Messages.Count; } }
public void Add(ProtoBufExample.Header xHeader) {
if (xHeader == null) throw new Exception("cannot add a null header");
if (!_Messages.TryAdd(xHeader.serialMessageId, xHeader)) {
throw new Exception("there must be a programming error somewhere");
}
} //
public void Remove(ProtoBufExample.Header xHeader) {
ProtoBufExample.Header lHeader;
if (!_Messages.TryRemove(xHeader.serialMessageId, out lHeader)) {
throw new Exception("there must be a programming error somewhere");
}
switch (xHeader.objectType) {
case ProtoBufExample.eType.eError:
Console.WriteLine("error: " + ((ProtoBufExample.ErrorMessage)xHeader.data).Text);
Console.WriteLine("the message that was sent out was: " + lHeader.objectType + " with serial id " + lHeader.serialMessageId);
Console.WriteLine("please check the log files" + Environment.NewLine);
break;
case ProtoBufExample.eType.eFeedback:
// all ok !
break;
default:
Console.WriteLine("warning: This message type was not expected.");
break;
}
} //
} // class
public static class NetworkTest {
public static void Test() {
NetworkListener lServer = new NetworkListener("127.0.0.1", 65432, "Server");
NetworkClient lClient = new NetworkClient("127.0.0.1", 65432, "Client");
lServer.Connect();
lServer.OnMessageBook += new NetworkListener.dOnMessageBook(OnMessageBook);
lServer.OnMessageFable += new NetworkListener.dOnMessageFable(OnMessageFable);
lClient.Connect();
ProtoBufExample.Header lHeader;
// send a book across the network
ProtoBufExample.Book lBook = ProtoBufExample.GetData();
lHeader = new ProtoBufExample.Header(lBook, ProtoBufExample.eType.eBook);
lClient.Send(lHeader);
System.Threading.Thread.Sleep(1000); // remove this to see the asynchonous processing (the output will look terrible)
// send a fable across the network
lHeader = new ProtoBufExample.Header(lBook.stories[1], ProtoBufExample.eType.eFable);
lClient.Send(lHeader);
System.Threading.Thread.Sleep(1000);
lClient.Disconnect();
lServer.Disconnect();
Console.ReadLine();
}
// demo: synchronous processing
static void OnMessageFable(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Fable xFable) {
Console.WriteLine(Environment.NewLine + "received a fable: ");
Console.WriteLine(xFable.ToString());
// demo: we tell the server that something went wrong
ProtoBufExample.ErrorMessage lErrorMessage = new ProtoBufExample.ErrorMessage() { Text = "The fable was rejected. It is far too short." };
ProtoBufExample.Header lErrorHeader = new ProtoBufExample.Header(lErrorMessage, ProtoBufExample.eType.eError, xHeader.serialMessageId);
NetworkListener.Send(xSender, lErrorHeader);
} //
// demo: asynchronous processing
static void OnMessageBook(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Book xBook) {
Task.Factory.StartNew(() => {
Console.WriteLine(Environment.NewLine + "received a book: ");
Console.WriteLine(xBook.ToString());
// send a feedback without any body to signal all was ok
ProtoBufExample.Header lFeedback = new ProtoBufExample.Header(null, ProtoBufExample.eType.eFeedback, xHeader.serialMessageId);
NetworkListener.Send(xSender, lFeedback);
return;
});
Console.WriteLine("Book event was raised");
} //
} // class
public class NetworkListener {
private bool _ExitLoop = true;
private TcpListener _Listener;
public delegate void dOnMessageBook(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Book xBook);
public event dOnMessageBook OnMessageBook;
public delegate void dOnMessageFable(TcpClient xSender, ProtoBufExample.Header xHeader, ProtoBufExample.Fable xFable);
public event dOnMessageFable OnMessageFable;
private List<TcpClient> _Clients = new List<TcpClient>();
public int Port { get; private set; }
public string IpAddress { get; private set; }
public string ThreadName { get; private set; }
public NetworkListener(string xIpAddress, int xPort, string xThreadName) {
Port = xPort;
IpAddress = xIpAddress;
ThreadName = xThreadName;
} //
public bool Connect() {
if (!_ExitLoop) {
Console.WriteLine("Listener running already");
return false;
}
_ExitLoop = false;
try {
_Listener = new TcpListener(IPAddress.Parse(IpAddress), Port);
_Listener.Start();
Thread lThread = new Thread(new ThreadStart(LoopWaitingForClientsToConnect));
lThread.IsBackground = true;
lThread.Name = ThreadName + "WaitingForClients";
lThread.Start();
return true;
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
return false;
} //
public void Disconnect() {
_ExitLoop = true;
lock (_Clients) {
foreach (TcpClient lClient in _Clients) lClient.Close();
_Clients.Clear();
}
} //
private void LoopWaitingForClientsToConnect() {
try {
while (!_ExitLoop) {
Console.WriteLine("waiting for a client");
TcpClient lClient = _Listener.AcceptTcpClient();
string lClientIpAddress = lClient.Client.LocalEndPoint.ToString();
Console.WriteLine("new client connecting: " + lClientIpAddress);
if (_ExitLoop) break;
lock (_Clients) _Clients.Add(lClient);
Thread lThread = new Thread(new ParameterizedThreadStart(LoopRead));
lThread.IsBackground = true;
lThread.Name = ThreadName + "CommunicatingWithClient";
lThread.Start(lClient);
}
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
finally {
_ExitLoop = true;
if (_Listener != null) _Listener.Stop();
}
} //
private void LoopRead(object xClient) {
TcpClient lClient = xClient as TcpClient;
NetworkStream lNetworkStream = lClient.GetStream();
while (!_ExitLoop) {
try {
ProtoBufExample.Header lHeader = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Header>(lNetworkStream, ProtoBuf.PrefixStyle.Fixed32);
if (lHeader == null) break; // happens during shutdown process
switch (lHeader.objectType) {
case ProtoBufExample.eType.eBook:
ProtoBufExample.Book lBook = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Book>(lNetworkStream, ProtoBuf.PrefixStyle.Fixed32);
if (lBook == null) break;
lHeader.data = lBook; // not necessary, but nicer
dOnMessageBook lEventBook = OnMessageBook;
if (lEventBook == null) continue;
lEventBook(lClient, lHeader, lBook);
break;
case ProtoBufExample.eType.eFable:
ProtoBufExample.Fable lFable = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Fable>(lNetworkStream, ProtoBuf.PrefixStyle.Fixed32);
if (lFable == null) break;
lHeader.data = lFable; // not necessary, but nicer
dOnMessageFable lEventFable = OnMessageFable;
if (lEventFable == null) continue;
lEventFable(lClient, lHeader, lFable);
break;
default:
Console.WriteLine("Mayday, mayday, we are in big trouble.");
break;
}
}
catch (System.IO.IOException) {
if (_ExitLoop) Console.WriteLine("user requested client shutdown");
else Console.WriteLine("disconnected");
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
Console.WriteLine("server: listener is shutting down");
} //
public static void Send(TcpClient xClient, ProtoBufExample.Header xHeader) {
if (xHeader == null) return;
if (xClient == null) return;
lock (xClient) {
try {
NetworkStream lNetworkStream = xClient.GetStream();
// send header (most likely a simple feedback)
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Header>(lNetworkStream, xHeader, ProtoBuf.PrefixStyle.Fixed32);
// send errors
if (xHeader.objectType != ProtoBufExample.eType.eError) return;
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.ErrorMessage>(lNetworkStream, (ProtoBufExample.ErrorMessage)xHeader.data, ProtoBuf.PrefixStyle.Fixed32);
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
} //
} // class
public class NetworkClient {
public int Port { get; private set; }
public string IpAddress { get; private set; }
public string ThreadName { get; private set; }
private NetworkStream _NetworkStream = null;
private TcpClient _Client = null;
private bool _ExitLoop = true;
private BlockingCollection<ProtoBufExample.Header> _Queue = new BlockingCollection<ProtoBufExample.Header>();
private readonly PendingFeedbacks _PendingFeedbacks = new PendingFeedbacks();
public NetworkClient(string xIpAddress, int xPort, string xThreadName) {
Port = xPort;
IpAddress = xIpAddress;
ThreadName = xThreadName;
} //
public void Connect() {
if (!_ExitLoop) return; // running already
_ExitLoop = false;
_Client = new TcpClient();
_Client.Connect(IpAddress, Port);
_NetworkStream = _Client.GetStream();
Thread lLoopWrite = new Thread(new ThreadStart(LoopWrite));
lLoopWrite.IsBackground = true;
lLoopWrite.Name = ThreadName + "Write";
lLoopWrite.Start();
Thread lLoopRead = new Thread(new ThreadStart(LoopRead));
lLoopRead.IsBackground = true;
lLoopRead.Name = ThreadName + "Read";
lLoopRead.Start();
} //
public void Disconnect() {
_ExitLoop = true;
_Queue.Add(null);
if (_Client != null) _Client.Close();
//if (_NetworkStream != null) _NetworkStream.Close();
} //
public void Send(ProtoBufExample.Header xHeader) {
if (xHeader == null) return;
_PendingFeedbacks.Add(xHeader);
_Queue.Add(xHeader);
} //
private void LoopWrite() {
while (!_ExitLoop) {
try {
ProtoBufExample.Header lHeader = _Queue.Take();
if (lHeader == null) break;
// send header
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Header>(_NetworkStream, lHeader, ProtoBuf.PrefixStyle.Fixed32);
// send data
switch (lHeader.objectType) {
case ProtoBufExample.eType.eBook:
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Book>(_NetworkStream, (ProtoBufExample.Book)lHeader.data, ProtoBuf.PrefixStyle.Fixed32);
break;
case ProtoBufExample.eType.eFable:
ProtoBuf.Serializer.SerializeWithLengthPrefix<ProtoBufExample.Fable>(_NetworkStream, (ProtoBufExample.Fable)lHeader.data, ProtoBuf.PrefixStyle.Fixed32);
break;
default:
break;
}
}
catch (System.IO.IOException) {
if (_ExitLoop) Console.WriteLine("user requested client shutdown.");
else Console.WriteLine("disconnected");
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
_ExitLoop = true;
Console.WriteLine("client: writer is shutting down");
} //
private void LoopRead() {
while (!_ExitLoop) {
try {
ProtoBufExample.Header lHeader = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.Header>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
if (lHeader == null) break;
if (lHeader.objectType == ProtoBufExample.eType.eError) {
ProtoBufExample.ErrorMessage lErrorMessage = ProtoBuf.Serializer.DeserializeWithLengthPrefix<ProtoBufExample.ErrorMessage>(_NetworkStream, ProtoBuf.PrefixStyle.Fixed32);
lHeader.data = lErrorMessage;
}
_PendingFeedbacks.Remove(lHeader);
}
catch (System.IO.IOException) {
if (_ExitLoop) Console.WriteLine("user requested client shutdown");
else Console.WriteLine("disconnected");
}
catch (Exception ex) { Console.WriteLine(ex.Message); }
}
Console.WriteLine("client: reader is shutting down");
} //
} // class
} // namespace
example output:
waiting for a client
new client connecting: 127.0.0.1:65432
waiting for a client
Book event was raisedreceived a book:
by Aesop, edition 13 Mar 1975, pages 203, price 1.99, isEbook False
title The Fox & the Grapes, rating 0.733333333333333
title The Goose that Laid the Golden Eggs, rating 0.7125
title The Cat & the Mice, rating 0.133333333333333
title The Mischievous Dog, rating 0.37received a fable:
title The Goose that Laid the Golden Eggs, rating 0.7125
error: The fable was rejected. It is far too short.
the message that was sent out was: eFable with serial id 2
please check the log filesclient: writer is shutting down
server: listener is shutting down
user requested client shutdown
client: reader is shutting down
Reflection (part 5, professional), CodeDOM (& lambda expression tree)
Today’s post is about code generation at runtime using CodeDOM, which is the logical step before runtime code generation using Emit. The .Net framework has been built in an amazing way. We can easily neglect some flaws. Compared to Java there are no flaws at all. Jealousy needs to be deserved, sympathy is for free. Somehow Java is for free, and this is what you get.
Make sure, you are aware of the following differences. There are Expression and Statement. Please follow the links if you feel uncertain about the definitions.
Expression
An expression is a sequence of one or more operands and zero or more operators that can be evaluated to a single value, object, method, or namespace.
Statement
The actions that a program takes are expressed in statements. Common actions include declaring variables, assigning values, calling methods, looping through collections, and branching to one or another block of code, depending on a given condition. The order in which statements are executed in a program is called the flow of control or flow of execution.
A statement can consist of a single line of code that ends in a semicolon, or a series of single-line statements in a block. A statement block is enclosed in {} brackets and can contain nested blocks.
Succinct:
An Expression is a kind of mathematical or logical expression.
A Statement is a single code instruction or block of code to execute.
The first example code generates source code for C# and VB and saves the generated files on your desktop. Of course it has to be the classical “hello world” program.
CodeDOM stands for Code Document Object Model. It is located in the System.CodeDom namespace and provides methods to create code at runtime.
// what you need for the code example: using System; using System.CodeDom; using System.CodeDom.Compiler; using System.IO; using Microsoft.CSharp; using Microsoft.VisualBasic;
First of all you have to create a compile unit object CodeCompileUnit. It is the top container for namespaces, classes and members. Then add the components as follows:
public static void Test() {
CodeCompileUnit lUnit = new CodeCompileUnit();
CodeNamespace lNamespace = new CodeNamespace("MyNamespace");
lNamespace.Imports.Add(new CodeNamespaceImport("System"));
lNamespace.Imports.Add(new CodeNamespaceImport("System.IO")); // not used, just for demo
lUnit.Namespaces.Add(lNamespace);
CodeTypeDeclaration lClass = new CodeTypeDeclaration("MyClass");
CodeMethodInvokeExpression lExpression = new CodeMethodInvokeExpression(new CodeTypeReferenceExpression("Console"), "WriteLine", new CodePrimitiveExpression("hello world !"));
lNamespace.Types.Add(lClass);
// write Main entry point method
CodeEntryPointMethod lMain = new CodeEntryPointMethod();
lMain.Statements.Add(lExpression);
lClass.Members.Add(lMain);
// write another method
CodeMemberMethod lMethod = new CodeMemberMethod();
lMethod.Name = "DoSomething";
lClass.Members.Add(lMethod);
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
CodeGeneratorOptions lOptions = new CodeGeneratorOptions();
lOptions.IndentString = " "; // or "\t";
lOptions.BlankLinesBetweenMembers = true;
// generate a C# source code file
CSharpCodeProvider lCSharpCodeProvider = new CSharpCodeProvider();
using (StreamWriter lStreamWriter = new StreamWriter(lDesktopPath + "HelloWorld.cs", false)) {
lCSharpCodeProvider.GenerateCodeFromCompileUnit(lUnit, lStreamWriter, lOptions);
}
// generate a VB source code file
VBCodeProvider lVBCodeProvider = new VBCodeProvider();
using (StreamWriter lStreamWriter = new StreamWriter(lDesktopPath + "HelloWorld.vb", false)) {
lVBCodeProvider.GenerateCodeFromCompileUnit(lUnit, lStreamWriter, lOptions);
}
} //
example output file HelloWorld.cs:
//------------------------------------------------------------------------------ // // This code was generated by a tool. // Runtime Version:4.0.30319.18408 // // Changes to this file may cause incorrect behavior and will be lost if // the code is regenerated. // //------------------------------------------------------------------------------ namespace MyNamespace { using System; using System.IO; public class MyClass { public static void Main() { Console.WriteLine("hello world !"); } private void DoSomething() { } } }
example output file HelloWorld.vb:
'------------------------------------------------------------------------------ ' ' This code was generated by a tool. ' Runtime Version:4.0.30319.18408 ' ' Changes to this file may cause incorrect behavior and will be lost if ' the code is regenerated. ' '------------------------------------------------------------------------------ Option Strict Off Option Explicit On Imports System Imports System.IO Namespace MyNamespace Public Class [MyClass] Public Shared Sub Main() Console.WriteLine("hello world !") End Sub Private Sub DoSomething() End Sub End Class End Namespace
In the above example I have added an empty method. We will make it more complex now. The method will be called with parameters from Main() and return a value to a local variable. The local variable will be stored in a field via a property.
public static void Test2() {
CodeCompileUnit lUnit = new CodeCompileUnit();
CodeNamespace lNamespace = new CodeNamespace("MyNamespace");
lNamespace.Imports.Add(new CodeNamespaceImport("System"));
lUnit.Namespaces.Add(lNamespace);
CodeTypeDeclaration lClass = new CodeTypeDeclaration("MyClass");
lClass.IsClass = true;
lClass.Attributes = MemberAttributes.Public;
lNamespace.Types.Add(lClass);
// -----------------------------------------------
// method DoSomething()
// -----------------------------------------------
CodeTypeParameter lClassAsParameter = new CodeTypeParameter(lClass.Name);
CodeTypeReference lClassReference = new CodeTypeReference(lClassAsParameter);
CodeParameterDeclarationExpression lClassExpression = new CodeParameterDeclarationExpression(lClassReference, "xClass");
CodeMemberMethod lDoSomething = new CodeMemberMethod();
lDoSomething.Attributes = MemberAttributes.Public;
lDoSomething.Comments.Add(new CodeCommentStatement("This is an example comment for our method DoSomething()."));
lDoSomething.Name = "DoSomething";
lDoSomething.ReturnType = new CodeTypeReference(typeof(double));
lDoSomething.Parameters.Add(new CodeParameterDeclarationExpression(typeof(int), "xInteger"));
lDoSomething.Parameters.Add(new CodeParameterDeclarationExpression(typeof(string), "xString"));
lDoSomething.Parameters.Add(lClassExpression);
lDoSomething.Statements.Add(new CodeMethodReturnStatement(new CodePrimitiveExpression(1.0)));
lClass.Members.Add(lDoSomething);
// -----------------------------------------------
// private field _MyField
// -----------------------------------------------
CodeMemberField lField = new CodeMemberField(typeof(long), "_MyField");
lField.Attributes = MemberAttributes.Private;
lField.InitExpression = new CodePrimitiveExpression(10);
lClass.Members.Add(lField);
// -----------------------------------------------
// property MyProperty
// -----------------------------------------------
CodeMemberProperty lProperty = new CodeMemberProperty();
lProperty.Name = "MyProperty";
lProperty.Attributes = MemberAttributes.Public;
lProperty.Type = new CodeTypeReference(typeof(long));
lProperty.GetStatements.Add(new CodeMethodReturnStatement(new CodePrimitiveExpression(359)));
lProperty.SetStatements.Add(new CodeAssignStatement(new CodeFieldReferenceExpression(new CodeThisReferenceExpression(), lField.Name), new CodePropertySetValueReferenceExpression()));
lClass.Members.Add(lProperty);
// -----------------------------------------------
// Main()
// -----------------------------------------------
CodeEntryPointMethod lMain = new CodeEntryPointMethod();
lClass.Members.Add(lMain);
// local double variable d
CodeVariableReferenceExpression lLocalVariable_d = new CodeVariableReferenceExpression("d");
CodeVariableDeclarationStatement lLocalVariableStatement_d = new CodeVariableDeclarationStatement(typeof(double), lLocalVariable_d.VariableName);
lMain.Statements.Add(lLocalVariableStatement_d);
// local integer variable i
CodeVariableReferenceExpression lLocalVariable_i = new CodeVariableReferenceExpression("i");
CodeVariableDeclarationStatement lLocalVariableStatement_i = new CodeVariableDeclarationStatement(typeof(int), lLocalVariable_i.VariableName);
lLocalVariableStatement_i.InitExpression = new CodePrimitiveExpression(0);
lMain.Statements.Add(lLocalVariableStatement_i);
// local class variable MyClass
CodeVariableReferenceExpression lLocalVariable_Class = new CodeVariableReferenceExpression("lClass");
CodeVariableDeclarationStatement lLocalVariableStatement_Class = new CodeVariableDeclarationStatement(lClassReference, lLocalVariable_Class.VariableName);
lLocalVariableStatement_Class.InitExpression = new CodeObjectCreateExpression(lClass.Name, new CodeExpression[] { });
lMain.Statements.Add(lLocalVariableStatement_Class);
// DoSomething() method call
CodeMethodInvokeExpression lDoSomethingExpression = new CodeMethodInvokeExpression(lLocalVariable_Class, lDoSomething.Name, new CodeExpression[] { lLocalVariable_i, new CodePrimitiveExpression("hello"), lLocalVariable_Class});
CodeAssignStatement lAssignment = new CodeAssignStatement(lLocalVariable_d, lDoSomethingExpression);
lMain.Statements.Add(lAssignment);
//lMain.Statements.Add(lDoSomethingExpression); // without assignment
// cast the "double" result to type "long" and write it to field _MyField via property MyProperty
CodePropertyReferenceExpression lPropertyExpression = new CodePropertyReferenceExpression(lLocalVariable_Class, lProperty.Name);
CodeCastExpression lCastExpression = new CodeCastExpression(typeof(long), lLocalVariable_d);
lAssignment = new CodeAssignStatement(lPropertyExpression, lCastExpression);
lMain.Statements.Add(lAssignment);
// -----------------------------------------------
// create source code
// -----------------------------------------------
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
CodeGeneratorOptions lOptions = new CodeGeneratorOptions();
lOptions.IndentString = " "; // or "\t";
lOptions.BlankLinesBetweenMembers = true;
// generate a C# source code file
CSharpCodeProvider lCSharpCodeProvider = new CSharpCodeProvider();
using (StreamWriter lStreamWriter = new StreamWriter(lDesktopPath + "MoreComplex.cs", false)) {
lCSharpCodeProvider.GenerateCodeFromCompileUnit(lUnit, lStreamWriter, lOptions);
}
} //
example output:
//------------------------------------------------------------------------------ // // This code was generated by a tool. // Runtime Version:4.0.30319.18408 // // Changes to this file may cause incorrect behavior and will be lost if // the code is regenerated. // //------------------------------------------------------------------------------ namespace MyNamespace { using System; public class MyClass { private long _MyField = 10; public virtual long MyProperty { get { return 359; } set { this._MyField = value; } } // This is an example comment for our method DoSomething(). public virtual double DoSomething(int xInteger, string xString, MyClass xClass) { return 1D; } public static void Main() { double d; int i = 0; MyClass lClass = new MyClass(); d = lClass.DoSomething(i, "hello", lClass); lClass.MyProperty = ((long)(d)); } } }
Is your head burning already? This was quite complex. As usual there are easier ways for lazy people. I do not recommend any shortcuts. It undermines the idea of flexibility.
public static void Test3() {
CodeCompileUnit lUnit = new CodeCompileUnit();
CodeNamespace lNamespace = new CodeNamespace("MyNamespace");
lNamespace.Imports.Add(new CodeNamespaceImport("System"));
lUnit.Namespaces.Add(lNamespace);
CodeTypeDeclaration lClass = new CodeTypeDeclaration("MyClass");
lNamespace.Types.Add(lClass);
// write Main entry point method
CodeEntryPointMethod lMain = new CodeEntryPointMethod();
lMain.Statements.Add(new CodeSnippetExpression("Console.WriteLine(\"hello world !\")"));
lClass.Members.Add(lMain);
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
CodeGeneratorOptions lOptions = new CodeGeneratorOptions();
lOptions.IndentString = " "; // or "\t";
lOptions.BlankLinesBetweenMembers = true;
// generate a C# source code file
CSharpCodeProvider lCSharpCodeProvider = new CSharpCodeProvider();
using (StreamWriter lStreamWriter = new StreamWriter(lDesktopPath + "HelloWorldForLazyPeople.cs", false)) {
lCSharpCodeProvider.GenerateCodeFromCompileUnit(lUnit, lStreamWriter, lOptions);
}
} //
We created source code in the runtime environment. How can we execute it?
The next example compiles a class into an assembly and then executes the Main() method, which is automatically defined as the entry point. This happens in memory. If you want to write the executable code to a disk, then set GenerateInMemory to false and define a proper OutputAssembly path in the compiler parameters (variable lParameters). Use CompileAssemblyFromFile() instead of CompileAssemblyFromSource(). Btw. CompileAssemblyFromDom() would accept CodeCompileUnits (eg. used in Test2) as input data. Therefore you don’t have to generate a source code file to compile your code structure.
private static string[] _MyCode = new string[] {
"using System;" +
"public class MyClass {" +
"public static void Main() { Console.WriteLine(\"hello world !\"); }" +
"}" };
public static void Test4() {
CSharpCodeProvider lCodeProvider= new CSharpCodeProvider();
CompilerParameters lParameters = new CompilerParameters();
//lParameters.ReferencedAssemblies.Add("System.dll");
lParameters.GenerateInMemory = true;
lParameters.GenerateExecutable = true;
lParameters.OutputAssembly = "HelloWorld";
CompilerResults lCompilerResults = lCodeProvider.CompileAssemblyFromSource(lParameters, _MyCode);
if (lCompilerResults.Errors.Count > 0) {
foreach (CompilerError lError in lCompilerResults.Errors) Console.WriteLine(lError.ToString());
}
else {
Console.WriteLine("Compiled succesfully " + lCompilerResults.PathToAssembly);
Console.WriteLine("Executing code now");
MethodInfo m = lCompilerResults.CompiledAssembly.EntryPoint;
m.Invoke(null, new object[] {});
}
Console.ReadLine();
} //
example output:
Compiled succesfully
Executing code now
hello world !
Lambda expressions are using expression trees. The data-structure is like a tree. We are not necessarily talking about code here. Lambda expressions are not source code in the classical sense, because compilers translate expressions into code. In fact you don’t know the result, all you need to do is describing the code in an expression tree. For instance LINQ can interact between your code and a database. It can generate SQL queries. And these have nothing to do with the C# source code.
Let’s use an expression tree to generate another “Hello World” example at runtime. For this we need to implement the System.Linq.Expressions namespace. This is not really a part of reflection. I touch on the topic here to have it mentioned before we continue with Emit, which is far more on spot.
public static void Test5() {
Expression A = Expression.Call(null, typeof(Console).GetMethod("WriteLine", new Type[] { typeof(String) }), Expression.Constant("hello world !"));
Expression B = Expression.Call(null, typeof(Console).GetMethod("ReadLine", new Type[] { }), null);
Expression[] Expressions = new Expression[] { A, B };
BlockExpression lBlockExpression = Expression.Block(Expressions);
Action lAction = Expression.Lambda<Action>(lBlockExpression).Compile();
lAction();
} //
Let’s build a simple calculator now. Imagine someone enters a mathematical formula in a TextBox and wants to calculate it. I have seen so many solutions on the internet and I can tell you for sure they are all far too long! In the following code you just need to add some “nice features” like pre-parsing to allow more user flexibility. The entire Math library is yours! May the 4th be with you 😉
public static void Test6() {
string lTextBoxText = "Math.Floor(5.0 * (3.0 + 7.0) / 9.0 * Math.Exp(3.0))";
string[] lCode = new string[] {
"using System;" +
"public class MyClass {" +
"public static double Calc() { return (double)" + lTextBoxText + "; }" +
"}" };
CSharpCodeProvider lCodeProvider = new CSharpCodeProvider();
CompilerParameters lParameters = new CompilerParameters();
lParameters.GenerateInMemory = true;
lParameters.GenerateExecutable = false;
lParameters.OutputAssembly = "CalcAssembly";
CompilerResults lCompilerResults = lCodeProvider.CompileAssemblyFromSource(lParameters, lCode);
MethodInfo m = lCompilerResults.CompiledAssembly.GetType("MyClass").GetMethod("Calc");
double d = (double)m.Invoke(null, null);
Console.WriteLine("calculation result is " + d);
Console.WriteLine("cross check result is " + (Math.Floor(5.0 * (3.0 + 7.0) / 9.0 * Math.Exp(3.0))));
Console.ReadLine();
} //
example output:
calculation result is 111
cross check result is 111
Do you remember the post about extension methods? Unfortunately we cannot write any extension method for static classes like the .Net framework Math class. So I am going to use the string class instead. The calculator looks a little bit nicer then.
public static class MathExtension {
public static double Formula(this string xFormula) {
string[] lCode = new string[] {
"using System;" +
"public class MyClass {" +
"public static double Calc() { return (double)" + xFormula + "; }" +
"}" };
CSharpCodeProvider lCodeProvider = new CSharpCodeProvider();
CompilerParameters lParameters = new CompilerParameters();
lParameters.GenerateInMemory = true;
lParameters.GenerateExecutable = false;
lParameters.OutputAssembly = "CalcAssembly";
CompilerResults lCompilerResults = lCodeProvider.CompileAssemblyFromSource(lParameters, lCode);
MethodInfo m = lCompilerResults.CompiledAssembly.GetType("MyClass").GetMethod("Calc");
return (double)m.Invoke(null, null);
} //
} //
// how to call the extension method
public static void Test7() {
string lTextBoxText = "Math.Floor(5.0 * (3.0 + 7.0) / 9.0 * Math.Exp(3.0))";
double d = lTextBoxText.Formula();
Console.WriteLine(d);
Console.ReadLine();
} //
Epilogue
There are some stock market trading applications, which allow users to add their own algorithmic trading strategies. Wouldn’t it be great to enter your code in C# and use the .Net framework instead of reinventing the wheel and come up with your own scripting language that will never reach any widely recognized state of the art?
Reflection (part 3, advanced, professional), plugin
We are progressing further.
Reflection tells us all we need for plugins. Once we know the assembly we can examine it like a poor frog on the dissection table. If the dissector cannot find something, then it is not the fault of the frog. Sorry for the macabre metaphor, I could not resist my ugly thoughts.
First of all we need an assembly. In the Solution Explorer right-click your Project. The tab-menu shows (Application, Build, Build Events, Debug …). Select Application. The first entry is “Assembly name:”. You can enter your prefered assembly name here. Let’s use “TestAssembly” for our example. Now the compiler will create an assembly with the name TestAssembly.
A short test program gives you an idea where your assembly is stored and what its name is.
Assembly lAssembly = Assembly.GetCallingAssembly(); Console.WriteLine(lAssembly.Location); // prints the full path of your assembly Console.WriteLine(lAssembly.FullName);
You can load assemblies into applications. We are concentrating on plugins today.
In theory you can get all required information to eg. call methods or properties via reflection itself. There is no need for extra definition files. But you can also make your life easier and predefine signatures, which are used by both the application and the plugin. Of course you are implementing constraints that reduce the flexibility, but on the other hand you reduce the error-proneness and save a lot of debugging time.
So we are going to use an interface to predefine signatures. It is important that you do not copy the interface source code into both projects. You should compile the interface into a DLL-library and access these definitions from outside in the two assemblies. Otherwise the compilers generate different identifiers (see GUID) and two equal interfaces cannot be recognized as the same type.
To clearly show the independence I want you to open Visual Studio three times. Please do not create three projects in one single solution.
Visual Studio 1
Create a library project and paste the following code. We don’t need any “using” or “namespace”. Five lines of code are enough in our example.
public interface IPlugin {
string Name { get; }
int Born { get; }
double DoSomething(double d, string s);
} // interface
Compile the project and move the library DLL-file to your preferred location.
Visual Studio 2
Add a reference to our interface library. (Right click on References in your Solution Explorer. Add Reference, then browse for the library file.)
We are creating the plugin now. The method Test() prints a path that you will need in the source code for the calling application (just to make your life easier).
public class MyPlugin1 : IPlugin {
public string Name { get { return "Diana Frances"; } }
public int Born { get { return 1961; } }
public double DoSomething(double d, string s) { return 1.0; }
} // class
public class MyPlugin2 : IPlugin {
public string Name { get { return "Jeanne d'Arc"; } }
public int Born { get { return 1412; } }
public double DoSomething(double d, string s) { return 1.0; }
} // class
public static class TestClass {
public static void Test() {
Assembly lPluginAssembly = Assembly.GetCallingAssembly();
// Assembly lPluginAssembly = Assembly.Load("TestAssembly"); // alternative
Console.WriteLine(lPluginAssembly.Location);
Console.WriteLine(lPluginAssembly.FullName);
Console.ReadLine();
} //
} // class
Visual Studio 3
The projects in Visual Studio 1 and Visual Studio 2 are compiled. You can close these two programs.
Once again add a reference to our interface library. In below example code you need to replace a path by the one, which was printed out in Visual Studio 2. (in line Assembly lPluginAssembly = Assembly.LoadFile(@”xxxxxx\TestAssembly.exe”);)
static void Main(string[] args) {
Assembly lPluginAssembly = Assembly.LoadFile(@"xxxxxx\TestAssembly.exe");
//Assembly lPluginAssembly = Assembly.Load("TestAssembly"); => if loaded already
var lPlugins = from t in lPluginAssembly.GetTypes()
where typeof(IPlugin).IsAssignableFrom(t) && !t.IsInterface
select t;
foreach (Type t in lPlugins) {
IPlugin lPlugin = Activator.CreateInstance(t) as IPlugin;
Console.WriteLine(lPlugin.Name + " was born in " + lPlugin.Born);
double d = lPlugin.DoSomething(2.0, "hello");
}
Console.ReadLine();
} // main
output example:
Diana Frances was born in 1961
Jeanne d’Arc was born in 1412
The program loaded the assembly, created instances for each class and called the properties and the method. This basically is all you need for Plugins.
Of course there are also dirty ways to invoke plugin methods. One is:
Assembly lPluginAssembly = Assembly.LoadFile(@"xxxxxx\TestAssembly.exe");
Type t = lPluginAssembly.GetType("MyNamespaceName.MyPlugin1");
MethodInfo m = t.GetMethod("DoSomething");
double d = (double)m.Invoke(Activator.CreateInstance(t), new object[] { 2.0, "hello" });
C# Hardcore Programming 