Blog Archives
JSON to LINQ to JSON
It is about 9:30pm and I am sitting on the train home. It has been quite tough organizing my life in the last months. This is not the promised follow-up on OpenGL. There has been no time to read any book.
Anyway, this post shows how to run LINQ queries on JSON. The use can be quite broad. Imagine a shell command executing your LINQ on any JSON file. You can query into depth and build complex return tree structures with 2 or more levels. LINQ itself is highly flexible. And – just to give it the right flair – it returns results in the good old JSON format.
You could even go a bit further and insert more than just a LINQ query. In theory, you could add any C# code at run-time. The changes to this program would be minuscule.
I have sufficiently commented the code. There is a minimum overhead to get the job done.
How it works:
- The JSON data is imported using the JavaScriptSerializer, which is part of the System.Web.Extensions library reference. We try to force the result into a Dictionary<string, object>. Do not lazily use ‘object’. Some type information would get lost. I played with this and got eg. arrays instead of ArrayLists.
- The resulting structure is then used to build classes. These are converted to legible C# source code.
- The LINQ command is inserted. You can see the result in the TextBox titled ‘C# source code output’.
- A compiler instance is created and some library references are added. We compile the generated source code at run-time and then execute the binary.
- A standard JSON object is returned, rudimentary formatted and displayed as the final result.
Btw. I have inserted a horizontal and a vertical GridSplitter into the WPF XAML. You can easily change the size of the TextBoxes while playing with this tool. A few JSON examples were also added.
<Window xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" mc:Ignorable="d" x:Class="LinqJson.MainWindow" Title="JSON LINQ JSON" Height="Auto" Width="876" d:DesignHeight="695">
<Grid>
<Grid.RowDefinitions>
<RowDefinition Height="1*"/>
<RowDefinition Height="43*"/>
<RowDefinition Height="30"/>
<RowDefinition Height="150*"/>
<RowDefinition Height="30"/>
<RowDefinition Height="150*"/>
<RowDefinition Height="10*"/>
</Grid.RowDefinitions>
<Grid.ColumnDefinitions>
<ColumnDefinition Width="10*"/>
<ColumnDefinition Width="250*"/>
<ColumnDefinition Width="10"/>
<ColumnDefinition Width="250*"/>
<ColumnDefinition Width="10*"/>
<ColumnDefinition/>
</Grid.ColumnDefinitions>
<Button Content="convert" Width="50" Height="30" Click="Button_Click" Grid.Row="1" HorizontalAlignment="Left" Grid.Column="1" Margin="0,8,0,7"/>
<StackPanel Grid.Column="1" Grid.Row="2" VerticalAlignment="Bottom" HorizontalAlignment="Left" Orientation="Horizontal" Height="27" >
<Label Content="JSON input" VerticalAlignment="Bottom" HorizontalAlignment="Left" Margin="0,1" />
<ComboBox Width="240" Margin="50,3,3,3" SelectionChanged="ComboBox_SelectionChanged" VerticalAlignment="Bottom">
<ComboBoxItem Name="ex1">Example 1</ComboBoxItem>
<ComboBoxItem Name="ex2">Example 2</ComboBoxItem>
<ComboBoxItem Name="ex3">Example 3</ComboBoxItem>
<ComboBoxItem Name="ex4">Example 4</ComboBoxItem>
</ComboBox>
</StackPanel>
<TextBox x:Name="JsonIn" HorizontalScrollBarVisibility="Auto" VerticalScrollBarVisibility="Auto" Grid.Column="1" Grid.Row="3"/>
<GridSplitter Grid.Column="2" Grid.RowSpan="999" HorizontalAlignment="Stretch" Width="10" Background="Transparent" ResizeBehavior="PreviousAndNext"/>
<GridSplitter Grid.Row="4" Grid.ColumnSpan="999" VerticalAlignment="Top" HorizontalAlignment="Stretch" Height="10" Background="Transparent" ResizeBehavior="PreviousAndNext" />
<Label Content="JSON output" Grid.Column="3" Grid.Row="2" VerticalAlignment="Bottom" />
<TextBox x:Name="JsonOut" VerticalAlignment="Stretch" HorizontalScrollBarVisibility="Auto" VerticalScrollBarVisibility="Auto" Grid.Column="3" Grid.Row="3" />
<Label Content="C# source code output" Grid.Column="3" Grid.Row="4" VerticalAlignment="Bottom" />
<TextBox x:Name="CSharpSourceCode" HorizontalScrollBarVisibility="Auto" VerticalScrollBarVisibility="Auto" Grid.Column="3" Grid.Row="5" />
<Label Content="LINQ input" Grid.Column="1" Grid.Row="4" VerticalAlignment="Bottom"/>
<TextBox x:Name="LinqIn" HorizontalScrollBarVisibility="Auto" VerticalScrollBarVisibility="Auto" Grid.Column="1" Grid.Row="5" />
</Grid>
</Window>
using Microsoft.CSharp;
using System;
using System.CodeDom.Compiler;
using System.Collections.Generic;
using System.Reflection;
using System.Text;
using System.Windows;
using System.Windows.Controls;
namespace LinqJson {
public partial class MainWindow : Window {
public MainWindow() {
InitializeComponent();
} // constructor
private void Button_Click(object sender, RoutedEventArgs e) {
// --------------------------------------------------
// get the JSON input
// --------------------------------------------------
string lJsonIn = JsonIn.Text;
// --------------------------------------------------
// construct the C# source code (class hierarchy)
// --------------------------------------------------
Converter lConverter = new Converter();
Dictionary<string, object> lTree = lConverter.JsonToDictionary(lJsonIn);
lConverter.DictionariesToClasses("root", 0, lTree);
var lStringBuilder = new StringBuilder();
lConverter.BuildClasses(lStringBuilder);
string lCSharpSourceCode = lConverter.GetUsings() + lStringBuilder.ToString();
// --------------------------------------------------
// add the LINQ command to the source code
// --------------------------------------------------
string lLinq = LinqIn.Text;
string lEntryPoint = "\n\n";
lEntryPoint += "public class baseClass {\n";
lEntryPoint += " public static object executeLinq(string xJson) {\n";
lEntryPoint += " xJson = xJson.Trim();";
lEntryPoint += " if (xJson[0] == '[') xJson = \"{home: \" + xJson + \"}\";";
lEntryPoint += " var lSerializer = new JavaScriptSerializer();\n";
lEntryPoint += " var root = lSerializer.Deserialize<Class_root>(xJson);\n";
lEntryPoint += " var lResult = " + lLinq.Replace("\n", "\n ") + ";\n";
lEntryPoint += " return lSerializer.Serialize(lResult);\n";
lEntryPoint += " }\n";
lEntryPoint += "}\n";
lCSharpSourceCode += lEntryPoint;
// --------------------------------------------------
// display the source code
// --------------------------------------------------
CSharpSourceCode.Text = lCSharpSourceCode;
// --------------------------------------------------
// compile the source code
// --------------------------------------------------
var lProviderOptions = new Dictionary<string, string>();
lProviderOptions.Add("CompilerVersion", "v4.0");
var lCSharpCodeProvider = new CSharpCodeProvider(lProviderOptions);
var lCompilerParameters = new CompilerParameters();
lCompilerParameters.ReferencedAssemblies.Add("System.dll");
lCompilerParameters.ReferencedAssemblies.Add("System.Core.dll");
lCompilerParameters.ReferencedAssemblies.Add("System.Data.Linq.dll");
lCompilerParameters.ReferencedAssemblies.Add("System.Threading.dll");
lCompilerParameters.ReferencedAssemblies.Add("System.Web.Extensions.dll");
lCompilerParameters.ReferencedAssemblies.Add("System.Xml.Linq.dll");
lCompilerParameters.GenerateInMemory = true;
lCompilerParameters.GenerateExecutable = false; // not required, we don't have a Main() method
lCompilerParameters.IncludeDebugInformation = true;
var lCompilerResults = lCSharpCodeProvider.CompileAssemblyFromSource(lCompilerParameters, lCSharpSourceCode);
if (lCompilerResults.Errors.HasErrors) {
var lError = new StringBuilder();
foreach (CompilerError lCompilerError in lCompilerResults.Errors) {
lError.AppendLine(lCompilerError.ErrorNumber + " => " + lCompilerError.ErrorText + Environment.NewLine);
}
JsonOut.TextWrapping = TextWrapping.Wrap;
JsonOut.Text = lError.ToString();
return;
}
JsonOut.TextWrapping = TextWrapping.NoWrap;
// --------------------------------------------------
// execute the compiled code
// --------------------------------------------------
Assembly lAssembly = lCompilerResults.CompiledAssembly;
Type lProgram = lAssembly.GetType("baseClass");
MethodInfo lMethod = lProgram.GetMethod("executeLinq");
object lQueryResult = lMethod.Invoke(null, new object[] { lJsonIn }); // returns a JSON string object
// --------------------------------------------------
// rudimentary JSON output formatting
// --------------------------------------------------
string lJsonOut = lQueryResult.ToString();
lJsonOut = lJsonOut.Replace(",", ",\n");
lJsonOut = lJsonOut.Replace(",{", ",{\n");
lJsonOut = lJsonOut.Replace("]", "]\n");
JsonOut.Text = lJsonOut;
} //
private void ComboBox_SelectionChanged(object xSender, SelectionChangedEventArgs e) {
var lComboBox = xSender as ComboBox;
switch (lComboBox.SelectedIndex) {
case 0:
JsonIn.Text = "{\n \"number\": 108.541,\n \"datetime\": \"1975-03-13T10:30:00\" ,\n \"serialnumber\": \"SN1234\",\n \"more\": {\n \"field1\": 123,\n \"field2\": \"hello\"\n },\n \"array\": [\n {\"x\": 2.0},\n {\"x\": 3.0},\n {\"x\": 4.0}\n ]\n}";
LinqIn.Text = "from a in root.array\nwhere a.x > 2.0M\nselect a";
break;
case 1:
JsonIn.Text = "[ 1, 9, 5, 7, 1, 4 ]";
LinqIn.Text = "from a in root.home\nwhere ((a == 4) || (a == 1))\nselect a";
break;
case 2:
JsonIn.Text = "{myLuckyNumbers: [ 1, 9, 5, 26, 7, 1, 4 ]}";
LinqIn.Text = "from x in root.myLuckyNumbers\nwhere x % 2 == 0\nselect new { simple=x, square=x*x, text=\"Field\"+x.ToString(), classInClass=new {veryOdd=x/7.0, lol=\":D\"}}";
break;
case 3:
string s;
s = "{\"mainMenu\": {\n";
s += " \"info\": \"Great tool.\",\n";
s += " \"value\": 100.00,\n";
s += " \"menu\": {\n";
s += " \"subMenu\": [\n";
s += " {\"text\": \"New\", \"onclick\": \"NewObject()\"},\n";
s += " {\"text\": \"Open\", \"onclick\": \"Load()\"},\n";
s += " {\"text\": \"Close\", \"onclick\": \"ByeBye()\"},\n";
s += " {\"text\": \"NNN\", \"onclick\": \"Useless()\"}\n";
s += " ]}\n";
s += "}}\n";
JsonIn.Text = s;
LinqIn.Text = "root.mainMenu.menu.subMenu.Where(t => t.text.StartsWith(\"N\")).Last();";
break;
default:
break;
}
} //
} // class
} // namespace
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Web.Script.Serialization; // requires reference to System.Web.Extensions, used by the JavaScriptSerializer
namespace LinqJson {
public class Converter {
private Dictionary<string, object> _Classes = new Dictionary<string, object>();
public string GetUsings() {
string s;
s = "using System;\n";
s += "using System.Collections.Generic;\n";
s += "using System.Linq;\nusing System.Text;\n";
s += "using System.Threading.Tasks;\n";
s += "using System.Collections;\n";
s += "using System.Web.Script.Serialization;\n\n";
return s;
} //
public Dictionary<string, object> JsonToDictionary(string xJson) {
xJson = xJson.Trim();
if (xJson[0] == '[') xJson = "{home: " + xJson + "}"; // nameless arrays cannot be converted to dictionaries
var lJavaScriptSerializer = new JavaScriptSerializer();
try { return lJavaScriptSerializer.Deserialize<Dictionary<string, object>>(xJson); }
catch (Exception) { return null; }
} //
public void BuildClasses(StringBuilder xStringBuilder) {
foreach (var lClass in _Classes) {
Dictionary<string, object> lMembers = lClass.Value as Dictionary<string, object>;
if (lMembers == null) continue;
if (lMembers.Count <= 0) continue;
xStringBuilder.Append("public class Class_");
xStringBuilder.Append(lClass.Key);
xStringBuilder.AppendLine(" {");
foreach (var lMember in lMembers) {
object lValue = lMember.Value;
string lKey = lMember.Key;
Type lType = (lValue == null) ? typeof(object) : lMember.Value.GetType();
xStringBuilder.Append(new String(' ', 2));
xStringBuilder.Append("public ");
if (lType.IsValueType || (lValue is string)) {
xStringBuilder.Append(lType.Name);
xStringBuilder.Append(" ");
xStringBuilder.Append(lKey);
xStringBuilder.AppendLine(";");
}
else if (lValue is Dictionary<string, object>) {
xStringBuilder.Append("Class_");
xStringBuilder.Append(lKey);
xStringBuilder.Append(" ");
xStringBuilder.Append(lKey);
xStringBuilder.AppendLine(";");
}
else if (lValue is ArrayList) {
ArrayList lArrayList = lValue as ArrayList;
var lMemberType = ArrayListType(lArrayList); // differentiate between the contents of the list
if (lMemberType.IsValueType || (lMemberType.Name == "String")) {
//xStringBuilder.Append(lMemberType.Name.Replace("`2")); // Dictionaries use name "Dictionary`2"
xStringBuilder.Append(" List<"); xStringBuilder.Append(lMemberType.Name); xStringBuilder.Append("> ");
xStringBuilder.Append(lKey);
xStringBuilder.AppendLine(";");
}
else {
// a class
xStringBuilder.Append("List<Class_" + lKey + "> ");
xStringBuilder.Append(lKey);
xStringBuilder.AppendLine(";");
}
}
}
xStringBuilder.AppendLine(" }");
xStringBuilder.AppendLine();
}
} //
public void DictionariesToClasses(string xRootName, int xIndent, object xObject) {
if (xObject == null) return;
Type lType = xObject.GetType();
if (lType.IsValueType || (xObject is string)) return;
var lDictionary = xObject as Dictionary<string, object>;
if (lDictionary != null) {
object lObj;
if (!_Classes.TryGetValue(xRootName, out lObj)) {
_Classes.Add(xRootName, lDictionary);
}
else {
foreach (var lKeyValuePair in lDictionary) {
// This is a weakness of the program.
// Two JSON objects must not use the same name.
// We would have to compare the JSON objects and determine wether to create multiple or just one C# class.
_Classes[lKeyValuePair.Key] = lKeyValuePair.Value; // object type will be overridden !!!!!!
}
return;
}
foreach (var lKeyValuePair in lDictionary) {
DictionariesToClasses(lKeyValuePair.Key, xIndent, lKeyValuePair.Value);
}
return;
}
var lArrayList = xObject as ArrayList;
if (lArrayList != null) {
object lObj;
if (!_Classes.TryGetValue(xRootName, out lObj)) {
lDictionary = new Dictionary<string, object>();
_Classes.Add(xRootName, lDictionary);
}
else lDictionary = lObj as Dictionary<string, object>;
var lElementType = ArrayListType(lArrayList);
if (lElementType == typeof(Dictionary<string, object>)) {
var lList = lArrayList.Cast<Dictionary<string, object>>().ToList(); // upgrade our object to have stronger types
foreach (var lDict in lList) {
foreach (var lKeyValuePair in lDict) {
lDictionary[lKeyValuePair.Key] = lKeyValuePair.Value; // object type will be overridden !!!!!!
}
}
foreach (var lKeyValuePair in lDictionary) {
DictionariesToClasses(lKeyValuePair.Key, xIndent, lKeyValuePair.Value);
}
}
return;
}
} //
private void Append(StringBuilder xStringbuilder, int xIndent, string xString) {
xStringbuilder.Append(new String(' ', xIndent));
xStringbuilder.Append(xString);
} //
private void Newline(StringBuilder xStringbuilder) {
xStringbuilder.AppendLine();
} //
private Type ArrayListType(ArrayList xArrayList) {
var lTypes = new Dictionary<string, Type>();
Type lType;
string lTypeName;
foreach (object o in xArrayList) {
if (o == null) lType = typeof(object);
else lType = o.GetType();
lTypeName = lType.Name;
if (!lTypes.ContainsKey(lTypeName)) lTypes.Add(lTypeName, lType);
}
if (lTypes.Count == 1) return lTypes.Values.First(); // distinct
return typeof(object);
} //
} // class
} // namespace
migration C#, Java, C++ (day 5), lambda and TBB
Phew, this was a terrible day. I wanted to do far more, but I had realized that there is no proper native threading in the C++ standard libraries. Back in the 1990s threading was a minor issue. We only had one CPU and FPU. We were concentrating more on the processes and devices at that time.
I had already mentioned that I wanted to avoid Boost and Qt. The best option I found was TBB, which seems to support multicore-processors very well. I will include TBB in my migration project from now on. Today, there is only one TBB example. It took me a lot of time to get the Intel software installed on my PC.
Finally, after all these years … we can solve problems with PCs that we would not have without them 😉
TBB covers only a small fraction today. This post is mainly about lambda functions.
In C++ you can find a lot of weird stuff that has been solved in a much better way in C#. As a hardcore C# programmer C++ might feel medieval sometimes. Rome was built bit by bit. Hence its streets are chaotic. C# maybe is like Chicago. When the city was built, people had time to think about the infrastructure before they started digging arbitrary holes in the ground.
Let’s see what my thoughts will be on day 15.
Please also check my post Lambda expressions (advanced). I mentioned that the C# compiler takes care of the local variable life-cycles, when lambda expressions use local variables, which would become invalid by the time these expressions try to use them. Remember, these local variables are not created inside the lambda expressions themselves.
Today’s prerequisite for C++
#include <iostream> #include <vector> #include <ctime> #include <thread> #include "tbb/tbb.h" #include "tbb/parallel_for.h" using namespace std;
Lambda
public void DoSomething() {
Console.WriteLine(Thread.CurrentThread.ManagedThreadId + " ");
Thread.Sleep(200);
} //
public void Lambdas() {
// functor
// There are no functors in C#.
// Action or Func delegates are the closest approaches.
Action lAction = DoSomething;
lAction();
// lambda expression
Func<int, int> func2 = (int x) => { return ++x; };
int r2 = func2(4);
Console.WriteLine(r2); // 5
// lambda expression capturing a local variable
// the compiler takes care of the lifetime of variable i
int i = 5;
Func<int, int> func3 = (int x) => { return x + i; };
int r3 = func3(4);
Console.WriteLine(r3); // 9
// lambda expression capturing any local variable by reference
List<int> lList = new List<int>() { 1, 2, 3, 5, 7, 11, 13, 17, 19, 23 };
int lSum1 = 0;
lList.ForEach((int q) => { lSum1 += q; });
Console.WriteLine(lSum1); // 101
// lambda expression with a specific return type
Func<int, bool> func6 = (int x) => { x++; return true; };
bool r6 = func6(4);
Console.WriteLine(r6); // True
} //
class Class1 implements Runnable {
@Override
public void run() {
try {
System.out.println(Thread.currentThread().getId() + " ");
Thread.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
} // class
public static void main(String[] args) {
Class1 c1 = new Class1();
c1.run();
// lambda expression
Function<Integer, Integer> func2 = i -> { return ++i; };
Integer r2 = func2.apply(4);
System.out.println(r2); // 5
// lambda expression capturing a local variable
// the compiler takes care of the lifetime of variable i
Integer i = 5;
Function<Integer, Integer> func3 = x -> { return x + i; };
int r3 = func3.apply(4);
System.out.println(r3); // 9
// lambda expression capturing any local variable by reference
java.util.ArrayList<Integer> lList = new java.util.ArrayList<>(java.util.Arrays.asList(new Integer[]{1, 2, 3, 5, 7, 11, 13, 17, 19, 23}));
Integer lSum1 = 0;
lList.forEach((Integer q) -> {
//lSum1 += q; compiler complains, because it has no write access to lSum1
});
// thus we have to use:
lSum1 = lList.stream().map((q) -> q).reduce(lSum1, Integer::sum);
System.out.println(lSum1); // 101
// lambda expression with a specific return type
Function<Integer, Boolean> func6 = x -> {
x++;
return true;
};
boolean r6 = func6.apply(4);
System.out.println(r6); // true
} //
class func1 {
public:
int operator ()(int x) { return ++x; }
};
void Lambdas() {
// functor
int r1 = func1()(3);
cout << r1 << endl; // 4
// lambda expression
auto func2 = [](int x) { return ++x; };
int r2 = func2(4);
cout << r2 << endl; // 5
// lambda expression capturing a local variable
int i = 5;
auto func3 = [i](int x) { return x + i; };
int r3 = func3(4);
cout << r3 << endl; // 9
// lambda expression capturing any local variable
auto func4 = [=](int x) { return x + i + r2 * r1; }; // 1 + 5 + (5 * 4)
int r4 = func4(1);
cout << r4 << endl; // 26
// lambda expression capturing a local variable by reference
vector<int> v1{ 1, 2, 3, 5, 7, 11, 13, 17, 19, 23 };
int lSum1 = 0;
for (int q : v1) { [&lSum1](int x) {lSum1 += x; } (q); }
cout << lSum1 << endl; // 101
// lambda expression capturing any local variable by reference
vector<int> v2{ 1, 2, 3, 5, 7, 11, 13, 17, 19, 23 };
int lSum2 = 0;
int lProduct2 = 1;
for (int q : v2) { [&](int x) {lSum2 += x; lProduct2 *= x; }(q); }
cout << lSum2 << " " << lProduct2 << endl; // 101 223092870
// lambda expression capturing any local variable by value
// The external variables can be assigned, but they are not referenced,
// because the lambda expression is only working with local copies.
// This can be achieved by using the keyword "mutable".
// By default lambdas are not mutable.
int lSum3 = 0;
int lProduct3 = 1;
auto func5 = [=](int x) mutable {lSum3 += x; lProduct3 *= x; return x; };
int r5 = func5(1);
cout << lSum3 << " " << lProduct3 << " " << r5 << endl; // 0 1 1
// lambda expression with a specific return type
auto func6 = [](int x) -> bool { x++; return true; };
bool b6 = func6(6);
cout << (b6 ? "true" : "false") << endl; // true
} //
TBB
public void TBB() {
const int lSize = 100;
Stopwatch lStopwatch = new Stopwatch();
lStopwatch.Start();
for (int i = 0; i < lSize; i++) DoSomething();
lStopwatch.Stop();
Console.WriteLine();
Console.WriteLine("time for serial loop was " + lStopwatch.ElapsedMilliseconds / 1000.0 + " seconds");
lStopwatch.Reset();
lStopwatch.Start();
Parallel.For(0, lSize, i => DoSomething());
lStopwatch.Stop();
Console.WriteLine();
Console.WriteLine("time for parallel loop was " + lStopwatch.ElapsedMilliseconds/1000.0 + " seconds");
} //
example output:
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
time for serial loop was 20.001 seconds
9 10 11 13 14 12 15 17 16 9 10 11 13 14 12 15 17 16 9 10 11 13 14 12 15 17 16 9
10 11 13 14 12 15 17 16 9 10 11 13 14 12 15 17 16 9 10 11 13 14 12 15 17 16 9 10
11 13 14 12 15 17 16 9 10 11 13 14 12 15 16 9 10 11 13 14 12 15 16 9 10 11 13 1
4 12 15 9 10 11 13 14 12 15 9 10 11 13 14 12 15
time for parallel loop was 2.408 seconds
public final void TBB() {
Class1 lDoSomething = new Class1();
final int lSize = 100;
long lStopwatch = System.currentTimeMillis();
for (int i = 0; i < lSize; i++) {
lDoSomething.run();
}
long lDiff = System.currentTimeMillis() - lStopwatch;
System.out.println();
System.out.println("time for serial loop was " + lDiff / 1000 + " seconds");
lStopwatch = System.currentTimeMillis();
ArrayList<Class1> lList = new ArrayList<>();
for (int i = 0; i < lSize; i++) lList.add(lDoSomething);
lList.parallelStream().forEach(x -> x.run());
lDiff = System.currentTimeMillis() - lStopwatch;
System.out.println();
System.out.println("time for parallel loop was " + lDiff / 1000.0 + " seconds");
} //
example output:
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
time for serial loop was 20 seconds
12 13 14 1 16 11 15 17 14 13 12 1 16 11 15 17 13 12 14 1 16 17 11 15 13 14 12 16 1 11 15 17 13 14 12 16 1 11 15 17 12 13 14 1 16 11 17 15 14 12 13 16 1 15 17 11 12 13 14 16 1 17 11 15 14 13 12 1 16 11 15 17 13 14 12 1 16 15 17 11 14 12 13 16 1 11 17 15 14 12 13 16 1 17 11 15 12 13 1 15
time for parallel loop was 2.841 seconds
void DoSomething() {
cout << this_thread::get_id() << " ";
this_thread::sleep_for(chrono::milliseconds(200));
} //
void TBBs() {
const int lSize = 100;
time_t lTimer;
time(&lTimer);
for (int i = 0; i < lSize; i++) DoSomething();
double lSecondsA = difftime(time(nullptr), lTimer);
cout << endl << "time for serial loop was " << lSecondsA << " seconds" << endl;
time(&lTimer);
tbb::parallel_for(0, lSize, 1, [&](int i) { DoSomething(); });
double lSecondsB = difftime(time(nullptr), lTimer);
cout << endl << "time for parallel loop was " << lSecondsB << " seconds" << endl;
} //
example output:
4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 46
16 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616
4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616
4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 46
16 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616
4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616
4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 4616 46
16 4616 4616 4616 4616 4616 4616
time for serial loop was 20 seconds
4616 4068 4916 3788 2880 4168 4752 6796 4616 4916 4068 2880 3788 41
68 6796 4752 4616 4068 4916 2880 3788 4168 6796 4752 4616 4916 4068
2880 3788 4168 6796 4752 4616 4916 4068 3788 2880 4168 6796 4752
4616 4068 4916 2880 3788 4168 6796 4752 4616 4068 4916 3788 2880 41
68 6796 4752 4616 4068 4916 2880 3788 6796 4168 4752 4616 4068 4916
2880 3788 6796 4168 4752 4616 4068 4916 2880 3788 6796 4168 4752
4616 4916 4068 3788 2880 6796 4168 4752 4616 4068 4916 3788 2880 67
96 4168 4752 4616 4916 4068 3788
time for parallel loop was 2 seconds
XML (part 2, basics), embedded, XDocument, XmlDocument, XmlTextReader, XPathNavigator
We covered basic file IO for XML files yesterday. This is very useful for settings files. There are some XML files though that you want to hide from the user. I am not talking about encryption. Today we are going to embed the XML file the project itself.
We are still using yesterday’s “Walmart.xml” example. Create a folder “XmlFiles” in your Solution Explorer and drag the file into it.
Now right-click the file “Walmart.xml” and select “Properties”. Change the “Build Action” to “Embedded Resource”.
Use the following code to load the resource as an XDocument:
// since .Net 3.5
// namespace System.Xml.Linq
// xPath: namespace.folder.file.ext => here: "DemoApp.XmlFiles.Walmart.xml"
public static XDocument getAssemblyXDocument(string xPath) {
Assembly lAssembly = Assembly.GetExecutingAssembly();
using (Stream lStream = lAssembly.GetManifestResourceStream(xPath)) {
XDocument lXDocument = XDocument.Load(lStream);
return lXDocument;
}
} //
You can also load the document as an XmlDocument.
// since .Net 1.1
// namespace System.Xml
// xPath: namespace.folder.file.ext => here: "DemoApp.XmlFiles.Walmart.xml"
public static XmlDocument getAssemblyXml(string xPath) {
Assembly lAssembly = Assembly.GetExecutingAssembly();
using (Stream lStream = lAssembly.GetManifestResourceStream(xPath)) {
XmlDocument lXmlDocument = new XmlDocument();
lXmlDocument.Load(lStream);
return lXmlDocument;
}
} //
public static void LoadXmlExample() {
string lPath = "DemoApp.XmlFiles.Walmart.xml";
XmlDocument lXml = getAssemblyXml(lPath);
XDocument lXDoc = getAssemblyXDocument(lPath);
} //
Using an XmlDocument is similar to XDocument. XmlDocument does not support LINQ, which is a big disadvantage.
XmlDocument reads the entire document and initializes a tree structure, which needs a lot of memory. Processing bigger Xml files can cause issues. The tree can be accessed either by browsing through the nodes or by using XPath queries.
You can browse through documents forward/backward and carry out modifications.
XDocument (LINQ to XML) is easy to code and understand. Again the entire document is loaded into memory. To read Xml files we can use the XDocument or XElement class. Both classes support Load() methods. XElement represents an XML fragment while XDocument represents an entire XML document.
XmlTextReader is very fast and memory efficient. Unlike XmlDocument it can only read in forward direction. Searching for something specific is awkward. As the name says, there is no method to write or to modify data. And data validation is also left behind.
XPathNavigator is more complex and slower than XmlReader. XPathNavigator has to read enough data to be able to execute XPath queries. XmlDocument internally uses XPathNavigator. Using XPathNavigator directly is faster than using it via XmlDocument.
Using XmlTextReader:
public static void UseTextReader() {
string lPath = "DemoApp.XmlFiles.Walmart.xml";
Assembly lAssembly = Assembly.GetExecutingAssembly();
using (Stream lStream = lAssembly.GetManifestResourceStream(lPath)) {
using (XmlTextReader lXmlReader = new XmlTextReader(lStream)) {
while (lXmlReader.Read()) {
if (!lXmlReader.IsStartElement()) continue;
switch (lXmlReader.Name) {
case "name":
Console.WriteLine("name: " + lXmlReader.ReadString());
break;
case "price":
Console.WriteLine("price: " + lXmlReader.ReadString());
break;
case "description":
Console.WriteLine("description: " + lXmlReader.ReadString());
break;
case "somethingElse":
Console.WriteLine("somethingElse: " + lXmlReader.ReadString());
break;
default:
Console.WriteLine(Environment.NewLine + lXmlReader.Name);
break;
}
}
}
}
Console.ReadLine();
} //
example output:
Walmart
food
name: Banana
price: 1.99
description: Mexican deliciousfood
name: Rice
price: 0.79
description: the best you can getfood
name: Cornflakes
price: 3.85
description: buy some milkfood
name: Milk
price: 1.43
description: from happy cowselectronic
name: Kindle fire
price: 100
description: Amazon loves you
somethingElse: the perfect Xmas gift for your kidsfood
name: baked beans
price: 1.35
description: very British
Using XPathNavigator:
public static void UseXPathNavigator() {
string lPath = "DemoApp.XmlFiles.Walmart.xml";
Assembly lAssembly = Assembly.GetExecutingAssembly();
using (Stream lStream = lAssembly.GetManifestResourceStream(lPath)) {
XPathDocument lXPathDoc = new XPathDocument(lStream);
XPathNavigator lXNavi = lXPathDoc.CreateNavigator();
lXNavi.MoveToRoot();
lXNavi.MoveToFirstChild();
do {
if (lXNavi.MoveToFirstAttribute()) {
Console.WriteLine(lXNavi.Name + "=" + lXNavi.Value);
lXNavi.MoveToParent();
}
if (!lXNavi.MoveToFirstChild()) continue;
do {
if (!lXNavi.MoveToFirstChild()) break;
do {
Console.WriteLine(lXNavi.Name + "=" + lXNavi.Value);
} while (lXNavi.MoveToNext());
Console.WriteLine();
lXNavi.MoveToParent();
} while (lXNavi.MoveToNext());
lXNavi.MoveToParent();
} while (lXNavi.MoveToNext());
Console.ReadLine();
}
} //
example output:
name=Banana
price=1.99
description=Mexican deliciousname=Rice
price=0.79
description=the best you can getname=Cornflakes
price=3.85
description=buy some milkname=Milk
price=1.43
description=from happy cowsname=Kindle fire
price=100
description=Amazon loves you
somethingElse=the perfect Xmas gift for your kidsname=baked beans
price=1.35
description=very British
Using XPathNavigator with query:
public static void UseXPathNavigator(string xQuery) {
string lPath = "DemoApp.XmlFiles.Walmart.xml";
Assembly lAssembly = Assembly.GetExecutingAssembly();
using (Stream lStream = lAssembly.GetManifestResourceStream(lPath)) {
XPathDocument lXPathDoc = new XPathDocument(lStream);
XPathNavigator lXNavi = lXPathDoc.CreateNavigator();
XPathNodeIterator lIterator = lXNavi.Select(xQuery);
if (lIterator.Count <= 0) {
Console.WriteLine("Nothing found!");
return;
}
while (lIterator.MoveNext()) {
Console.WriteLine(lIterator.Current.Value);
}
}
} //
public static void TestQueries() {
UseXPathNavigator(@"/Walmart/electronic");
Console.WriteLine();
UseXPathNavigator(@"/Walmart/food[name='Banana']");
Console.ReadLine();
} //
example output:
Kindle fire100Amazon loves youthe perfect Xmas gift for your kidsBanana1.99Mexican delicious
My opinion is: Make use of LINQ to XML. It is very comfortable and can save you a lot of time. If you have a lot of data, then you should consider a database rather than XML files.
We will discuss XML object serialization tomorrow. Let C# save and load objects the nice way.
Reflection (part 5, professional), CodeDOM (& lambda expression tree)
Today’s post is about code generation at runtime using CodeDOM, which is the logical step before runtime code generation using Emit. The .Net framework has been built in an amazing way. We can easily neglect some flaws. Compared to Java there are no flaws at all. Jealousy needs to be deserved, sympathy is for free. Somehow Java is for free, and this is what you get.
Make sure, you are aware of the following differences. There are Expression and Statement. Please follow the links if you feel uncertain about the definitions.
Expression
An expression is a sequence of one or more operands and zero or more operators that can be evaluated to a single value, object, method, or namespace.
Statement
The actions that a program takes are expressed in statements. Common actions include declaring variables, assigning values, calling methods, looping through collections, and branching to one or another block of code, depending on a given condition. The order in which statements are executed in a program is called the flow of control or flow of execution.
A statement can consist of a single line of code that ends in a semicolon, or a series of single-line statements in a block. A statement block is enclosed in {} brackets and can contain nested blocks.
Succinct:
An Expression is a kind of mathematical or logical expression.
A Statement is a single code instruction or block of code to execute.
The first example code generates source code for C# and VB and saves the generated files on your desktop. Of course it has to be the classical “hello world” program.
CodeDOM stands for Code Document Object Model. It is located in the System.CodeDom namespace and provides methods to create code at runtime.
// what you need for the code example: using System; using System.CodeDom; using System.CodeDom.Compiler; using System.IO; using Microsoft.CSharp; using Microsoft.VisualBasic;
First of all you have to create a compile unit object CodeCompileUnit. It is the top container for namespaces, classes and members. Then add the components as follows:
public static void Test() {
CodeCompileUnit lUnit = new CodeCompileUnit();
CodeNamespace lNamespace = new CodeNamespace("MyNamespace");
lNamespace.Imports.Add(new CodeNamespaceImport("System"));
lNamespace.Imports.Add(new CodeNamespaceImport("System.IO")); // not used, just for demo
lUnit.Namespaces.Add(lNamespace);
CodeTypeDeclaration lClass = new CodeTypeDeclaration("MyClass");
CodeMethodInvokeExpression lExpression = new CodeMethodInvokeExpression(new CodeTypeReferenceExpression("Console"), "WriteLine", new CodePrimitiveExpression("hello world !"));
lNamespace.Types.Add(lClass);
// write Main entry point method
CodeEntryPointMethod lMain = new CodeEntryPointMethod();
lMain.Statements.Add(lExpression);
lClass.Members.Add(lMain);
// write another method
CodeMemberMethod lMethod = new CodeMemberMethod();
lMethod.Name = "DoSomething";
lClass.Members.Add(lMethod);
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
CodeGeneratorOptions lOptions = new CodeGeneratorOptions();
lOptions.IndentString = " "; // or "\t";
lOptions.BlankLinesBetweenMembers = true;
// generate a C# source code file
CSharpCodeProvider lCSharpCodeProvider = new CSharpCodeProvider();
using (StreamWriter lStreamWriter = new StreamWriter(lDesktopPath + "HelloWorld.cs", false)) {
lCSharpCodeProvider.GenerateCodeFromCompileUnit(lUnit, lStreamWriter, lOptions);
}
// generate a VB source code file
VBCodeProvider lVBCodeProvider = new VBCodeProvider();
using (StreamWriter lStreamWriter = new StreamWriter(lDesktopPath + "HelloWorld.vb", false)) {
lVBCodeProvider.GenerateCodeFromCompileUnit(lUnit, lStreamWriter, lOptions);
}
} //
example output file HelloWorld.cs:
//------------------------------------------------------------------------------ // // This code was generated by a tool. // Runtime Version:4.0.30319.18408 // // Changes to this file may cause incorrect behavior and will be lost if // the code is regenerated. // //------------------------------------------------------------------------------ namespace MyNamespace { using System; using System.IO; public class MyClass { public static void Main() { Console.WriteLine("hello world !"); } private void DoSomething() { } } }
example output file HelloWorld.vb:
'------------------------------------------------------------------------------ ' ' This code was generated by a tool. ' Runtime Version:4.0.30319.18408 ' ' Changes to this file may cause incorrect behavior and will be lost if ' the code is regenerated. ' '------------------------------------------------------------------------------ Option Strict Off Option Explicit On Imports System Imports System.IO Namespace MyNamespace Public Class [MyClass] Public Shared Sub Main() Console.WriteLine("hello world !") End Sub Private Sub DoSomething() End Sub End Class End Namespace
In the above example I have added an empty method. We will make it more complex now. The method will be called with parameters from Main() and return a value to a local variable. The local variable will be stored in a field via a property.
public static void Test2() {
CodeCompileUnit lUnit = new CodeCompileUnit();
CodeNamespace lNamespace = new CodeNamespace("MyNamespace");
lNamespace.Imports.Add(new CodeNamespaceImport("System"));
lUnit.Namespaces.Add(lNamespace);
CodeTypeDeclaration lClass = new CodeTypeDeclaration("MyClass");
lClass.IsClass = true;
lClass.Attributes = MemberAttributes.Public;
lNamespace.Types.Add(lClass);
// -----------------------------------------------
// method DoSomething()
// -----------------------------------------------
CodeTypeParameter lClassAsParameter = new CodeTypeParameter(lClass.Name);
CodeTypeReference lClassReference = new CodeTypeReference(lClassAsParameter);
CodeParameterDeclarationExpression lClassExpression = new CodeParameterDeclarationExpression(lClassReference, "xClass");
CodeMemberMethod lDoSomething = new CodeMemberMethod();
lDoSomething.Attributes = MemberAttributes.Public;
lDoSomething.Comments.Add(new CodeCommentStatement("This is an example comment for our method DoSomething()."));
lDoSomething.Name = "DoSomething";
lDoSomething.ReturnType = new CodeTypeReference(typeof(double));
lDoSomething.Parameters.Add(new CodeParameterDeclarationExpression(typeof(int), "xInteger"));
lDoSomething.Parameters.Add(new CodeParameterDeclarationExpression(typeof(string), "xString"));
lDoSomething.Parameters.Add(lClassExpression);
lDoSomething.Statements.Add(new CodeMethodReturnStatement(new CodePrimitiveExpression(1.0)));
lClass.Members.Add(lDoSomething);
// -----------------------------------------------
// private field _MyField
// -----------------------------------------------
CodeMemberField lField = new CodeMemberField(typeof(long), "_MyField");
lField.Attributes = MemberAttributes.Private;
lField.InitExpression = new CodePrimitiveExpression(10);
lClass.Members.Add(lField);
// -----------------------------------------------
// property MyProperty
// -----------------------------------------------
CodeMemberProperty lProperty = new CodeMemberProperty();
lProperty.Name = "MyProperty";
lProperty.Attributes = MemberAttributes.Public;
lProperty.Type = new CodeTypeReference(typeof(long));
lProperty.GetStatements.Add(new CodeMethodReturnStatement(new CodePrimitiveExpression(359)));
lProperty.SetStatements.Add(new CodeAssignStatement(new CodeFieldReferenceExpression(new CodeThisReferenceExpression(), lField.Name), new CodePropertySetValueReferenceExpression()));
lClass.Members.Add(lProperty);
// -----------------------------------------------
// Main()
// -----------------------------------------------
CodeEntryPointMethod lMain = new CodeEntryPointMethod();
lClass.Members.Add(lMain);
// local double variable d
CodeVariableReferenceExpression lLocalVariable_d = new CodeVariableReferenceExpression("d");
CodeVariableDeclarationStatement lLocalVariableStatement_d = new CodeVariableDeclarationStatement(typeof(double), lLocalVariable_d.VariableName);
lMain.Statements.Add(lLocalVariableStatement_d);
// local integer variable i
CodeVariableReferenceExpression lLocalVariable_i = new CodeVariableReferenceExpression("i");
CodeVariableDeclarationStatement lLocalVariableStatement_i = new CodeVariableDeclarationStatement(typeof(int), lLocalVariable_i.VariableName);
lLocalVariableStatement_i.InitExpression = new CodePrimitiveExpression(0);
lMain.Statements.Add(lLocalVariableStatement_i);
// local class variable MyClass
CodeVariableReferenceExpression lLocalVariable_Class = new CodeVariableReferenceExpression("lClass");
CodeVariableDeclarationStatement lLocalVariableStatement_Class = new CodeVariableDeclarationStatement(lClassReference, lLocalVariable_Class.VariableName);
lLocalVariableStatement_Class.InitExpression = new CodeObjectCreateExpression(lClass.Name, new CodeExpression[] { });
lMain.Statements.Add(lLocalVariableStatement_Class);
// DoSomething() method call
CodeMethodInvokeExpression lDoSomethingExpression = new CodeMethodInvokeExpression(lLocalVariable_Class, lDoSomething.Name, new CodeExpression[] { lLocalVariable_i, new CodePrimitiveExpression("hello"), lLocalVariable_Class});
CodeAssignStatement lAssignment = new CodeAssignStatement(lLocalVariable_d, lDoSomethingExpression);
lMain.Statements.Add(lAssignment);
//lMain.Statements.Add(lDoSomethingExpression); // without assignment
// cast the "double" result to type "long" and write it to field _MyField via property MyProperty
CodePropertyReferenceExpression lPropertyExpression = new CodePropertyReferenceExpression(lLocalVariable_Class, lProperty.Name);
CodeCastExpression lCastExpression = new CodeCastExpression(typeof(long), lLocalVariable_d);
lAssignment = new CodeAssignStatement(lPropertyExpression, lCastExpression);
lMain.Statements.Add(lAssignment);
// -----------------------------------------------
// create source code
// -----------------------------------------------
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
CodeGeneratorOptions lOptions = new CodeGeneratorOptions();
lOptions.IndentString = " "; // or "\t";
lOptions.BlankLinesBetweenMembers = true;
// generate a C# source code file
CSharpCodeProvider lCSharpCodeProvider = new CSharpCodeProvider();
using (StreamWriter lStreamWriter = new StreamWriter(lDesktopPath + "MoreComplex.cs", false)) {
lCSharpCodeProvider.GenerateCodeFromCompileUnit(lUnit, lStreamWriter, lOptions);
}
} //
example output:
//------------------------------------------------------------------------------ // // This code was generated by a tool. // Runtime Version:4.0.30319.18408 // // Changes to this file may cause incorrect behavior and will be lost if // the code is regenerated. // //------------------------------------------------------------------------------ namespace MyNamespace { using System; public class MyClass { private long _MyField = 10; public virtual long MyProperty { get { return 359; } set { this._MyField = value; } } // This is an example comment for our method DoSomething(). public virtual double DoSomething(int xInteger, string xString, MyClass xClass) { return 1D; } public static void Main() { double d; int i = 0; MyClass lClass = new MyClass(); d = lClass.DoSomething(i, "hello", lClass); lClass.MyProperty = ((long)(d)); } } }
Is your head burning already? This was quite complex. As usual there are easier ways for lazy people. I do not recommend any shortcuts. It undermines the idea of flexibility.
public static void Test3() {
CodeCompileUnit lUnit = new CodeCompileUnit();
CodeNamespace lNamespace = new CodeNamespace("MyNamespace");
lNamespace.Imports.Add(new CodeNamespaceImport("System"));
lUnit.Namespaces.Add(lNamespace);
CodeTypeDeclaration lClass = new CodeTypeDeclaration("MyClass");
lNamespace.Types.Add(lClass);
// write Main entry point method
CodeEntryPointMethod lMain = new CodeEntryPointMethod();
lMain.Statements.Add(new CodeSnippetExpression("Console.WriteLine(\"hello world !\")"));
lClass.Members.Add(lMain);
string lDesktopPath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory) + @"\";
CodeGeneratorOptions lOptions = new CodeGeneratorOptions();
lOptions.IndentString = " "; // or "\t";
lOptions.BlankLinesBetweenMembers = true;
// generate a C# source code file
CSharpCodeProvider lCSharpCodeProvider = new CSharpCodeProvider();
using (StreamWriter lStreamWriter = new StreamWriter(lDesktopPath + "HelloWorldForLazyPeople.cs", false)) {
lCSharpCodeProvider.GenerateCodeFromCompileUnit(lUnit, lStreamWriter, lOptions);
}
} //
We created source code in the runtime environment. How can we execute it?
The next example compiles a class into an assembly and then executes the Main() method, which is automatically defined as the entry point. This happens in memory. If you want to write the executable code to a disk, then set GenerateInMemory to false and define a proper OutputAssembly path in the compiler parameters (variable lParameters). Use CompileAssemblyFromFile() instead of CompileAssemblyFromSource(). Btw. CompileAssemblyFromDom() would accept CodeCompileUnits (eg. used in Test2) as input data. Therefore you don’t have to generate a source code file to compile your code structure.
private static string[] _MyCode = new string[] {
"using System;" +
"public class MyClass {" +
"public static void Main() { Console.WriteLine(\"hello world !\"); }" +
"}" };
public static void Test4() {
CSharpCodeProvider lCodeProvider= new CSharpCodeProvider();
CompilerParameters lParameters = new CompilerParameters();
//lParameters.ReferencedAssemblies.Add("System.dll");
lParameters.GenerateInMemory = true;
lParameters.GenerateExecutable = true;
lParameters.OutputAssembly = "HelloWorld";
CompilerResults lCompilerResults = lCodeProvider.CompileAssemblyFromSource(lParameters, _MyCode);
if (lCompilerResults.Errors.Count > 0) {
foreach (CompilerError lError in lCompilerResults.Errors) Console.WriteLine(lError.ToString());
}
else {
Console.WriteLine("Compiled succesfully " + lCompilerResults.PathToAssembly);
Console.WriteLine("Executing code now");
MethodInfo m = lCompilerResults.CompiledAssembly.EntryPoint;
m.Invoke(null, new object[] {});
}
Console.ReadLine();
} //
example output:
Compiled succesfully
Executing code now
hello world !
Lambda expressions are using expression trees. The data-structure is like a tree. We are not necessarily talking about code here. Lambda expressions are not source code in the classical sense, because compilers translate expressions into code. In fact you don’t know the result, all you need to do is describing the code in an expression tree. For instance LINQ can interact between your code and a database. It can generate SQL queries. And these have nothing to do with the C# source code.
Let’s use an expression tree to generate another “Hello World” example at runtime. For this we need to implement the System.Linq.Expressions namespace. This is not really a part of reflection. I touch on the topic here to have it mentioned before we continue with Emit, which is far more on spot.
public static void Test5() {
Expression A = Expression.Call(null, typeof(Console).GetMethod("WriteLine", new Type[] { typeof(String) }), Expression.Constant("hello world !"));
Expression B = Expression.Call(null, typeof(Console).GetMethod("ReadLine", new Type[] { }), null);
Expression[] Expressions = new Expression[] { A, B };
BlockExpression lBlockExpression = Expression.Block(Expressions);
Action lAction = Expression.Lambda<Action>(lBlockExpression).Compile();
lAction();
} //
Let’s build a simple calculator now. Imagine someone enters a mathematical formula in a TextBox and wants to calculate it. I have seen so many solutions on the internet and I can tell you for sure they are all far too long! In the following code you just need to add some “nice features” like pre-parsing to allow more user flexibility. The entire Math library is yours! May the 4th be with you 😉
public static void Test6() {
string lTextBoxText = "Math.Floor(5.0 * (3.0 + 7.0) / 9.0 * Math.Exp(3.0))";
string[] lCode = new string[] {
"using System;" +
"public class MyClass {" +
"public static double Calc() { return (double)" + lTextBoxText + "; }" +
"}" };
CSharpCodeProvider lCodeProvider = new CSharpCodeProvider();
CompilerParameters lParameters = new CompilerParameters();
lParameters.GenerateInMemory = true;
lParameters.GenerateExecutable = false;
lParameters.OutputAssembly = "CalcAssembly";
CompilerResults lCompilerResults = lCodeProvider.CompileAssemblyFromSource(lParameters, lCode);
MethodInfo m = lCompilerResults.CompiledAssembly.GetType("MyClass").GetMethod("Calc");
double d = (double)m.Invoke(null, null);
Console.WriteLine("calculation result is " + d);
Console.WriteLine("cross check result is " + (Math.Floor(5.0 * (3.0 + 7.0) / 9.0 * Math.Exp(3.0))));
Console.ReadLine();
} //
example output:
calculation result is 111
cross check result is 111
Do you remember the post about extension methods? Unfortunately we cannot write any extension method for static classes like the .Net framework Math class. So I am going to use the string class instead. The calculator looks a little bit nicer then.
public static class MathExtension {
public static double Formula(this string xFormula) {
string[] lCode = new string[] {
"using System;" +
"public class MyClass {" +
"public static double Calc() { return (double)" + xFormula + "; }" +
"}" };
CSharpCodeProvider lCodeProvider = new CSharpCodeProvider();
CompilerParameters lParameters = new CompilerParameters();
lParameters.GenerateInMemory = true;
lParameters.GenerateExecutable = false;
lParameters.OutputAssembly = "CalcAssembly";
CompilerResults lCompilerResults = lCodeProvider.CompileAssemblyFromSource(lParameters, lCode);
MethodInfo m = lCompilerResults.CompiledAssembly.GetType("MyClass").GetMethod("Calc");
return (double)m.Invoke(null, null);
} //
} //
// how to call the extension method
public static void Test7() {
string lTextBoxText = "Math.Floor(5.0 * (3.0 + 7.0) / 9.0 * Math.Exp(3.0))";
double d = lTextBoxText.Formula();
Console.WriteLine(d);
Console.ReadLine();
} //
Epilogue
There are some stock market trading applications, which allow users to add their own algorithmic trading strategies. Wouldn’t it be great to enter your code in C# and use the .Net framework instead of reinventing the wheel and come up with your own scripting language that will never reach any widely recognized state of the art?
Events (part 3, advanced)
Events: Let’s go multithreading! We want the crème de la crème. Well, multitasking is also fine.
The code does not need to run in a specific sequence. The required independence is given. Again, there are many ways to use multithreading. An easy approach is to start a task in each method that is called by the event.
C# does support BeginInvoke() for delegates. This method is not supported by the .NET Compact Framework though. We don’t care, because our hardcore programs are for serious applications, definitely not for mobile phone apps. Let’s see how good BeginInvoke() works. Maybe we don’t have to reinvent the wheel.
BeginInvoke() initiates asynchronous calls, it returns immediately and provides the IAsyncResult, which can be used to monitor the progress of the asynchronous call.
EndInvoke() retrieves the results. It blocks until the thread has completed.
You have the following options after calling BeginInvoke():
1) Call EndInvoke() to block the current thread until the call completes.
2) Obtain the WaitHandle from IAsyncResult.AsyncWaitHandle, use its WaitOne() and then call EndInvoke().
3) Poll the IAsyncResult to check the current state, after it has completed call EndInvoke().
4) Pass a callback to BeginInvoke(). The callback will use the ThreadPool to notify you. In the callback you have to call EndInvoke().
public event EventHandler OnChange;
public void DoSomeWork(object sender, object e) {
Thread.Sleep(2100);
Console.WriteLine("Thread " + Thread.CurrentThread.ManagedThreadId + " mission accomplished!");
} //
public void RunMyExample() {
OnChange += new EventHandler(DoSomeWork);
//OnChange += new EventHandler(DoSomeWork);
//OnChange += new EventHandler(DoSomeWork);
IAsyncResult lAsyncResult;
Console.WriteLine("Choice 1");
lAsyncResult = OnChange.BeginInvoke(this, EventArgs.Empty, null, null);
OnChange.EndInvoke(lAsyncResult);
Console.WriteLine("Choice 2");
lAsyncResult = OnChange.BeginInvoke(this, EventArgs.Empty, null, null);
lAsyncResult.AsyncWaitHandle.WaitOne();
Console.WriteLine("Choice 3");
lAsyncResult = OnChange.BeginInvoke(this, EventArgs.Empty, null, null);
while (!lAsyncResult.IsCompleted) {
Thread.Sleep(500);
Console.WriteLine("work still not completed :(");
}
Console.WriteLine("Choice 4");
OnChange.BeginInvoke(this, EventArgs.Empty, (xAsyncResult) => {
Console.WriteLine("callback running");
OnChange.EndInvoke(xAsyncResult);
}, null);
Console.WriteLine("press return to exit the program");
Console.ReadLine();
}//
example output:
Choice 1
Thread 6 mission accomplished!
Choice 2
Thread 6 mission accomplished!
work still not completed 😦
work still not completed 😦
work still not completed 😦
work still not completed 😦
Thread 10 mission accomplished!
work still not completed 😦
press return to exit the program
Thread 6 mission accomplished!
callback running
After having lived such a nice life, we have to face a major issue with BeginInvoke(). Uncomment “//OnChange += new EventHandler(DoSomeWork);”, don’t get annoyed now!
You will get an error message saying
“The delegate must have only one target”.
Although the delegate class can deal with multiple targets, asynchronous calls accept just one target.
So let’s try something else.
public event EventHandler OnChange;
public void DoSomeWork(object sender, object e) {
Thread.Sleep(2100);
Console.WriteLine("Thread " + Thread.CurrentThread.ManagedThreadId + " mission accomplished!");
} //
public void RunMyExample() {
OnChange += new EventHandler(DoSomeWork);
OnChange += new EventHandler((sender, e) => { throw new Exception("something went wrong"); });
OnChange += new EventHandler(DoSomeWork);
EventHandler lOnChange = OnChange;
if (lOnChange == null) return; // just to demonstrate the proper way to call events, not needed in this example
foreach (EventHandler d in lOnChange.GetInvocationList()) {
Task lTask = Task.Factory.StartNew(() => d(this, EventArgs.Empty));
lTask.ContinueWith((i) => { Console.WriteLine("Task canceled"); }, TaskContinuationOptions.OnlyOnCanceled);
lTask.ContinueWith((i) => { Console.WriteLine("Task faulted"); }, TaskContinuationOptions.OnlyOnFaulted);
lTask.ContinueWith((i) => { Console.WriteLine("Task completion"); }, TaskContinuationOptions.OnlyOnRanToCompletion);
}
Console.WriteLine("press return to exit the program");
Console.ReadLine();
}//
It seems that Microsoft has a serious bug here.This code does not execute properly each time. I guess it has to do with the asynchronous behaviour of StartNew(). It sometimes calls the wrong method DoSomeWork() three times and does not raise the exception.
It seems foreach overrides variable “d” before it is inserted in StartNew(). With a little tweak we can avoid this bug. We simply assign “d” to a new local variable. That way we have unique copies of “d”. Weird stuff, but that is the life of a coder sometimes.
public event EventHandler OnChange;
public void DoSomeWork(object sender, object e) {
Thread.Sleep(2100);
Console.WriteLine("Thread " + Thread.CurrentThread.ManagedThreadId + " mission accomplished!");
} //
public void RunMyExample() {
OnChange += new EventHandler(DoSomeWork);
OnChange += new EventHandler((sender, e) => { throw new Exception("something went wrong"); });
OnChange += new EventHandler(DoSomeWork);
EventHandler lOnChange = OnChange;
if (lOnChange == null) return; // just to demonstrate the proper way to call events, not needed in this example
foreach (EventHandler d in lOnChange.GetInvocationList()) {
EventHandler e = d;
Task lTask = Task.Factory.StartNew(() => e(this, EventArgs.Empty));
lTask.ContinueWith((i) => { Console.WriteLine("Task canceled"); }, TaskContinuationOptions.OnlyOnCanceled);
lTask.ContinueWith((i) => { Console.WriteLine("Task faulted"); }, TaskContinuationOptions.OnlyOnFaulted);
lTask.ContinueWith((i) => { Console.WriteLine("Task completion"); }, TaskContinuationOptions.OnlyOnRanToCompletion);
}
Console.WriteLine("press return to exit the program");
Console.ReadLine();
}//
Example output:
press return to exit the program
Thread 11 mission accomplished!
Thread 10 mission accomplished!
Task completion
Task faulted
Task completion
This tiny tweak worked. You just have to know the compiler bugs. I hope this little notice saves you at least 3 hours of work.
You probably remember that we faced a similar issue in my post “Exiting Tasks (advanced)” https://csharphardcoreprogramming.wordpress.com/2013/12/11/exiting-tasks/ . I would suggest to only use Task.Factory.StartNew() with caution. Maybe it only happens in conjunction with lambda expressions.
The following code is also running very well. The variable “d” is not used directly in Task.Factory.StartNew().
...
foreach (EventHandler d in lOnChange.GetInvocationList()) {
Action a = () => d(this, EventArgs.Empty);
Task lTask = Task.Factory.StartNew(a);
lTask.ContinueWith((i) => { Console.WriteLine("Task canceled"); }, TaskContinuationOptions.OnlyOnCanceled);
lTask.ContinueWith((i) => { Console.WriteLine("Task faulted"); }, TaskContinuationOptions.OnlyOnFaulted);
lTask.ContinueWith((i) => { Console.WriteLine("Task completion"); }, TaskContinuationOptions.OnlyOnRanToCompletion);
}
...
Events (part 2, advanced)
We are going to construct our custom event accessor now. It deals with additions and removals of subscriptions. Accessors do pretty much look like property definitions. But instead of set and get you have to use add and remove.
public class MyActionEvent4 {
private object _Lock = new object(); // a new object simply to avoid lock conflicts
private event EventHandler<MyArgs> _OnChange;
private event EventHandler<MyArgs> OnChange {
add {
lock (_Lock) { _OnChange += value; }
}
remove {
lock (_Lock) { _OnChange -= value; }
}
} //
public void RaiseEvent() {
lock (_Lock) {
EventHandler<MyArgs> lHandler = _OnChange;
if (lHandler == null) return;
lHandler(this, new MyArgs(0));
}
}//
} // class
Now we have one big problem here. The RaiseEvent() method has to obtain a lock each time, which causes a serious impact on time sensitive programs. Luckily you do not need to care about changing subscriptions during the invocation. Delegate references are thread-safe, because they are immutable like strings. Let’s simply take the lock out of the RaiseEvent() method, et voilà !
public class MyActionEvent5 {
private object _Lock = new object();
private event EventHandler<MyArgs> _OnChange;
private event EventHandler<MyArgs> OnChange {
add {
lock (_Lock) { _OnChange += value; }
}
remove {
lock (_Lock) { _OnChange -= value; }
}
} //
public void RaiseEvent() {
EventHandler<MyArgs> lHandler = _OnChange;
if (lHandler == null) return;
lHandler(this, new MyArgs(0));
}//
} // class
It is clear that events are not delegates. They restrict access rights from outside of the event class. To describe events you could most likely say that they are wrappers around delegates.
Whenever an exception is thrown during an event call then all following calls will not be executed. And it is tricky to determine which calls were not executed, because the order of event calls is not guaranteed to be in sequence. In fact it does execute in sequence, but there is no guarantee.
static void EventExceptions1() {
MyActionEvent3 lEvent = new MyActionEvent3();
lEvent.OnChange += (sender, e) => Console.WriteLine("Executed subscription 1");
lEvent.OnChange += (sender, e) => { throw new Exception("OMG!"); };
lEvent.OnChange += (sender, e) => Console.WriteLine("Executed subscription 3");
lEvent.RaiseEvent();
} //
So you have to deal with exceptions manually if you want to satisfy/execute as many event subscriptions as possible. You could add a try/catch block for each subscription. Or you could invoke the InvocationList yourself by calling the GetInvocationList() method [System.Delegate] and execute each item manually in a try/catch block. Let’s have a look at the following practical solution:
public class MyActionEvent6 {
public event EventHandler OnChange = delegate { };
public void RaiseEvent() {
List<Exception> lExceptions = new List<Exception>();
foreach (Delegate lHandler in OnChange.GetInvocationList()) {
try { lHandler.DynamicInvoke(this, EventArgs.Empty); }
catch (Exception ex) { lExceptions.Add(ex); }
}
if (lExceptions.Count > 0) throw new AggregateException(lExceptions);
}//
} // class
static void EventExceptions6() {
MyActionEvent6 lEvent = new MyActionEvent6();
lEvent.OnChange += (sender, e) => Console.WriteLine("Executed subscription 1");
lEvent.OnChange += (sender, e) => { throw new Exception("OMG!"); };
lEvent.OnChange += (sender, e) => Console.WriteLine("Executed subscription 3");
try { lEvent.RaiseEvent(); }
catch (AggregateException ex) {
foreach (Exception lException in ex.InnerExceptions) {
Console.WriteLine(lException.InnerException.Message);
}
}
} //
Events (part 1, advanced)
Events link code dynamically together via subscriptions. There are many ways to achieve this. To better understand events, we will start with the old school approach. It is pretty much Java style. You would not do such in C#.
public interface IListener {
void OnEvent(int xDummy);
} // interface
public class AnyClass {
private List<Ilistener> _Listeners = new List<Ilistener>();
public void AddListener(IListener xListener) {
lock (_Listeners) { _Listeners.Add(xListener); }
} //
public void RemoveListener(IListener xListener) {
lock (_Listeners) { _Listeners.Remove(xListener); }
} //
protected void RaiseEvent() {
lock (_Listeners) {
foreach (IListener lListener in _Listeners) lListener.OnEvent(0);
}
}//
public void DoSomeCalc() {
Console.WriteLine("calculating something");
Console.WriteLine("done");
Console.WriteLine("going to tell others");
RaiseEvent();
} //
} // class
public class MyLittleProgram : IListener {
public void StartDemo() {
AnyClass c = new AnyClass();
c.AddListener(this);
c.DoSomeCalc();
c.RemoveListener(this);
} //
public void OnEvent(int xDummy) {
Console.WriteLine("Hey, cool! I got a notification");
} //
} // class
static void Main(string[] args) {
MyLittleProgram p = new MyLittleProgram();
p.StartDemo();
Console.ReadLine();
} //
Above example program uses a List that stores classes. An event is raised by iterating through that list and calling the desired method. This is a lot of code for something really simple.
In the post “Delegates (basics)” https://csharphardcoreprogramming.wordpress.com/2013/12/16/delegates-basics/ we have learned to use delegates. And let’s emphasize it again: We are talking about MulticastDelegates. In the post “Lambda expressions (advanced)” https://csharphardcoreprogramming.wordpress.com/2013/12/17/lambda-expressions-advanced/ we then came across Func and Action. The next step would be to make use of Action.
public class MyActionEvent {
public Action OnChange { get; set; }
public void RaiseEvent() {
Action lAction = OnChange;
if (lAction == null) return;
lAction();
}//
} // class
static void Main(string[] args) {
MyActionEvent lEventClass = new MyActionEvent();
lEventClass.OnChange += () => Console.WriteLine("I got a notification.");
lEventClass.OnChange += () => Console.WriteLine("Me too.");
lEventClass.RaiseEvent();
Console.ReadLine();
} //
Notice that we used a local delegate variable (Action lAction = OnChange;) and did not call OnEvent directly. This is important in a multithreaded environment. Allthough it is unlikely, the event could still turn null before it gets raised.
Above code is much more legible now. But it is really bad practice. The event is accessible from outside. The class MyActionEvent loses control over the event. The Action OnChange is accessible and can be raised from outside the class. This is why C# implemented the “event” keyword. Code no longer uses public properties but public event fields, which are properly protected.
An event can only be assigned by using “+=”, whereas an Action can also be entirely overridden with “=”. It makes programming a little bit safer, you don’t run the risk of removing all previous subscriptions by mistake. Also events can only be raised by code within the same class. There is no way events could be raised from outside.
To make your life easier you can assign an empty delegate during the initialization process. By using “= delegate { };” there is no further need to check for the null condition, especially that no outside code can assign null to the event. Only code inside the class can assign null. Therefore the class is in full control of the event.
public class MyActionEvent2 {
public event Action OnChange = delegate { };
public void RaiseEvent() {
OnChange();
Console.WriteLine("number of event handlers: " + OnChange.GetInvocationList().Length);
}//
} // class
Personally I do not like the approach with empty delegates. The number of delegates is artificially increased by one. A null check is pretty much using no time at all, so why should someone prefer to call an empty method each time an event is raised?
The next step is to replace the dirty Action by a proper delegate definition, which is EventHandler<>. By default its parameters are the sender object and some event arguments. This is a C# convention and should be followed. Surely you are familiar with such event calls from the Winforms/WPF environment.
public class MyArgs : EventArgs {
public MyArgs(int xValue) {
Value = xValue;
} // constructor
public int Value { get; set; }
} // class
public class MyActionEvent3 {
public event EventHandler<myargs> OnChange;
public void RaiseEvent() {
EventHandler<myargs> lHandler = OnChange;
if (lHandler == null) return;
lHandler(this, new MyArgs(0));
}//
} // class
static void Main(string[] args) {
MyActionEvent3 lEventClass = new MyActionEvent3();
lEventClass.OnChange += (sender, e) => Console.WriteLine("Event raised, field value is: " + e.Value);
lEventClass.RaiseEvent();
Console.ReadLine();
} //
Lambda expressions (advanced)
Well, I have used lambda expressions in previous posts already. I was expecting that you know them already.
A Lambda function does not have a name, therefore it is also refered to as anonymous function.
Let’s have a closer look now. In my last post I was introducing delegates and we started with the following example:
// methods
private double Multiply(double a, double b) { return a * b; }
private double Add(double a, double b) { return a + b; }
// a delegate for any of the above methods
private delegate double dCalc(double a, double b);
void Delegates1() {
dCalc calc = Add; // creates a new delegate
Console.WriteLine("Sum " + calc(6.0, 3.0));
calc = Multiply; // creates a new delegate
Console.WriteLine("Product " + calc(6.0, 3.0));
Console.ReadLine();
} //
Above code can be shortened by using lambda expressions. It also becomes more legible then:
private delegate double dCalc(double a, double b);
void Lambda1() {
dCalc calc = (x, y) => x + y;
Console.WriteLine("Sum " + calc(6.0, 3.0));
calc = (x, y) => x * y;
Console.WriteLine("Product " + calc(6.0, 3.0));
Console.ReadLine();
} //
(x, y) => x + y; is a lambda expression with x and y defined as doubles. The types were defined in the delegate dCalc, which has two doubles as input parameters and one double as return value. The “@” sign is pronounced “at”, likewise the “=>” sign is pronounced “go to” or “goes to”.
As usual you can use curly braces to group multiple statements:
(x, y) => { Console.WriteLine(“hello”); return x + y;}
You don’t have to use parentheses if you only have one parameter.
(x) => x + x;
equals
x => x + x;
And you don’t need the “return” when there is only one statement.
x => x + x;
equals
x => { return x + x; }
C# offers built-in types. They are Func and Action. You can use them to define delegates. Both can take between 0 and 16 parameters. Action has no return value, Func must define a return type in the last parameter.
void Lambda2() {
// syntax Func<first parameter type, second parameter type, return value type>
Func<double, double, int> func = (x, y) => (int)(x + y);
// syntax Action<first parameter type, second parameter type>
Action<double, double> action = (x, y) => Console.WriteLine(x + y);
} //
Btw. you need empty parentheses whenever you don’t use any parameters.
Func<int> f = () => 2;
You can be more formal and declare types explicitly.
void Lambda3() {
Func<double, double, int> func = (double x, double y) => (int)(x + y);
Action<double, double> action = (double x, double y) => Console.WriteLine(x + y);
} //
If a delegate refers to a local variable it can happen that the local variable does not exist anymore at the time of the lambda expression/(code) execution. It is important to understand that C# is taking care of this problem. The lifetimes of local variables are extended to at least the lifetime of the longest-living delegate. This is called closure.
void Lambda4() {
string s = "hello world";
Action action = () => { Thread.Sleep(2000); Console.WriteLine(s); };
Task.Factory.StartNew(action);
// Exit the method before the task has completed.
// This does not invalidate the string s !!!
} //
C# Hardcore Programming 