Robert Pickering's Strange Blog

OpenWrap – A Package Management System for .NET

Robert Pickering — Sun, 16 May 2010 14:41:05 GMT

At Progressive.NET one of the sessions I attended was Sebastien Lambla’s talk on OpenWrap. OpenWrap is a package management system for .NET projects. I’m excited about this project for a very simple reason: it’s something the .NET Open Source community desperately needs.

What is package management?

Package manager is basically a piece of software that allows you to install the software or components you need on to your system. In one sense in the windows world this is handled by MSI, but that’s only half the picture of a package management system; ideally a package management system would provide a centralised server that handlers allowing to query what the latest version of a package is and download that or any other version of a package. Since in general software is build using other packages then it would be nice if it would automatically handle downloading the complete depancie graph.

Why does .NET open source world need one?

Well, leaving aside the fact that every other open source development community has one, it is need because handling dependencies between packages manually is time consuming and painful. Last year I was involved in setting up a small new web project. Naturally, I was keen to try out ASP.NET MVC, and as I had more or less a free hand, I also wanted to use Castle Windsor, Fluent NHibernate, Linq2NHibernate and some of ASP.NET MVC contrib which would allow me to easily integrate my controllers into Windsor. Now, although once I had everything going these technologies certainly made life easier. However, getting them in place was not so simple, firstly, and it’s quite a minor point, there’s no centralised localation where all these items can be downloaded so you have hunt around various project hosting sites, such as sourceforge, codeplex, google code, and github to find all the relevant binaries. Once found, it’s then that all the real fun and games start. As each of these projects depend on the other projects you need to ensure that you have the correct version of each project and if you download the latest version of each project, it’s odds on that you will not. To add to the fun and games the projects with incompatible versions will fail in different ways depending on if they are signed or not. An unsigned assembly will always be loaded, as the .NET loader simply ignores the version, number if it’s not signed. This will cause a MissingMethodException if there is an incompatibility in the interface. If an assembly is signed then an incompatible version will simply refuse to load, yes you can mess around with loading policy in the app.config to force the .NET loader to load a different version, but this isn’t a fun or good solution to the problem. So in general you have to decide which projects are important that you have the right version, then find versions of the other projects that are compatible with them.

Is hornget a package management system?

No. While like hornget and it does solve a similar problem, it does it in a slightly different way. The hornget tool allows you to download the source of a particular project, along with its dependencies, to your local computer and build it from scratch. While this will solve many of your version compatibility problems there’s a couple of problems with this approach:

- Building can fail and is a time consuming process

- Control over the version your building is somewhat limited, though this may improve overtime

How does open wrap work?

Packages are stored as zips on a central server. OpenWrap provides a DSL that will allow you to specific which packages you want to use and which version of that package you would like (the syntax here is quite flexible allowing you to provide a minimum and maximum version). OpenWrap provides a set of msbuild targets that allow you to hook this DSL based description into your build process and, of course, visual studio. These msbuild targets will then take care of downloading the projects you require and there dependencies to a centralised cache on your local machine. This centralised cache will then be used to provide the references required to build you’re project.

How can I get involved?

Simple. Download the source. Try it out. Talk to Seb to contribute feedback, packages and patches to improve and grow the functionality. There is much work still to be done.

So I hope that this article has convinced you a package manager is need for .NET. Is Seb the man to pull it off with his OpenWrap system? I certainly hope so, definitely a project to keep a close watch on.

(Other info about the stuff at Progressive.NET to follow shortly)

My Talks in London: The F# User Group & At Progressive.NET

Robert Pickering — Fri, 14 May 2010 16:29:44 GMT

I’ve been in London for the past few days. I arrived Tuesday afternoon and gave at talk at “The F#unctional Londoners Meetup Group” on the Tuesday evening. The talk was entitled “Using Language Oriented Programming to Execute Computations on the GPU” and thanks to the efficiency of the good folks at Skills Matter the talk is already available on line. The event was very well attended, with around 50 people turning up. The talk covers more the Language Oriented Programming aspects than the knitty gritty details for detailing with the GPU and the audience seemed to enjoy it. It was a fun evening and good to meet Carolyn Miller, who started the group, as well fellow F# Guru’s Jon Harrop and Phil Trelford.

I’ve also gave a talk at the Progressive.NET event organised by Skills Matter. This was a four hour tutorial, than covered the basics of the syntax, and how to use all the features. It then moved on to cover more advanced topics like language oriented programming, parallel programming and collective intelligence. The video is already online and there are a few pictures of me presenting on flickr:

Strings and F# Immutable Lists

Robert Pickering — Sun, 18 Apr 2010 08:10:01 GMT

In .NET strings are immutable. I think most .NET programmers agree that this is a good thing, as it generally makes code that works with strings safer and more predictable. However, operations that involve large amounts of string manipulation perform poorly because each time a string concatenation occurs then both strings must be copied. His often leads people to believe that all immutable objects will perform poorly. The aim of the article is to show that this is not the case, or at least it’s not as black and white as you may think. Specifically we’re going to compare how.NET strings differ considerable in performance characteristics to F#’s immutable lists.

Before we dive into immutable lists, let’s take a look at some code that demonstrates some poorly performing string concatenation and then take a look at how we can fix this.

class Program

{

static void Main(string[] args)

{

var strings = new List<string> { "Toto", "Titi", "Tata" };

var res = "";

foreach (var s in strings)

{

res += s;

}

Console.WriteLine(res);

}

As the length of the list “strings” grows, we will start to see the performance of this code degrade. The performance of this code for a list of 3 items, will be perfectly acceptable, but we’ll start to serious perf problems as we approach 100s or 1000s of items (also depending on the length of the strings in the list). This is because both “res” and “s” must be copied to perform the concatenation. So as “res” grows with each iteration of the loop so does the time to perform the copy.

The way to fix this is fairly obvious, we use a string builder:

class Program2

{

static void Main(string[] args)

{

var strings = new List<string> { "Toto", "Titi", "Tata" };

var res = new StringBuilder();

foreach (var s in strings)

{

res.Append(s);

}

Console.WriteLine(res.ToString());

}

What’s more interesting is to look at why this fixes the problem. It would be fairly easy to say a string builder is mutable therefore faster, but this isn’t quite the answer we’re looking for. The reason using a string builder is faster, in this case, is because we avoid the cost of copying “res” each iteration, only the newly input “s” needs to be copied into the string builder. While a string builder is capable of performing inserts/removes in this case we get good performance because we limit ourselves to append to the end of collection of characters (string builder could be thought of as just a special type of collections of characters).

F#’s lists are immutable linked list. This means each items in the list is either empty, which always represents the end of a list, or the item stored in the list along with a reference to the remaining list. This means a graphical representation of an F# list would look something like this:

We build up F# lists by appending new items to the head of the list. Here in C# is how we would build up an F# list from the frameworks generic List<> class.

class Program3

{

static void Main(string[] args)

{

var strings = new List<string> { "Toto", "Titi", "Tata" };

var res = ListModule.Empty<string>();

foreach (var s in strings)

{

res = FSharpList<string>.Cons(s, res);

}

Console.WriteLine(res.ToString());

}

While the syntax for working with F# lists from C# is rather ugly, I think this example makes it clear how F# list work. Each item is the collection “strings” is taken in turn and append to the head of the F# list using its “Cons” operator. In many ways this is similar to how the string builder behaves, we have no need to make a copy of “res”, we simple append each new “s” to the end of it. For those of you curious to see what this would look like in F# here’s the equivalent in F#, although it is really idiomatic F# since it uses a mutable identifier:

let main() =

let strings = new ResizeArray<string>(["Tata"; "Toto"; "Titi"])

let mutable res = List.empty

for s in strings do

res <- s :: res

printfn "%A" res

main()

In conclusion, you need to look beyond whether a data structure is mutable or immutable to see whether it will perform well. In some circumstances F#’s immutable list can perform very well, i.e. when you only need the ability to append to the end of the collection and enumerate the collection. In other circumstance they may perform poorly, i.e. when you need random access to items within the lists. These performance tradeoffs also need to be balanced with the fact the F# list can also safely be shared between threads or tasks, we can be very convenient in many concurrency scenarios.

#Develop, F# and Google Summer of Code

Robert Pickering — Wed, 24 Mar 2010 19:43:09 GMT

A few days ago Chris Wille, the man behind #Develop contacted me to say that #Develop had been accepted on the Google summer of code scheme. Essentially Google summer of code means that Google will pay CS students to spend their summer writing code for open source projects. Chris told me one area they are interested in extending is the F# integration into #Develop; particularly they’d like to get the auto-completion working. So if you’re a CS student on the lookout for a summer job, you could end up being paid by Google to work on F#! Contact me if you are interested.

Beginning F# Workshop and Progressive.Net

Robert Pickering — Sat, 20 Mar 2010 14:04:04 GMT

I’ll in London on the 10th/11th May giving a two day course entitle “Beginning F# Workshop” in conjunction with the lovely folks at Skills Matter. I think I give a fairly good description of the course on the workshop webpage, so I wanted to use this blog post to talk about why you would want to attend. I believe, as I stated in my book, that functional programming will be the next big wave in programming, and that F# will be one of the languages at the forefront of this functional programming renaissance. I don’t believe this will be sole driven by the move from one core processors to many core processors and the move towards parallel programming that this will create, though this will undoubtedly help. I believe functional programming is will be the next big wave in programming as functional languages provide better ways of raising the level of abstraction than procedural and object oriented programming languages do. Sadek Drobi has a nice talk on computation abstraction in he explains some of these new types of abstraction that functional programming languages offer. With the kind of systems we are building today, we are now ready to start working at these new levels of abstraction.

While I don’t believe functional programming to be any harder than object oriented programming or procedural programming, I do believe it takes a bit of a mind shift to get to know the functional programming style. This course will offer a nice gentle introduction to how to use functional programming techniques in F# and how to mix these with the object oriented and imperative styles were necessary. So whether or not you’re planning to use F# or not (although of course, I think you should be), I think this course will give you a new and interesting perspective on programming. Book before 26th March and save over £300.

I’m also very happy to announce that I’ll be running a workshop on F# at progressive.net, its taking place in London on 12th -14th May. It’s a real pleasure to be able to be involved with an event that has speaks like David Laribee, Oren Eini (a.k.a. Ayende Rahien), Hamilton Verissimo de Oliveira, Sebastien Lambla, and Mike Hadlow. For those of you who made it last year, (or have seen the podcast) the workshop will be along similar lines except I’ll be trying to make it much more interactive. Instead of just following along, as I present the concepts I’ll be adding a series of hands on labs to help get participants solving problems in F# for themselves. In many respects the tutorial I’m giving will be a cut down version of the tutorial I’m giving beforehand (but if you planning to attend both you can always go and see Seb’s session on RESTful architectures). If you’re in any doubt whether you should attend you should probably read this blog post by Dylan Beattie. Book before tomorrow (March 21st) and you'll save nearly 50%!

My Talk with Don and Adam at TechDays 2010

Robert Pickering — Sat, 13 Feb 2010 16:13:21 GMT

It was TechDays this week and I was lucky enough to be invited to speak with Don Syme and Adam Granicz. Don gave a short introduction to mailboxes/agent programming in F# and Adam gave an introduction to IntelliFactory’s WebSharper project.

My section of the talk was on using F# with Microsoft Accelerator. Accelerator is an interesting product, it allows you to take advance of the GPU of your PC. Even the most basic PCs have a specialized graphics chip and for most of the tasks you perform on your PC it just sits there doing nothing. GPU are very good at performing floating point operations on matrices, they’re specialized hardware that allow you to perform matrix operations in a highly parallel way. The DirectX runtime takes advantage of this to perform the operations need to for 3D shading. However, there are other algorithms that could benefit from being run on the GPU and Accelerator opens up the GPU for just about any algorithm that you can imagine.

The usage I demonstrated was the game of life running either on the CPU or with the calculations performed on the GPU. On my current machine this yields a 10 fold increase in execution time. It takes advantage of F# quotations system (similar to C#’s expression trees), these are used to allow you to compile an algorithm in F# to operations that the Accelerator runtime understands. This is how we can easily show an algorithm running on the CPU and then switch it to the GPU. The slides can be found here and code examples here. The example comes from Tomáš Petříček’s web site and more details can be found about in this series of articles.

I had several people come and access me for more information about Accelerator at the end of the talk. Accelerator v2 is available from Microsoft connected, here. It’s currently licensed under a restrictive MSR license that stipulates non-commercial use only. However, in stating the objectives of their project, they say “Accelerator v2 is an MSR incubation project whose goal is to validate the architecture and API approach with high quality engineering sufficient to gather real-world usage data.” It would seem to me that in order to achieve this objective they would need to have some commercial users, so maybe if you contact the team directly they would be able to get you a less restrictive license. I certainly have a project in mind for accelerator, though at this point it would be non-commercial. If you’re interested check out activity on my github account, you can probably guess what my idea is.

The slides can be found here and the code can be found here. The session was videoed, I’ll let you know as soon as it’s on line.

Update: The video can now be seen here.

FunctionalNHibernate: RCed and now with added build server

Robert Pickering — Sat, 13 Feb 2010 14:48:16 GMT

I’m very happy to announce that FunctionalNHibernate has now been updated to work with the recently released Visual Studio 2010 release candidate. Also, it has now been setup to run on the codebetter build server. You can create a log on and download the latest binaries from here (you need an account but it’s very simple to create one).

A huge thanks to Steffen Forkmann, who did the work to get this set up.

Functional Programming eXchange: The Videos

Robert Pickering — Sat, 13 Feb 2010 14:30:23 GMT

I’ve recently noticed that talks from Functional Programming eXchange, that I organized and spoke at, are now online. This is a great resource for people looking for some more advanced material on Functional Programming. So here are the videos:

SADEK DROBI: Computation Abstraction

http://skillsmatter.com/podcast/cloud-grid/computation-abstraction

MATTHEW SACKMAN: Supercharged Rabbit: Resource Management at High Speed in Erlang

http://skillsmatter.com/podcast/cloud-grid/supercharged-rabbit-resource-management-at-high-speed-in-erlang

ANTON SCHWAIGHOFER: F# and Units-of-measure for Technical Computing

http://skillsmatter.com/podcast/cloud-grid/f-and-units-of-measure-for-technical-computing

GANESH SITTAMPALAM: Functional Programming for Quantitative Modeling at Credit Suisse

http://skillsmatter.com/podcast/cloud-grid/functional-programming-for-quantitative-modelling-at-credit-suisse

DUNCAN COUTTS: Strong Types and Pure Functions

http://skillsmatter.com/podcast/cloud-grid/strong-types-and-pure-functions

ROBERT PICKERING: Combinators, DSLs, HTML and F#

http://skillsmatter.com/podcast/design-architecture/using-combinators-to-tackle-the-html-rendering-problem

Update about FunctionalNHibernate

Robert Pickering — Sat, 26 Dec 2009 13:21:16 GMT

I’ve been working quite a lot on the FunctionalNHibernate project recently. I’ve implemented the following features:

- Save records with native F# lists.

- Create one-to-many and many-to-many relations

- Generate identities automatically

We can now do just about everything you the FluentNHibernate first example project does, although admittedly this is just a very small subset of what one can do with FluentNHibernate and NHibernate itself.

In the future I’ll be looking at how to broadening what you can do with FunctionalNHibernate ClassMap descriptions and improve data access by integrating Linq like features.

In terms of usability, it’s still early days but I think it’s probably usable for a project with simple data access needs. Obviously you need to ready to put up with some teething problems and there will definitely be some breaking changes as the project evolves.

The project uses Mercurial for source control and is hosted at bitbucket.org, but don’t need to install Mercurial to use it, you can download the zipped source here: http://bitbucket.org/robertpi/functionalnhibernate/downloads/

I’ve licensed it under a simple BSD like license: http://bitbucket.org/robertpi/functionalnhibernate/src/tip/LICENSE.txt

I’ve also created a google group for announcements, discussions and questions about the project:
http://groups.google.com/group/functionalnhibernate

Introducing FunctionalNHibernate

Robert Pickering — Sun, 20 Dec 2009 11:26:44 GMT

It’s already fairly well documented that F# doesn’t play too well with NHibernate and Fluent NHibernate, although you can make it play a littler nice with a bit of effort. However there are a few fundamental problems with this approach. The first is F# class’ are not really designed to be data containers because that’s what its record type is for. The second is that F# class’ do not put too much emphasis on virtual methods as in functional programming we tend to using pass functions as values as a way to achieve polymorphism rather than virtual methods. Third and finally FluentNHibernate works with C#’s expression trees, it would be nice use F#’s quotation system instead.

Record types are a fixed collection of names and fields a lot like the row in a database. Below shows a record type we’d like to be able to store in a database table. Here we show an invoice record type that we’d like to serialize in the database.

What we’d like to be able to write is a mapping class that describes how it should be serialized, like you can do in FluentNHibernate:

The idea of FunctionalNHibernate is to provide a layer to do just this. As you see we provide a neat way to describe the mappings. We achieve the record serialization by providing a custom tupilizer for F# record types. There’s also the start of a wrapper for the SessionFactory that will hopefully allow you to store and retrieve data in way more natural for F#.

The idea of this project is not to detract from FluentNHibernate, it’s a great project and works really nicely from C#. I want to provide something that will work equally well from F#. We’re a long way behind them and to be honest the project isn’t really usable yet. I have project in mind that I want to use it on, so hopefully I’ll be able to improve it as I go along.

The result is you can now write the following program to store, update and retrieve our invoice record type:

The project is hosted on bitbucket.org, where you can explore or download the code. Feel free to drop me a line if you want to help out.