Recalculating Values Only When Dependencies Change – Part II – Testing the Frameworks

In part one we build a couple of simple frameworks that allow the programmer to specify that a value should be cached until one of its dependencies changed. I wanted to put these frameworks though their paces in a semi realistic environment.

I decided to use the idea of pricing european options, since it’s similar to what we do at LexiFi, rather that work we the LexiFi pricer I took the simplest pricer I could find written in OCaml and adapted it to F# (this was hardly any work at all). I did this as I simply wanted the price to provide some work, I did not care about the accuracy of the process.

Note: Do not use the pricer used here to price real options, it is too simple to give an accurate price.

Our pricer exposes a nice and simple interface that consists of passing it a series of floats and an integer: simple_monte_carlo1 expiry strike spot vol r num_paths. I decided I want to change the spot and the volatility (vol) at random intervals while increasing the work load by increasing the number of paths.

To do this I wrote a testing framework that expected to receive a number of functions that would actually perform the work of doing the test. This code is shown below:

let runTest name f set_paths set_spot set_vol =

    let stopwatch = new Stopwatch();

    for paths in [100000.0 .. 100000.0 .. 1000000.0] do

        set_paths paths

        for odds in [0 .. 5] do

            let times = new ResizeArray<int64>()

            for index in indexes do

                let setValue name set get =

                    if rand.Next(odds) = 0 then

                        let spot = (get index)

                        if printAll then

                            printfn "%s %s Changed to %f" name index spot

                        set index spot

                setValue "Spot" set_spot get_spot

                setValue "Vol" set_vol get_vol

                stopwatch.Reset()

                stopwatch.Start()

                let res = f index

                stopwatch.Stop()

                times.Add(stopwatch.ElapsedMilliseconds)

                if printAll then

                    let printer =

                        match outputFormat with

                        | Readable -> printfn "type = %s, result = %f, paths = %f, odds = %i, index = %s, time taken = %i"

                        | Csv -> printfn "%s;%f;%f;%i;%s;%i"

                    printer name res paths odds index stopwatch.ElapsedMilliseconds

            let avgTime = (times |> Seq.fold (+) 0L) / (int64 indexes.Length)

            let printer =

                match outputFormat with

                | Readable -> printfn "type = %s, paths = %f, odds = %i, avg time = %i"

                | Csv -> printfn "%s;%f;%i;%i"

            printer name paths odds avgTime

 

The parameter name is a name for recording the results, f is the function that does the work and set_paths, set_spot and set_vol control the parameters that the function f should use. Impmenting these frameworks is fairly straight forward, implementing the auto-enroll framework is shown below, the others are available for download as they are very similar:

let acal, aset_paths, aset_spot, aset_vol =

    let state = new AutoEnrollFramework.StateWrapper<float>()

    let paths = state.NewNode(AutoEnrollFramework.Value 100.0)

    let set_paths x =

        paths.Value <- AutoEnrollFramework.Value x

    let ftseSpot = state.NewNode(AutoEnrollFramework.Value ftseSpotBase)

    let cacSpot = state.NewNode(AutoEnrollFramework.Value cacSpotBase)

    let nasdaqSpot = state.NewNode(AutoEnrollFramework.Value nasdaqSpotBase)

    let set_spot index x =

        match index with

        | "FTSE_100" -> ftseSpot.Value <- AutoEnrollFramework.Value x

        | "CAC_40" -> cacSpot.Value <- AutoEnrollFramework.Value x

        | "NASDAQ_100" -> nasdaqSpot.Value <- AutoEnrollFramework.Value x

        | _ -> failwith "unknown index"

    let ftseVol = state.NewNode(AutoEnrollFramework.Value ftseVolBase)

    let cacVol = state.NewNode(AutoEnrollFramework.Value cacVolBase)

    let nasdaqVol = state.NewNode(AutoEnrollFramework.Value nasdaqVolBase)

    let set_vol index x =

        match index with

        | "FTSE_100" -> ftseVol.Value <- AutoEnrollFramework.Value x

        | "CAC_40" -> cacVol.Value <- AutoEnrollFramework.Value x

        | "NASDAQ_100" -> nasdaqVol.Value <- AutoEnrollFramework.Value x

        | _ -> failwith "unknown index"

    let ftseCal =

        state.NewNode(AutoEnrollFramework.Calculation (fun () ->

                MonteCarlo.simple_monte_carlo1 0.2 160.0 ftseSpot.Value ftseVol.Value 0.045 (int paths.Value)))

    let cacCal =

        state.NewNode(AutoEnrollFramework.Calculation (fun () ->

                MonteCarlo.simple_monte_carlo1 0.2 160.0 cacSpot.Value cacVol.Value 0.045 (int paths.Value)))

    let djCal =

        state.NewNode(AutoEnrollFramework.Calculation (fun () ->

                MonteCarlo.simple_monte_carlo1 0.2 160.0 cacSpot.Value cacVol.Value 0.045 (int paths.Value)))

    let cal index =

            match index with

            | "FTSE_100" -> ftseCal.Value

            | "CAC_40" -> cacCal.Value

            | "NASDAQ_100" -> djCal.Value

            | _ -> failwith "unknown index"

    cal, set_paths, set_spot, set_vol

So once the results have been correlated and loaded into excel we get the following picture:

This is pretty much what we expected for the no caching case we see a very strong correlation between the number of paths and the time take. With the other two case we see that the time taken dips as cached results are used with the time rising as when there is more chance of the value being recalculated. The worst cases of the caching scenarios are almost exactly equal to the results for the no caching case, showing that in this test very little over head is added by the framework. This because we remove our values from the cache before performing the work, we do not fetch the result from the cache at each computation. If we did fetch the calculation from cache each time, we might expect the worst case value to increase considerable for the caching scenarios – but then what I have implemented in the test is probably what you’d do in a real programming situation.

The full listing for this test and the results gathered are available for download here.

In the next it this series we’ll look at how we can use the frameworks to automatically visualise calculations and their results using reflection and WinForms.

Bookmark
dotnetkicks+, digg+, reddit+, del.icio.us+, dzone+, facebook+

Print | posted @ Saturday, September 01, 2007 8:23 AM

Comments on this entry:

Gravatar # Robert on &quot;Recalculating Values Only When Dependencies Change&quot;
Robert Pickering has just posted his second article on the topic of incremental evaluation in F# . His

Your comment:

(Note: all comments are moderated so it may take sometime to appear)

Title:
Name:
Email:
Website:
 
Italic Underline Blockquote Hyperlink
 
 
Please add 3 and 1 and type the answer here: