These days, I keep running across references to this wonderful New Thing called ‘reactive programming’ so I spent a day researching this wonderful New Thing.

I worked with the Scala language when I played with Spark so I’ve had my initiation to the new functional programming world.

Now, I’m a gray beard and have been programming for several decades and I’ve seen New Things come and go. I’ve also noticed that New Things usually have been done before in one form or another, so, the thing I need to discover is what form this most recent New Thing is taking, how it is being applied, and what it is good for. Like all New Things, I expect it is not the solution to All Problems like the hype seems to make it out to be, but it is instead a solution for a particular problem in some environment. I need to discover when and how to apply this particular tool.

James Coglan’s talk at JSConf EU 2014 entitled Practical Functional programming: pick two, nicely steps through the process of converting if/then logic implementing a multi-stage, asynchronous task and converting it into event flows and functional/reactive form.

The original program is a state machine implemented with if/then statements. My experience is that when approaching a problem as a state machine, one first sees the few, straight-forward states and one says “this will be easy”. But, when dealing with multi-stage, asynchronous processes, once one considers error states, stage order problems, and stage failures, the state space explodes and many of the states are the same. The response is to drop implementing the state machine as a state machine and resort to if/then statements to cover the state space. This approach has a chance of working especially if the original programmer adds some comments about what is expected from the statements and thus giving the next code visitor (the poor person debugging a problem) a hint at the logic.

If you transform the program from if/then procedural programming into a functional program, the state table doesn’t go away. It’s still there but is now embodied in the end nodes of the functional projection of the stream interactions. I mean, the relationship and movement between the states still has to be there but it is in a different form.

I see the state machine form and the functional programming form as two expressions of the same problem and solution. The question is which one affords the best adaptation to the input, the best execution implementation and the best vehicle for debugging.

For me, two of the advantages of functional programming are:

• better model of problems relating to flow of events
• better representation of the program for lazy evaluation and distribution.

The first point is about using a programming tool to easily express the desired solution and to easily debug any problems. For the state machine approach expressed above, say one forgot about an error condition. For a pure state machine implementation, the solution is to add some more states and actions. For an if/then implementation of the state space coverage, the solution is to add more if/than statements which often leads to spaghetti code and unexpectedly breaking things that used to work. The functional program version seems closer to the latter situation – a new error condition means adding .filter or .map to some transform to patch up the condition. Are there tools for finding the new conditions? Say there was a test suite that pushed values through the code. How would error conditions be found? Are there ASSERTs for functional code?

For the second point, computer languages have added features allowing the computer to do some jobs and thus relieving the programmer of some onerous task. Garbage collectors meant that programmers didn’t have to worry about memory allocation and deallocation. Lazy evaluation and distribution of closures means a programmer doesn’t need to worry about optimization of a stream of data transforms (usually done as some complex collapsing of multiple dissimilar transform operations) or the scheduling and placement of data and parallelizable computations.

But the infrastructure has to be able to implement lazy evaluation and distribution. Thomas Reynolds’ Functional Programming in Javascript Equals Garbage describes two problems with functional/reactive programming in JavaScript: 1) there are usually easier ways to accomplish the same in the language without all the new functional cruft, and 2) functionalizing is done in JavaScript with piles and piles of functions that do not know, for instance, how to collapse tail recursion and all the other lazy evaluation features of language implementations designed for functional programming.

After writing the above, I will go back and read The Reactive Manifesto and General Theory of Reactivity and think about how to apply this technology. I am not one to discount a New Thing just because “programmers have been doing this for years” or because this is just the Latest Thing. Tools and conventions are being developed that could make it a better solutions for some problems – a better solution than the way I’ve always done it.

Comments

My previous job had me programming in Scala which was my most recent introduction to the world of functional programming after years of declarative programming. While trying to bend my head around the functional way of describing programming, I found that, to me, the way the languages and concepts are described to be completely un-useful.

Take monads.

As a programmer, I need a feeling about what my program is going to do. How it works under the covers. Not that I need to know what specific binary bits are generated or how exactly it will be executed, but some concept of now my code is mapped to the underlying execution of it. When writing in declarative languages, the mapping from language to execution is straight forward: first this step, then that. We all know there will be optimizers that will take out unneeded operations and ordering might be rearranged but basically the program I write defines the steps for execution.

For functional languages, that mapping is not as straight forward. Not that it is bad or weird. It is just different. Once you get that mapping into your head, then you ‘get it’. Somehow the light dawns and you have the ‘feel’ for the language.

But functional language constructs are not explained that way. If one asks “What is a monad?”, the explanations are about the abstract features of the language construct – its definition. But this doesn’t really get the uninitiated closer to a feeling for what it does.

Getting a handle on the mapping is made all the harder by the layers of the new languages. Scala (the one I’m familiar with) has all forms of ‘syntactic sugar’ which maps what one writes into lower level operations which are then mapped into lazily evaluated execution closures, which are then mapped onto execution threads that vary at runtime based on the input data.

In the beginning, one starts learning Scala types and operations (List.map(), …) and one starts getting the feel for data with operations vs operations on data.

Then you hear about these functional programming things called monads and you look them up. There, you find discussions about bind and return operations (or is it unit?) and an introduction of new syntax and operators to do what exactly? Something about creating a pipeline of functions. Really? Wouldn’t you just write them one after another? Why a new operator?

The explanations aren’t helped by advancing into language sugar that you aren’t handy with yet. For Scala, it is the optional function dot and parentheses. This stuff is not covered in the introduction to that language which has already confused your reading of code with postfix type declarations.

After a lots of web searching, you start stumbling over articles like Monads - Another way to abstract computations in Scala and Scala FAQ: Yield which start to explain the mapping of some language idioms to the underlying functions. After a while one realizes that writing:

for {
  i <- 1 until n
  j <- 1 until (i-1)
  if i > j
}
yield (i, j)

is the same as writing:

for ( i <- Range(1,n); j <- Range(1,i-1); if i<j) yield (i,j)

which is the same as writing:

Range(1,n).flatMap(i => Range(1,i-1).flatMap(j => List((i,j)).filter(c => c._1 > c._2)))

That is, the Scala coding convention for building up monads is a bunch of rewriting syntactic sugar around the application of map and filter operations.

For me, learning functional programming and languages like Scala are made difficult by the need to learn several different features of the language at the same time. There is the syntax of the language (postfix type declarations, optional syntax, multiple ways of writing the same operation, …), all of the syntactical sugars (with documentation that explains different features using different sugar conventions), functional coding conventions (monads, pipelines, …), lazy evaluation, and on and on.

Well, maybe this is just me twisting in the wind over learning a new programming method. It’s good for me.

Comments

The latest Communications of the ACM (6/2016 Vol 59. No 6) has the interesting article Enhancing Symbolic Execution with Veritesting. This describes a program testing method called “symbolic testing”. This is where, rather than stepping through the program for specific test values, one pushes all values through every path of the program. Got that?

It is a way of testing programs by pushing “symbols” through all the paths of the program and the symbols represent all possible values that values could have at that point in the program.

Mind blown.

Because I have been programming computers for decades, I have a lot of programming habits left over from a more constrained era. I still worry about the amount of memory or the number of iterations. While it is a consideration and performance is always necessary, the limits and bounds have changed a lot.

What Moore’s Law has made posssible is astonishing. From deploying deep learning everywhere to, well, simulating every path in a program.

It is a lesson I need to internalize and I should not be limited by what I feel is computable.

Comments

Notes on looking at the merge diffs:

OpenSim/Addons/Groups/GroupsModule.cs

0.8.2 in OnMakeRoot, does SendAgentGroupDataUpdate. 0.9.0 removes this with comment
    that this is a wrong thread to do this and that viewer ignores this message.
    This update missing causes problems (Mantis 7920 and 7915)
    Restored the SendAgentGroupDataUpdate with a comment.
The code for group updates was refactored differently in the two versions:
    0.8.2 added an OnAgentDataUpdateRequest for extensibility and used that
        in SendAgentGroupDataUpdate
    SendAgentGroupDataUpdate was changed to send to optionally update other clients
    0.8.2 put logic for sending update via Caps in GroupsModule.
        Merge properly put it in LLClientView
    0.9.0 cleaned up update with SendAgentGroupDataUpdate and SendDataUpdate
8.2 commented out OnClientClosed event commenting that InstantMessageModule doesn't do this
    Merge added logic to OnClientClosed to find who to send the update to
        The original problem was that the event was fired when there were several
            possible destination
OnAgentDataUpdateRequest: 0.9.0 calls SendAgentGroupUpdate while 0.8.2 calls
    both SendAgentGroupUpdate and SendScenePresenceUpdate. 0.9.0 has comment that
    region arrival will send the group title. Keep lookout for not getting group titles.
0.9.0 sends group updates to all clients more often than 0.8.2.
    Could be extra traffic when changing groups but probably not a problem

Comments

About 6 months ago, a big source merge was done in OpenSimulator. The merge added many fixes and improvements to OpenSimulator (more stable TPs, vehicle region border crossing, better permissions, etc). The merged sources were from an old fork of OpenSimulator. The sources had been forked and improved for several years before the sources came back to be merged. This sort of thing would happen if any of the public grids donated their sources back to the base OpenSimulator projects (hint, hint – you other grids should really be donating back!).

Anyway, over the years between the fork and the source coming back, there were independent changes in OpenSimulator and the other grid. Doing a ‘git merge’ on two related, but independently modified sources is an iffy thing. You never know what changes will end up in the final sources.

On 20151116, commit f93ce485b1df8c1c8a7ff1a44b280ce30b8707b9 merged the branch ‘avinationmerge’ into ‘master’ and the version 0.9.0 journey began.

There are now two versions/branches of OpenSimulator: branch ‘master’ which is the unreleased version 0.9.0 and the branch ‘0.8.2-post-fixes’ which is the last released, ‘working’ version of OpenSimulator.

The ‘0.8.2-post-fixes’ branch is rooted in master on November 11, 2015 (commit f980355da4e233f714534afc6b958ceda1d07115) with patches applied thereafter for various bug fixes.

The problem is that Git will merge but that doesn’t mean what you wanted to do actually happened. My goal is to do a big diff between 0.9.0 and 0.8.2-post-fixes and go through the changes by hand. It will take a while but, I will be posting progress reports. The plan is to spend an hour a day going through the diffs.

Comments