Aug 15, 2016 - Pesto to Python

After sending more time than I wanted on a NodeJS version of Pesto, I came to the conclusion that NodeJS was not the language of choice for that service. This was given away when I found the multi-threaded Thrift server classes for Python.

Sound-of-forehead-slap.

Of course! JavaScript is not a multi-treaded language. Pesto, though, is supposed to be the responsive messaging center of the whole viewer framework. This kinda requires multi-threading.

I originally chose JavaScript/NodeJS because I wanted to build a fancy, interactive, and responsive web interface to Pesto. Guess I will have to do that with some Python libraries. The Python 2 vs Python 3 fork is concerning when thinking of the long term but I’ll have to see how that plays out.

Comments

Aug 14, 2016 - Cassandra and Docker

In my continuing effort to learn all the new technologies, I wanted to use one of the NoSQL databases. Since I want to store geographical data as well as meta data for virtual world objects, I steered away from document oriented ones like ElasticSearch. That leaves ones like MondoDB or CouchDB or pure Hadoop.

The fickle finger of databases then lead to Cassandra. It was used in the Sirikata virtual world project and is scalable and clusterable and continues to be used in a lot of places. It is also available on AWS and other infrastructures so it’s a good candidate.

The two target usages for Basil development are standalone, single computer installations and in larger, production installations. So, how to easily run Cassandra on a single desktop.

Another concern of mine is how to stay up-to-date with the latest versions and sources of any package. I want an answer for ‘what will building and maintaining look like a year from now’.

I pulled the Cassandra sources and looked into building it. It requires a specific version of Java and several other packages. This looked like it was going to be a nightmare to build on general use Linux system and the dependencies would make upgrading difficult. And, like I said above, I wanted the latest and greatest and not the older versions that will be in the Ubuntu package repositories.

But wait. Looking around I found a Docker version of Cassandra. Docker containers would provide some of the isolation of the different versions four all the libraries are going to Cassandra and he would get me again tangled up in other interesting internet technologies.

Docker also provides a solution for creating simple standalone systems as well as complex, production, scalable systems. A simple Docker setup would allow someone to just run a local Cassandra or they could deploy all of the services into the cloud and have a production environment.

So, off down the rabbit hole of learning Docker and learning all of its setup and configuration options.

Comments

Aug 4, 2016 - Thrift and Python

Previous posts have talked about choosing an interconnection system for the multiple parts of a Basil viewer session. I’ve started with Thrift and I am learning of the bumps in the road.

First of all, Thrift is not well documented. Since it is a low level transport that was built for larger projects, it makes some sense that the writers of Thrift really care more about the larger project, but, for us people trying to use Thrift in a new project, a bunch more documentation would be a help. Others have started to fill the gap (like Diwaker’s Thrift: The Missing Guide) but there is still a big barrier to getting started.

I started of trying to use Thrift in JavaScript (the Pesto server) and Python (the test routine for same). Both of these languages are not, shall I say, first class languages in the implementation. It seems most of the work on Thrift is for Java and the examples for the other languages show one simple example and leave the reader wondering about installation, configuration, and the exact syntax for including the generated code.

Thrift is perfectly happy with hyphens in names, Python is not. Make sure all your service names use camel case or similar rather than hyphens or underscores.

As of today (August 3, 2016) the latest Thrift sources require TwistedTrial to be installed to build the Python libraries. The documentation does not mention this and it is frustrating in that no error is generated – Python is just not built even with the --with-python configuration parameter. Since TwistedTrial is only needed to run the test suite, I edited configure.ac to remove that requirement. And viola, Python libraries built and installed.

The change is:

builder@sys:~/thrift-git$ git diff configure.ac
diff --git a/configure.ac b/configure.ac
index c72dd43..bcdaec3 100755
--- a/configure.ac
+++ b/configure.ac
@@ -281,8 +281,9 @@ AM_PATH_PYTHON(2.6,, :)
 AX_THRIFT_LIB(python, [Python], yes)
 if test "$with_python" = "yes";  then
   AC_PATH_PROG([PIP], [pip])
-  AC_PATH_PROG([TRIAL], [trial])
-  if test -n "$TRIAL" && test "x$PYTHON" != "x" && test "x$PYTHON" != "x:" ; then
+  if test "x$PYTHON" != "x" && test "x$PYTHON" != "x:" ; then
     have_python="yes"
   fi
 fi
builder@sys:~/thrift-git$

More notes are most likely on the way as I write Python code to use Thrift.

Comments

Jul 11, 2016 - Some VR Technologies

I attended the Portland Virtual Reality Meetup nearly a month ago and, at the time, I made a list of VR technologies that I needed to understand. This is that short list:

WebVR

WebVR is a JavaScript/DOM API standard for interfacing and application to VR input/output devices. The WebVR API specifies data structures and conventions for controlling the headset. This includes focal length, resolution, and timing. The interface is important because the realtime nature of displaying on the quickly moving head display requires a lot of coordination between the display and the renderer.

Hopefully this API will allow browser games and such to adapt to multiple different headsets and controllers.

aframe.io

AFrame is a JavaScript system that specifies HTML extensions to define 3D scenes which are displayed with WebGL. So it is a simple authoring tool for creating 3d scenes. It is fairly new and doesn’t have a lot of controls for very complex scenes (multi-threading, …).

OSVR

OSVR is an effort to build an open source VR system. OSVR on Github. While there are several proprietary systems, this tries to be the open, hackable version. They have an SDK and a 2160 X 1200 dual display headset which is part of their Hacker Development Kit (HDK). The software seems to be built for Windows and Mac. OSVR is maintained by Sensics and Razer.

JanusVR

JanusVR is an attempt to build a 3D browser for the web. Their description is: “JanusVR is a collaborative web browser for Virtual Reality, where links are portals and pages are rooms. The platform is written in C++ with the OpenGL, OpenAL and Qt libraries.”.

The principle innovation seems to be their integration with existing web pages. You can display web pages, navigate the 2D and 3D works through links, and enhance existing web pages so that they display 3D when viewed by the JanusVR viewer. This latter feature includes a definition of an embeddable XML scene description language. Haven’t looked to see if it has any relation to the XML3D definition.

Comments

Jul 11, 2016 - Peeking at Reactive Programming

These days, I keep running across references to this wonderful New Thing called ‘reactive programming’ so I spent a day researching this wonderful New Thing.

I worked with the Scala language when I played with Spark so I’ve had my initiation to the new functional programming world.

Now, I’m a gray beard and have been programming for several decades and I’ve seen New Things come and go. I’ve also noticed that New Things usually have been done before in one form or another, so, the thing I need to discover is what form this most recent New Thing is taking, how it is being applied, and what it is good for. Like all New Things, I expect it is not the solution to All Problems like the hype seems to make it out to be, but it is instead a solution for a particular problem in some environment. I need to discover when and how to apply this particular tool.

James Coglan’s talk at JSConf EU 2014 entitled Practical Functional programming: pick two, nicely steps through the process of converting if/then logic implementing a multi-stage, asynchronous task and converting it into event flows and functional/reactive form.

The original program is a state machine implemented with if/then statements. My experience is that when approaching a problem as a state machine, one first sees the few, straight-forward states and one says “this will be easy”. But, when dealing with multi-stage, asynchronous processes, once one considers error states, stage order problems, and stage failures, the state space explodes and many of the states are the same. The response is to drop implementing the state machine as a state machine and resort to if/then statements to cover the state space. This approach has a chance of working especially if the original programmer adds some comments about what is expected from the statements and thus giving the next code visitor (the poor person debugging a problem) a hint at the logic.

If you transform the program from if/then procedural programming into a functional program, the state table doesn’t go away. It’s still there but is now embodied in the end nodes of the functional projection of the stream interactions. I mean, the relationship and movement between the states still has to be there but it is in a different form.

I see the state machine form and the functional programming form as two expressions of the same problem and solution. The question is which one affords the best adaptation to the input, the best execution implementation and the best vehicle for debugging.

For me, two of the advantages of functional programming are:

  • better model of problems relating to flow of events
  • better representation of the program for lazy evaluation and distribution.

The first point is about using a programming tool to easily express the desired solution and to easily debug any problems. For the state machine approach expressed above, say one forgot about an error condition. For a pure state machine implementation, the solution is to add some more states and actions. For an if/then implementation of the state space coverage, the solution is to add more if/than statements which often leads to spaghetti code and unexpectedly breaking things that used to work. The functional program version seems closer to the latter situation – a new error condition means adding .filter or .map to some transform to patch up the condition. Are there tools for finding the new conditions? Say there was a test suite that pushed values through the code. How would error conditions be found? Are there ASSERTs for functional code?

For the second point, computer languages have added features allowing the computer to do some jobs and thus relieving the programmer of some onerous task. Garbage collectors meant that programmers didn’t have to worry about memory allocation and deallocation. Lazy evaluation and distribution of closures means a programmer doesn’t need to worry about optimization of a stream of data transforms (usually done as some complex collapsing of multiple dissimilar transform operations) or the scheduling and placement of data and parallelizable computations.

But the infrastructure has to be able to implement lazy evaluation and distribution. Thomas Reynolds’ Functional Programming in Javascript Equals Garbage describes two problems with functional/reactive programming in JavaScript: 1) there are usually easier ways to accomplish the same in the language without all the new functional cruft, and 2) functionalizing is done in JavaScript with piles and piles of functions that do not know, for instance, how to collapse tail recursion and all the other lazy evaluation features of language implementations designed for functional programming.

After writing the above, I will go back and read The Reactive Manifesto and General Theory of Reactivity and think about how to apply this technology. I am not one to discount a New Thing just because “programmers have been doing this for years” or because this is just the Latest Thing. Tools and conventions are being developed that could make it a better solutions for some problems – a better solution than the way I’ve always done it.

Comments