I don’t know why this is not documented better but…

While I was porting my old ROS2 Python test application to the newest ROS2 Foxy on my Raspberry Pi’s, I discovered that I needed to use OpenCV to capture images because picamera does not work in 64 bit](https://github.com/waveform80/picamera/issues/540) .

What’s not documented very well (and I spent several days figuring out) is that, after one has ported to OpenCV, converting an OpenCV image into a ROS2 message is not obvious. After a bunch of Google’ing, one discovers that the ROS2 folks have packaged all the magic into a module called CvBridge.

OpenCV keeps images in cv:im structures and converting that into a byte array that fits into the ROS2 message is tricky. Actually, what’s expected in a ROS2 image message is not documented very well so what works is hard to figure out in the first place.

The ROS2 developers have created some tools for the OpenCV users and one is a tool that converts OpenCV images into ROS2 image messages. So, the code ends up looking like:

from sensor_msgs.msg import Image
import cv2
from cv_bridge import CvBridge
...
self.bridge = CvBridge()
self.camera = cv2.VideoCapture(0)
self.frame_num = 0
self.publisher = self.create_publisher(Image, "topic", 10)
...
ret, frame = self.camera.read()
if ret == True:
    msg = self.bridge.cv2_to_imgmsg(frame, 'bgr8')
    msg.header.frame_id = str(self.frame_num)   # 'msg' is a ROS2 sensor_msgs/Image.msg
    self.frame_num += 1
    self.publisher.publish(msg)

That is, don’t try to create the image message yourself. Instead, rely on the CvBridge package to create the whole message and then fill it in with other information.

Comments

Since July of last year, I have been coding a metaverse-server for the Vircadia virtual worlds. The Iamus metaverse-server acts as the coordination point for the independent domain-servers which host the virtual world space.

When High Fidelity open-sourced the code for their virtual world, they kept the central coordination services private so a new service for accounts and discovery was needed.

I created Iamus with NodeJS, ExpressJS, and MongoDB.

For anyone picking up the project and making modifications, there is written documentation on the server’s design, building, running, running as a Docker image, and various notes.

In addition to the written documentation, I created a video walking through the internals of the code:

This should give anyone modifying Iamus an good start.

Comments

I’ve been generally offline for the last year. Doesn’t have anything to do with the pandemic or anything going on in the world. Just me not focusing on computers. An odd thing after all these years.

Some of the time has been spent thinking about what I want to be when I grow up. I have been thinking about formulating a 5 year plan if only so the years don’t go by with a “where did all the time go” sort of outcome.

So, where do I want to be in a few years?

WIP

In the short term, I have three software/development projects: Vircadia, Robotics, and Basil.

Vircadia Project

The Vircadia Project is an open-source virtual-world grid, server, and viewer system derived from the High Fidelity business. I wrote the Iamus metaverse-server that handles centralized domain-server registration, user accounts, and other grid level functions.

A year ago I wrote in Vircadia Directions how I wanted the project to succeed and thus jumped in to help. A year later, there is lots of work still to be done and I have lots of other projects I want to work on.

Robotics Project

I am interested in “artificial intelligence” and I want to manifest it in the real world. To that end, I have a TurtleBot3 and a Petoi Bittle for ranging around the house.

My real focus is on their intelligence. There is a lot of AI work around neural networks, training, auditing, and otherwise doing vision and context classification. Once I get these robots working, I’ll be developing a reason for them to be around the house.

Basil Project

For many years, I have been hacking on my [Herbal3d] and Basil projects. I still think it is the correct architecture for an eventual VR and AR world but I just haven’t completed it. I keep rewriting and treaking the code.

In one vlog I watched some time ago, the vlogger promoted “Always finish your projects”. There is some sense to that as directories and workbenches full of half finished projects are depressing.

I haven’t given up on Basil.

Real Life

And then there is the garden (it’s Spring!) and the house and having relationships with the other people in the house. Reality takes up a lot of time.

Social Presence

Another question is what I should do for a social presence. When I was working, I loved being part of a group of like minded people working on a goal. I would like to build up a collection of people and be part of that collection. That means, at least to start, a social media presence.

I think that means picking a platform and then commiting to generating content. The content should also be every day or close to that to create a changing narrative that people can follow and, hopefully, join in to.

Which platform? There is this blog but there are the main stream people aggregators (Twitter, Facebook, Instagram, …) as well as the less data-collecting, non-ad-supported platforms (Mastodon, WeMe, …). There are the real-time collaboration systems (Discord, Slack, …) and a hundred different categories and platforms I don’t even know exist.

Besides finding other like-minded people, the other purpose for a social presence is forcing me to do and then report something every day. It would be a forcing function to keep me making progress and doing something reportable often.

So, I need to choose a place where I’d be visible and where I’d feel like I’d need to keep up-to-date because people would see me.

Comments

As I discussed in my ideas on Vircadia Directions, it would be cool to implement virtual world federation using existing social networks. I discussed ActivityPub but the other one mentioned is Matrix.

Both ActivityPub and Matrix are open-source, non-centralized federation systems with active and busy communities. But they each seem to be built for different appliction designs and have different architectures and uses.

Let me discuss each.

Matrix

Matrix is a decentrialized infrastructure where anyone can run a “home-server” that links into the larger synchronization space. The collection of “home-server”s implement “rooms” which contain JSON objects that are signed and securely synchronized across the home servers. Matrix design leans toward security and authorization so there is a strong emphasis on identity and encryption.

The base communication system is JSON sent between clients and servers using HTTP REST operations, the model presented is of “rooms” that are magically distributed across all of the connected home-servers. Of course, there are optimizations so rooms are only synchronized across home-servers with clients that are listening to those rooms. Encryption and identity management prevent snooping and corruption.

Matrix application model leans toward the concept of chat rooms with decentralized infrastructure thus applications that mimic Discord or IRC are easy to implement. But the infrastructure has been generalized into a de-centralized, JSON object synchronization system which enables applications like IoT sensor devices reporting status and many more applications.

One downside is that, even though room contents are synchronized between homeservers that have room clients, the room contents themselves “live” on that room’s homeserver. If that one homeserver is down, then the room is unavailable. This means Matrix is a distributed communication system rather than a distributed data system.

Pros:

• Secure encrypted communication.
• “Rooms” could fit with chatting in domains.

Cons:

• Not completely distributed
• Adding a metaverse-server to the federation would involve connecting it to a Matrix home-server;
• One needs a “matrix account”. When a user creates a metaverse-server account, the metaverse-server would have to create an account on a home-server

ActivityPub

ActivityPub is a W3C standard (the group that does the web application level standards like HTML, CSS, SVG, URL, RDF, …) that defines client-to-server and server-to-server interactions which implement a decentralized “social network”. Through multiple JSON-formatted HTTP requests, content in “streams” is added to, read, edited and deleted while activity notifications and content is distributed.

The definitions of ActivityPub is very much around “micro-blogging” (think Twitter). ActivityPub defines “actors” who each have an “inbox” and “outbox”. An actor POSTs a message to their outbox and the message is sent to the inboxes of addressed recepients. Actors can also have “collections”. A defined collection is “followers” thus a message can easily be addressed to one’s “followers” (like Twitter). This would be like a Tweet that shows up in the “streams” of the receivers (if a user views their inbox like user’s view their Twitter stream).

This is how Mastodon implements a distributed micro-blogging service where many servers make up a system where one could “follow” people and bots from other servers and view one’s own stream of subscribed posts.

Using other W3C standards, identities are distributed, referenced “names” can be dereferenced WebFinger into URLs to inboxes and profiles, messages can be authorized HTTP Authentication and verified HTTP Signatures.

Several services have adapted the microblogging model to other publishing services like PeerTube and PixelFed.

ActivityPub has the same architectural feature as Matrix in that an actor/collection/inbox/outbox all “live” on one server and when that server is down, that actor cannot be accessed.

ActivityPub keeps content encrypted in transport using TLS connections but end-to-end message encryption would have to be added. This is a feature that is built into Matrix.

Pros:

• Identites can easily be created and shared across federated servers
• Actors and collections could fit with chatting in domains and groups.
• REST/JSON APIs allowing scripts easy access to users, actors, …

Cons:

• Not completely distributed
• Would have a lot of URL traffic between metaverse-servers

Comments

In my previous post ( Vircadia Directions ) on the directions I’d like to take the Vircadia project, I talked about integrating ActivityPub as the socialization backend to the metaverse-server. That has lead me to several days of researching ActivityPub, ActivityStreams, the various implmentations of same.

The idea is to use the ActivityPub system to create a “Fediverse” of virtual worlds and thus leverage an existing infrastructure and community.

The original Project Apollo metaverse-server design has a server that presents the MetaverseAPI (REST/JSON/…) and has a custom designed backend database to store information about accounts, domains, account relationships (friends, …), and domain connection information (ICE servers, chat servers, …).

The diagram to the left shows this structure (with the “domain-servers” simplified into one box that includes the “assignment clients”). In this structure, the only interface that is “standard” is the “MetaverseAPI”.

In thinking about adding ActivityPub, I first thought I’d add a MongoDB database on the backend (since everything is JSON) and an ActivityPub interface on the frontend. This would allow the existing domain-servers to talk to the metaverse-server and any other metaverse-servers and ActivityPub application to communicate with the users in that metaverse.

But this would mean writing a lot of code (the ActivityPub interface) just to interface with the metaverse database. Looking around, there are several system implmenting ActivityPub and some services that could just be used.

But then, I realized that the metaverse-server just creates and manages accounts and manages the domain-server interconnections. What if I just considered the metaverse-server a MetaverseAPI to ActivityPub protocol converter?

Since the ActivityPub protocol already has all the functions for creating and managing account as well as functions to create and and manage collections of objects (like domains), an ActivityPub server could act as the database for the metaverse-server.

ActivityPub servers already have authentication and access control conventions.

I first thought that I could just use a Mastodon server as the backend metaverse database but I believe that we will want to be extending the functionality of the server. This work is extending the ActivityPub social network model into virtual worlds so there will be special features like chat groups that only work with the people close together. Thus, location and distance need to be added to status and filtering.

At the moment, I am considering the ActivityPub-Express project which is a pretty well written NodeJS/ExpressJS/MongoDB based ActivityPub server.

Comments