Recipes#

MacArthur! I can’t even get McDonald’s!

—Julian Simon

Networked Statecharts#

Drawing Statecharts#

UMLet is a free tool that can be used to make statechart diagrams (see their statemachine panel). You can run UMLet from most operating systems or from your web browser using UMLetino. Here is their training video:

You can adjust their default templates to improve your drawing experience.

To export an UMLet picture from their native uxf to other formats:

umlet.exe -action=convert -format=pdf -filename=<your_file_name>.uxf
umlet.exe -action=convert -format=svg -filename=<your_file_name>.uxf

Making a NetworkedActiveObject#

If you would like to make polyamorous statecharts using the flat method technique then the NetworkedActiveObject is for you.

from miros_rabbitmq import NetworkedActiveObject

nao = NetworkedActiveObject(
        "name_of_statechart",
        rabbit_user="<rabbitmq_user_name>",
        rabbit_password="<rabbitmq_password>",
        tx_routing_key="heya.man",
        rx_routing_key="#.man",
        mesh_encryption_key=b'u3u...')

To make your instrumentation easier to understand, ensure that all the nodes in your distributed system have different names.

The rabbit_user and rabbit_password credentials need to match that of your RabbitMQ server.

To build your own encryption key(s) see this. You can set different encryption keys for your instrumentation networks; the snoop trace network and the snoop spy network. If you do not specify these keys, these networks will use the mesh_encryption_key.

If you want to specify different encryption keys for your instrumentation networks:

from miros_rabbitmq import NetworkedActiveObject

nao = NetworkedActiveObject(
        "name_of_statechart",
        rabbit_user="<rabbitmq_user_name>",
        rabbit_password="<rabbitmq_password>",
        tx_routing_key="heya.man",
        rx_routing_key="#.man",
        mesh_encryption_key=b'u3u...',
        snoop_trace_encryption_key=b'u4f...',
        snoop_spy_encryption_key=b's44...',
        )

The nao object will have a transmit method that can be used to put event’s into all of the other subscribed statecharts across the network. To build your statechart using the nao object, you would follow the rules used by an ActiveObject.

Making a NetworkedFactory#

If you have a preference to build up statecharts using a Factory, but you want your statecharts to work together across a network, then the NetworkedFactory is for you:

from miros_rabbitmq import NetworkedFactory

nf = NetworkedActiveFactory(
        "name_of_statechart",
        rabbit_user="<rabbitmq_user_name>",
        rabbit_password="<rabbitmq_password>",
        tx_routing_key="heya.man",
        rx_routing_key="#.man",
        mesh_encryption_key=b'u3u...')

To make your instrumentation easier to understand, ensure that all the nodes in your distributed system have different names.

The rabbit_user and rabbit_password credentials need to match that of your RabbitMQ server.

To build your own encryption key(s) see this. You can set different encryption keys for your instrumentation networks; the snoop trace network and the snoop spy network. If you do not specify these keys, these networks will use the mesh_encryption_key.

If you want to specify different encryption keys for your instrumentation networks:

from miros_rabbitmq import NetworkedActiveObject

nf = NetworkedFactory(
        "name_of_statechart",
        rabbit_user="<rabbitmq_user_name>",
        rabbit_password="<rabbitmq_password>",
        tx_routing_key="heya.man",
        rx_routing_key="#.man",
        mesh_encryption_key=b'u3u...',
        snoop_trace_encryption_key=b'u4f...',
        snoop_spy_encryption_key=b's44...',
        )

The nf object will have a transmit method that can be used to put event’s into all of the other subscribed statecharts across the network. To build your statechart using the nf object, you would follow the rules used by a miros Factory.

Making Unique Names#

A simple way to make a unique name:

import uuid

def make_name(post):
  return str(uuid.uuid4())[0:5] + '_' + post

make_name('bob') # => 0ca32_bob

Transmitting an Event#

To transmit an event to other connected statecharts use the transmit method. This method works the same for a NetworkedActiveObject and a NetworkedFactory:

from miros.event import signals, Event, return_status

# where 'chart' is a NetworkedFactory or NetworkedActiveObject object
def some_function(chart, e):
  chart.transmit(Event(signal=signals.other_name_of_signal))
  return return_status.HANDLED

When a networked statechart receives this event, it will be posted into it’s first in first out queue. The statechart will not be able to distinguish that it was an event coming back from the network.

Follow some sort of signal naming-standard so that you know if your events are generated locally or from somewhere else on the network. In this example, the signal is pre-pended with the word other_, for this reason.

Transmitting a Payload in an Event#

All information that is passed between networked nodes is serialized, encrypted, decrypted and de-serialized. Therefore you can pass an object into your payload that can be serialized using the standard pickle algorithm for python 3. So to pass objects between your statecharts, you can just put them into the payload of an event.

from miros.event import signals, Event, return_status

# where 'chart' is a NetworkedFactory or NetworkedActiveObject object
def some_function(chart, e):
  chart.transmit(Event(signal=signals.other_with_payload, payload="a string"))
  return return_status.HANDLED

Making an Encryption Key#

The miro-rabbitmq library uses Fernet symmetric encryption. The same keys need to be used by all nodes.

To make a new key:

from cryptography import Fernet
new_encryption_key = Fernet.generate_key()

print(new_encryption_key) # => b'u3u...' # => copy this

Snoop Trace#

# nsc could be a NetworkedActiveObject or a NetworkedFactory
nsc.enable_snoop_trace()
nsc.start_at(<state_you_want_to_start_at>)

Trace instrumentation is useful for seeing what events caused what state transitions. They provide you with information about how your design is reacting to its incoming events.

The snoop trace is an extension of this idea. It is a network which connects all of the live trace streams of all of the statecharts that are working together. You can see the live trace instrumentation of all of the contributing nodes, by opting into this snoop trace network. To do this, you call enable_snoop_trace prior to starting your statechart (see the code listing above).

It doesn’t take long before this information is hard to see. For this reason, I coloured the node names. The name of your local machine’s statechart will be blue, and any other name will be purple in your snoop trace output. You can see this colouring in the following video:

In the video, I have logged into two different computers that are running the same statechart program as part of a distributed system. You can see that the blue names are different in each terminal window, because we are viewing the system from two different computers who’s statecharts have their own local name.

To print something into the snoop trace use the snoop_scribble.

If you want to log your snoop trace enable your trace without the ANSI color codes:

# nsc could be a NetworkedActiveObject or a NetworkedFactory
nsc.enable_snoop_trace_no_color()
nsc.start_at(<state_you_want_to_start_at>)

You can use the results of your snoop trace to generate network sequence diagrams for you.

The trace output hides a lot of the details about how your event processor is searching your statechart to determine how to react to an event. For this reason, some statechart dynamics will be invisible to the trace stream; like a hook. If you want to see everything that your statechart is doing, or if you want to see everything that your entire network is doing turn on the snoop_spy.

Snoop Scribble#

Use the snoop scribble to output custom strings into the snoop trace and snoop spy networks.

ao.snoop_scribble("broadcast to all monitoring snoop programs")

The snoop_scribble output is coloured a dark grey so it can be distinguished from the other parts of the network instrumentation.

If you want to turn off this colouring:

ao.snoop_scribble("broadcast to all monitoring snoop programs",
                   enable_color=False)

Drawing Sequence Diagrams#

The text output of a snoop trace can be used to generate sequence diagrams using the sequence tool:

Snoop Spy#

# nsc could be a NetworkedActiveObject or a NetworkedFactory
nsc.enable_snoop_spy()
nsc.start_at(<state_you_want_to_start_at>)

The spy instrumentation shows you all of the work done by an event processor. This really isn’t that useful since you will be drown in information. If you are having issues with your statechart, you should debug it as a singular instance before you connect it to a distributed system. But, if your problem demands that you see everything that is going on at once, the snoop spy network provides this capability.

Instead of viewing everything, you could turn on the spy instrumentation on one machine and have it routed to another machine. The snoop spy network has an opt in model, to snoop on others, you need to let them snoop on you. So to route another machine’s snoop information onto your machine, you will need to release your information into the snoop spy network. This may clutter your log stream with unwanted information. If this is a concern, you can create a grep filter using the name of the other node that you are trying to monitor.

It is easy to lose track of the context of what is going on while viewing a snoop spy, for this reason you might want to enable the snoop spy and snoop trace at the same time. You can use the snoop trace output to delimit the deluge of spy information within the context of state transitions.

The name of the local nodes in the distributed system will appear blue and the names of other nodes will appear purple.

Logging The Snoop Trace#

To removed the ANSI clutter from your log.txt file you can do something like this (cargo-culted from stack overflow):

python3 networkable_active_object.py 2>&1 | \
   sed -r 's/'$(echo -e "\033")'\[[0-9]{1,2}(;([0-9]{1,2})?)?[mK]//g' | \
   tee log.txt grep -F [+s] log.txt | grep <name>

Or, turn the colour off when you enable the snoop trace in your code:

nsc.enable_snoop_trace_no_color()
nsc.start_at(<state_you_want_to_start_at>)

Then write your trace information to the log.txt without the command-line complexity:

python3 networkable_active_object.py >> log.txt

Logging the Snoop Spy#

To removed the ANSI clutter from your log.txt file you can do something like this (cargo-culted from stack overflow):

python3 networkable_active_object.py 2>&1 | \
   sed -r 's/'$(echo -e "\033")'\[[0-9]{1,2}(;([0-9]{1,2})?)?[mK]//g' | \
   tee log.txt grep -F [+s] log.txt | grep <name>

Or, turn the colour off when you enable the snoop spy in your code:

nsc.enable_snoop_spy_no_color()
nsc.start_at(<state_you_want_to_start_at>)

Then write your spy information to the log.txt without the command-line complexity:

python3 networkable_active_object.py >> log.txt

Credentials and Encryption Keys#

Changing your RabbitMQ credentials#

If you are installing RabbitMQ using the ansible script in the DevOps section, set the rabbit_name to the username you want, and the rabbit_password to whatever you want your password to be. Update your code to use your new credentials.

You can also change your user name and password with the RabbitMQ management GUI.

Managing your Encryption Keys and RabbitMQ Credentials (Short Version)#

Use this deployment strategy to ensure a .env file is in your top level directory of your application (the same directory you would find your .git directory). Your RabbitMQ credentials and your miros-rabbitmq encryption keys will be in this .env file, which will look something like this:

MESH_ENCRYPTION_KEY=u3Uc-qAi9iiCv3fkBfRUAKrM1gH8w51-nVU8M8A73Jg=
SNOOP_TRACE_ENCRYPTION_KEY=u3Uc-qAi9iiCv3fkBfRUAKrM1gH8w51-nVU8M8A73Jg=
SNOOP_SPY_ENCRYPTION_KEY=u3Uc-qAi9iiCv3fkBfRUAKrM1gH8w51-nVU8M8A73Jg=
RABBIT_USER=peter
RABBIT_PASSWORD=rabbit
RABBIT_PORT=5672
RABBIT_HEARTBEAT_INTERVAL=3600
CONNECTION_ATTEMPTS=3
RABBIT_GUEST_USER=rabbit567

Then in your setup.py or your actual application code, include the following:

import os
from dotenv import load_dotenv
from pathlib import Path

# write .env items to the environment
env_path = Path('.') / '.env'
if env_path.is_file():
  load_dotenv(env_path)
else:
  # recurse outward to find .env file
  load_dotenv()

RABBIT_USER = os.getenv('RABBIT_USER')
RABBIT_PASSWORD = os.getenv('RABBIT_PASSWORD')
MESH_ENCRYPTION_KEY = os.getenv("MESH_ENCRYPTION_KEY")
SNOOP_TRACE_ENCRYPTION_KEY = os.getenv("SNOOP_TRACE_ENCRYPTION_KEY")
SNOOP_SPY_ENCRYPTION_KEY = os.getenv("SNOOP_SPY_ENCRYPTION_KEY")
# .. etc

For details about how to set up your .env file without the mentioned deployment procedure read the next section.

Managing your Encryption Keys and RabbitMQ Credentials (Long Verion)#

To keep your encryption keys and RabbitMQ credentials out of your source code, you can put them into environment variables, then load the contents of these environment variables into your program.

A slight extension of this idea is to put your keys into a .env file, then load its contents into environment variables, then load these variables into your program. By placing your encryption keys and RabbitMQ credentials into a .env file, it makes it easier to transfer them between the machines in your distributed system.

Note

By convention the .env file is kept in the outermost directory of your project, the same directory that you would find your .git folder.

The python-dotenv package converts the contents of a .env file into environment variables. We will use it in this recipe. If you have installed miros-rabbitmq, you will have this package installed on your system already.

In the outermost directory of your project, create a setup.py file and add the following:

import os
from dotenv import load_dotenv
from pathlib import Path

# write .env items to the environment
env_path = Path('.') / '.env'
if env_path.is_file():
  load_dotenv(env_path)
else:
  # recurse outward to find .env file
  load_dotenv()

# get RabbitMQ credentials and encryption keys from the environment
RABBIT_USER = os.getenv('RABBIT_USER')
RABBIT_PASSWORD = os.getenv('RABBIT_PASSWORD')
MESH_ENCRYPTION_KEY = os.getenv("MESH_ENCRYPTION_KEY")
SNOOP_TRACE_ENCRYPTION_KEY = os.getenv("SNOOP_TRACE_ENCRYPTION_KEY")
SNOOP_SPY_ENCRYPTION_KEY = os.getenv("SNOOP_SPY_ENCRYPTION_KEY")

This program will be able to read your encryption keys and RabbitMQ credentials, from either your shell’s environment variables or from a .env file.

For the setup.py program to work there needs to be a .env file, so let’s make an empty one:

$ touch .env

Now we have options. We can put our keys into our environment using shell commands, or we can put them into the .env file.

Let’s explore these options:

  • I will add the encryption keys to the shell’s environment

  • Get the program working

  • Remove the encryption keys from the shell’s environment

  • Show that the program crashes

  • Add the encryption keys to the .env file

  • Show that the program works

  • Talk about how to transfer the .env file securely.

To create an environment variable we use the shell’s export command.

$ export MESH_ENCRYPTION_KEY=u3Uc-qAi9iiCv3fkBfRUAKrM1gH8w51-nVU8M8A73Jg=
$ export SNOOP_TRACE_ENCRYPTION_KEY=u3Uc-qAi9iiCv3fkBfRUAKrM1gH8w51-nVU8M8A73Jg=
$ export SNOOP_SPY_ENCRYPTION_KEY=u3Uc-qAi9iiCv3fkBfRUAKrM1gH8w51-nVU8M8A73Jg=
$ export RABBIT_USER=peter
$ export RABBIT_PASSWORD=rabbit

In our main program, we can access these encryption keys by importing them from our setup.py file:

# in miros-rabbitmq file
from setup import MESH_ENCRYPTION_KEY
from setup import SNOOP_TRACE_ENCRYPTION_KEY
from setup import SNOOP_SPY_ENCRYPTION_KEY
from setup import RABBIT_USER
from setup import RABBIT_PASSWORD

Now, when we construct a NetworkedActiveObject or a NetworkedFactory object we can use our environment variables rather than writing our secret encryption keys directly into our code base.

# in miros-rabbitmq file
ao = NetworkedActiveObject(
  make_name('ao'),
  rabbit_user=RABBIT_USER,
  rabbit_password=RABBIT_PASSWORD,
  tx_routing_key='heya.man',
  rx_routing_key='#.man',
  mesh_encryption_key=MESH_ENCRYPTION_KEY,
  snoop_spy_encryption_key=SNOOP_SPY_ENCRYPTION_KEY,
  snoop_trace_encryption_key=SNOOP_TRACE_ENCRYPTION_KEY)

If we set up our environment variables the same on all of our machines, our distributed system will work.

Before we add our encryption keys to our .env file, lets first confirm that we can break our program by removing them from our shell environment. In your shell type:

$ unset RABBIT_USER
$ unset RABBIT_PASSWORD
$ unset MESH_ENCRYPTION_KEY
$ unset SNOOP_TRACE_ENCRYPTION_KEY
$ unset TRACE_TRACE_ENCRYPTION_KEY

Then if we re-run the program we will see that it crashes for lack of encryption keys. By confirming that our program has crashed, we can trust that the information that we will put into our .env file will be used rather than the information we had previously placed in our environmental variables (we won’t fool ourselves into thinking are .env file is working if it isn’t).

Let’s move our encryption keys into the .env file.

# in .env file
MESH_ENCRYPTION_KEY=u3Uc-qAi9iiCv3fkBfRUAKrM1gH8w51-nVU8M8A73Jg=
SNOOP_TRACE_ENCRYPTION_KEY=u3Uc-qAi9iiCv3fkBfRUAKrM1gH8w51-nVU8M8A73Jg=
SNOOP_SPY_ENCRYPTION_KEY=u3Uc-qAi9iiCv3fkBfRUAKrM1gH8w51-nVU8M8A73Jg=
RABBIT_USER=peter
RABBIT_PASSWORD=rabbit

Warning

Don’t put the .env file into your code revision system. If you do, you might as well leave the encryption keys in your source code.

Add .env to your .gitignore file so you don’t accidentally add it in the future.

If you have added .env to git, remove it: git rm --cached .env, then change your encryption keys.

Note

There is a .env file in the examples directory of this code base. It was included so you can see how to do things.

Now we re-run our program and confirm that it works.

This .env file can be shared between machines, but transfer it using scp, or using a flash key or in other ways that you can keep your secrets, secret. Don’t use email.

Here is a reminder about how to use scp:

# scp <source> <destination>
$ scp /path/to/.env <username>@ip/path/to/destination/.env

If you use this deployment strategy, the .env files will be placed at the top level of all of working directory on all of the machines in your distributed system.

Networking#

The problem of how to network your distributed system can be broken into two different pieces:

  • how to deploy your servers, code, secrets and infrastructure

  • how a node can discover other nodes in the same system

To read about a deployment strategy see this deployment example.

A node determines what other nodes it can talk to, by using two different strategies:

  • it uses a list of addresses in the .miros_rabbit_hosts.json file.

  • it automatically discovers other nodes like itself on your LAN, then it caches this information into the .miros_rabbitlan_cache.json file.

The .miros_rabbit_hosts.json and .miros_rabbitlan_cache.json files are kept in the same directory as the code that is using the miros-rabbitmq package.

Manually setting Node Addresses#

The .miros_rabbit_hosts.json file in the same directory as your miros-rabbitmq program looks like this:

{
  "hosts": [
    "192.168.1.71",
    www.myurl.xyz
  ]
}

This file will be written for you at your top level directory if you use this deployment strategy.

Node Discovery on your LAN#

This is taken care of by the library. If your node is on your local area network and it’s IP address is in the ARP table or it’s computer will respond to a ping; The miros-rabbitmq library should find it. If two nodes are on the same network and share encryption and rabbit credentials they will be able to communicate.

The results of the process are cached in the .miros_rabbitlan_cache file which looks something like this:

{
  "addresses": [
    "192.168.1.75",
    "192.168.1.71"
  ],
  "amqp_urls": [
    "amqp://bob:dobbs@192.168.1.75:5672/%2F?connection_attempts=3&heartbeat_interval=3600",
    "amqp://bob:dobbs@192.168.1.71:5672/%2F?connection_attempts=3&heartbeat_interval=3600"
  ],
  "time_out_in_minutes": 30
}

To force your program to update this cache, change the time_out_in_minutes setting to 0 and re-run your program. Your application will cause another LAN search and update the time_out_in_minutes back to it’s default setting.

Aliens, and what to do about them#

An Alien is a node that can talk to you, but that you didn’t know about when your program started. There are many reasons why you might not know about a node that knows about you:

  • Your LAN cache hadn’t expired while a new node was added to your LAN

  • The Alien machine does not respond to ping requests

  • the machine is beyond your LAN and you don’t have it’s address in your .miros_rabbit_hosts file.

The Alien policy is, if it can talk to us we will talk to it too. If the node knows about us, has the correct encryption keys and RabbitMQ credentials, then it’s secure and we can work with it. So, if an Alien is discovered, the miros-rabbitmq package will respond by building producers to talk to it. To see how this is done you can read this.

On this page: