Menu Close

ianmjones

Always Developing
actordb-for-docker

ActorDB for Docker

Last weekend I re-stumbled across ActorDB, a very interesting distributed database system that scales near linearly by scoping work to “Actors”. Every database action, whether that’s an insert, select, update, delete or whatnot, always starts by specifying the Actor. This effectively scopes the changes to a single database, which then replicates to its sibling databases on other ActorDB nodes in a cluster. There’s a lot more to it than that, as you can of course work with multiple actors within a transaction, and can connect to any node, in any cluster, and the data will be routed properly. In short, think user centric databases all clustered and replicated together, using the SQLite engine backed by LMDB storage and using Raft for consensus. There’s a little buzz-word bingo for you.

After reading everything I could find on ActorDB, and still having a bit of time to kill before having to go transfer our daughter from one place to another yet again, I figured I’d have a quick play with it.

As is rapidly becoming my norm, I went to Docker Hub to find a Docker image to spin up. To my utter shock and horror (there might be a little bit of exaggeration there), there was not a single image for ActorDB to be found!

My first thought was that maybe there was a very good reason for there not being a Docker image available, but after a little look around that there interwebs I came to the conclusion that it was simply because it’s a young project, and generally run on real machines in production. Nothing made me think it was a terrible idea to run ActorDB in a Docker container, so why not give it a bash? ActorDB for Docker was born!

The Dockerfile

Having recently done a bunch of work for Delicious Brains that involved using Docker, I wasn’t too phased by the idea of spinning up a new Dockerfile, and luckily there’s an ActorDB package for Debian.

I built a basic image based on Debian Jessie, and then logged into it and experimented with the steps I needed to run to get ActorDB installed and running. It took much less time than I expected as ActorDB is pretty self contained, the biggest issues I had were with making sure curl had all its dependencies in place to actually download the .deb file from within the running container (it needs some extra root certificates installed via a package).

I also rather stupidly forgot that the actordb program runs as a daemon by default, so the container kept just running and exiting cleanly. Once I realised what was going on, I simply ran the container with --entrypoint /bin/bash and then ran actordb without any arguments to get its usage info. Turns out you can run actordb forground rather than actordb start to get it to run in the current shell. Just what I needed.

I’ve been working with a PhantomJS image recently, and noticed that it used a script to ensure that the phantomjs binary did not run as the first process in the container (PID 1) as it may cause issues with shutting it down. So I nicked that idea.

The final Dockerfile is pretty simple, makes sure to expose the required ports and volume primary paths that ActorDB uses, and the image is available as an automated build on Docker Hub.

Docker Compose

A distributed database is no good without multiple nodes to distribute its data around, and of course I wanted to ensure the image could be used in a network of containers. I used Docker Compose to set up a very rudimentary cluster of three nodes, you can see the setup in the project root’s docker-compose.yml file.

While the containers seemed to work (on second start, see issue #1 for details), could ping each other by their service names (e.g. “actordb-server-1” could ping “actordb-server-2”), I spent ages trying to get the instances of ActorDB to actually replicate data between them. I eventually sorted it out with the following two important realisations:

  1. The -name setting in each node’s vm.args file must have a unique name before the “@”, the ip address/domain name after the “@” doesn’t make it unique.
  2. You must update your initialisation script that is run on the “leader” to also register the other nodes in the cluster (you can register them manually later, but will need to copy the leader’s lmdb file to them first).

Once I’d worked that out, I was super happy to have a fully working cluster of ActorDB nodes!

Quick Example

The following text is a quick run through of bringing up a cluster, initialising it, writing some data to the first node, switching to the second node and showing the data replicated from the first, creating some more data, and then showing it all on the third node, but using the MySQL client. Enjoy!

Ians-MBP:actordb-for-docker ian$ ./up.sh 
Creating network "actordbfordocker_default" with the default driver
Creating actordbfordocker_actordb-server-1_1
Creating actordbfordocker_actordb-server-2_1
Creating actordbfordocker_actordb-server-3_1

actordb-server-1 – Use actordb_console to run the SQL script that initialises the database:

Ians-MBP:actordb-for-docker ian$ docker-compose exec actordb-server-1 actordb_console -f /etc/actordb/init.example.sql
Config updated.
Config updated.
Schema updated.

actordb-server-1 – Use actordb_console to show the schema, create and select some data:

Ians-MBP:actordb-for-docker ian$ docker-compose exec actordb-server-1 actordb_console -u myuser -pw mypass
*******************************************************************
Databases:
use config (use c)  initialize/add nodes and user account management
use schema (use s)  set schema
use actordb (use a) (default) run queries on database
*******************************************************************
Commands:
open         (windows only) open and execute .sql file
q            exit
h            print this header
commit (c)   execute transaction
rollback (r) abort transaction
print (p)    print transaction
show (s)     show schema
show status  show database status
show queries show currently running queries
show shards  show shards on node
*******************************************************************

actordb> actor type1(hello_world);
actordb (1)> insert into tab(id, txt) values(1,'Hello World');
actordb (2)> c
Error: {error,{nocreate,<<"Query without create flag was attempted on an actor which does not exist.">>}}
actordb> show
****************************************************************************************************************************************************************
sql                                                                                                                                                 type       |
----------------------------------------------------------------------------------------------------------------------------------------------------------------
$CREATE TABLE tab (id INTEGER PRIMARY KEY, txt TEXT);                                                                                               type1      |
$CREATE TABLE tab1 (id INTEGER PRIMARY KEY, txt TEXT);                                                                                              type1      |
$ALTER TABLE tab ADD i INTEGER;                                                                                                                     type1      |
$CREATE TABLE tabx (id INTEGER PRIMARY KEY CHECK (typeof(id) == 'integer'), txt TEXT CHECK (typeof(id) == 'text'));                                 type1      |
$CREATE TABLE asdf (id INTEGER PRIMARY KEY AUTOINCREMENT, txt BLOB);                                                                                type2      |
$CREATE TABLE actors (id TEXT PRIMARY KEY, hash INTEGER, val INTEGER) WITHOUT ROWID;                                                                counters   |
$CREATE TABLE actors (id TEXT PRIMARY KEY, hash INTEGER, size INTEGER)  WITHOUT ROWID;                                                              filesystem |
$CREATE TABLE users (id INTEGER PRIMARY KEY AUTOINCREMENT, fileid TEXT, uid INTEGER, FOREIGN KEY (fileid) REFERENCES actors(id) ON DELETE CASCADE); filesystem |
----------------------------------------------------------------------------------------------------------------------------------------------------------------
actordb> actor type1(hello_world) create;                 
actordb (1)> insert into tab(id, txt) values(1,'Hello World');
actordb (2)> c
Rowid: 1, Rows changed: 1
actordb> actor type1(hello_world) create;                 
actordb (1)> select * from tab;
actordb (2)> c
*********************
i    id txt         |
---------------------
null 1  Hello World |
---------------------
actordb> q
Bye!     

actordb-server-2 – Use actordb_console to see the data created on actordb-server-1, and create some more:

Ians-MBP:actordb-for-docker ian$ docker-compose exec actordb-server-2 actordb_console -u myuser -pw mypass
*******************************************************************
Databases:
use config (use c)  initialize/add nodes and user account management
use schema (use s)  set schema
use actordb (use a) (default) run queries on database
*******************************************************************
Commands:
open         (windows only) open and execute .sql file
q            exit
h            print this header
commit (c)   execute transaction
rollback (r) abort transaction
print (p)    print transaction
show (s)     show schema
show status  show database status
show queries show currently running queries
show shards  show shards on node
*******************************************************************

actordb> actor type1(hello_world) create;
actordb (1)> select * from tab;
actordb (2)> c
*********************
i    id txt         |
---------------------
null 1  Hello World |
---------------------
actordb> actor type1(hello_world) create;
actordb (1)> insert into tab(id, txt) values(2, 'ActorDB Rules!');
actordb (2)> c
Rowid: 2, Rows changed: 1
actordb> q                                                    
Bye!     

actordb-server-3 – Use the mysql client to connect to actordb-server-3’s MySQL protocol port:

Ians-MBP:actordb-for-docker ian$ mysql -u myuser -p -h127.0.0.1 -P33337
Enter password: 

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 0
Server version: 5.5.0-myactor-proto 5.5.0-myactor-proto

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> actor type1(hello_world) create; select * from tab;
Query OK, 0 rows affected (0.01 sec)

+------+----------------+------+
| id   | txt            | i    |
+------+----------------+------+
|    1 | Hello World    | NULL |
|    2 | ActorDB Rules! | NULL |
+------+----------------+------+
2 rows in set (0.00 sec)

mysql> actor type1(hello_world) create; update tab set i = id * 100;
Query OK, 0 rows affected (0.00 sec)

Query OK, 2 rows affected (0.02 sec)

mysql> actor type1(hello_world) create; select * from tab;
Query OK, 0 rows affected (0.01 sec)

+------+----------------+------+
| id   | txt            | i    |
+------+----------------+------+
|    1 | Hello World    |  100 |
|    2 | ActorDB Rules! |  200 |
+------+----------------+------+
2 rows in set (0.00 sec)

mysql> exit;
Bye
db-reactivewpelm-p3

Building Reactive WordPress Plugins – Part 3 – Elm

In the final part of the Building Reactive WordPress Plugins series across on the Delicious Brains blog, I explore using Elm rather than JavaScript to write the front end of WP Cron Pixie, the small WordPress dashboard widget for showing what’s in the WordPress cron.

I had more fun writing the code for this article than any other in the series. I hope you enjoy reading it and maybe it gets you thinking about trying out Elm for your next web frontend.

options-pixie-pro-released-finally-banner

Options Pixie Pro Released, Finally.

Well, I finally did it, I got Options Pixie Pro 1.0 for WordPress finished and released!

After many late nights (but no weekends), I finally got to a point where I was very happy with how Options Pixie Pro was working, and confident that it provided enough value to be worthy of release.

It’s pretty awesome to have a new product out in the wild, it’s been over 10 years since I released my last product on my own, through pinching time here and there in the evenings.

It feels a bit weird too when I realise that Options Pixie Pro is new to everybody. I’ve been using it for months during my normal work day, especially when testing WP Offload S3 as it’s been perfect for inspecting the options records created for its settings and processing queues. It’s very handy too when you need to add, edit or delete options records to mimic various scenarios while testing a plugin.

Favourite Feature

I think one of my favourite features of Options Pixie Pro is how you can edit a Base64 encoded serialized array, or fix a broken serialized array hidden behind Base64 encoding. Options Pixie Pro lets you see the hidden serialized string, edit or auto-fix it, and re-encodes it for you on save. I can only assume that the reason theme developers Base64 encode their settings is to preserve extended character sets regardless of MySQL’s character encoding settings, but now you can edit and auto-fix those values. And if you can’t be bothered to count the number of characters in the string you’ve just edited deep within a serialized array or object string, no problem, as long as the format is still valid Options Pixie Pro will fix those pesky character counts for you on save.

Row Actions

Of course, there are row level and bulk actions for fixing serialized values…

Options Pixie Pro - Fix Serialized Row Action

Options Pixie Pro - Fixed Serialized Record

Bulk Actions

And the same goes for row and bulk actions for deleting records…

Options Pixie Pro - Delete Row Action

Multisite Support

And if you manage a multisite install of WordPress, you can inspect, add, edit, delete and fix options records across all your subsites…

Options Pixie Pro - Multisites Supported

Quick Demo

Below I’ve embedded a quick animated gif of me exercising some of the features of Options Pixie Pro, click it to get a larger format version.

options-pixie-pro-demo

Learning Experience

It’s been quite some learning experience. There’s a lot involved in developing any kind of software product that you intend to sell and support. Apart from the actual software development, getting a website set up to sell direct downloads is generally more involved than it might look at first. But the good thing is, once it’s set up for the first product, the second should be a lot easier to add.

If you’re looking to “List, sort, search, view, add, edit, delete and fix your WordPress site’s options records with style”, then please pop over to my business site and take a look at Options Pixie Pro!

db-reactivewpbackbonejs-p1

Building Reactive WordPress Plugins – Part 1 – Backbone.js

I’m writing a short series for Delicious Brains on building reactive style frontend WordPress plugins. Part 1, which uses Backbone.js on the frontend, just went live.

This first article is rather long as it introduces the plugin and its backend as well as how to use Backbone.js. Follow up articles should be shorter as each one will concentrate on building the frontend with a different technology, and won’t deal with the backend so much.

banner-1544x500

My First WordPress Plugin: Options Pixie

There’s been a number of times when I’ve wanted to quickly check the value of a record in the wp_options table of a WordPress site’s database, but not had easy access to the database, usually when the site and database are on a remote server. I’ve tried using the /wp-admin/options.php page, but it’s not ideal and does not show the contents of serialized values, which of course is where so many interesting values lie in wait.

There are of course a few existing WordPress plugins for working with the wp_options database table, but none have quite fit my needs, either by not showing values in a usable way, not having good search and sort, or simply throwing a million errors when in debug mode (which you always seem to want on when the need arises for checking your options records).

Although I’ve been developing WordPress plugins for Delicious Brains since mid 2014, I’ve never actually released a plugin of my own. That doesn’t seem right, especially as most of my team mates have excellent plugins of their own.

So I’ve scratched my itch as they say, and have developed and released my first open source WordPress plugin called Options Pixie.
Options Pixie List View

Options Pixie Rich View
Actually, as I write this, Options Pixie has been released and available from the WordPress plugins repository, GitHub and my own business site for over 7 months, as I released it at the beginning of July 2015. I guess the fact that I haven’t spoken about it here, only once on twitter, is a testament to my awesome marketing skills. Maybe not.

I’m not sure why I haven’t been jumping up and down and shouting about Options Pixie, as I’m really quite proud of it. It works very well for what I need it to do, and I went to great pains to make it a very high quality and robust WordPress plugin. Maybe I can just blame my natural and very British reserve?

Regardless, I’ve finally finished off this post (I started it just a few days after releasing Options Pixie, 7 months ago), and encourage anyone that finds themselves in the position of needing to check the contents of their WordPress site’s options table to search the WordPress plugins repository and install Options Pixie. It’s currently at version 1.0.1 having had a few bug fixes, and there’s a new version in the wings which improves some of the behind the scenes functionality, enhances working with base64 encoded values, and is tested with the latest versions of WordPress.

Here’s the highlights of what Options Pixie offers:

List, filter, sort and view options records, even serialized and base64 encoded values.

  • List, sort and search options
  • “Rich view” of serialized and JSON string values
  • Works with base64 encoded serialized and JSON string values
  • Highlights broken serialized values
  • Supports Multisites

With Options Pixie you can find out what is really going on with your WordPress options.

I’m also working on Options Pixie Pro, a paid addon that adds bulk actions such as delete and fix serialized, add, edit and delete functionality, and of course priority email support.

photo-1421091242698-34f6ad7fc088

Our Struggles to Stay Healthy While Working From Home

Recently in a Slack chat we discussed how we try and stay healthy while working from home. We all have different methods, some have been consistent and dedicated to staying fit and healthy, some not so much. We’ve all decided that we’re going to “air our laundry” with this post, whether it’s clean or not, …

Miised Three Banner

My First iOS Game: Missed Three

Today I had an email from Apple to say that my first iOS game, Missed Three, had entered “In Review” status, which I thought was pretty good as it’d only been submitted for review a week ago. Then, just 30 minutes later I received another email to tell me that Missed Three was ready for sale through the App Store.

I guess when you make what is probably the simplest game on the App Store, it doesn’t take very long to review! 😉

It really is a very simple game, keep tapping the targets as they appear, when you’ve missed a total of three the game is over. Of course, it starts off at a reasonably slow pace, giving you a little while to tap a target before it disappears, but as the game goes on the targets disappear in an ever shorter time. It really does get pretty hectic if you’re doing well.

If you get a new high score, you get an opportunity to share it with a friend by sending them an email. I intentionally did not add sharing by Twitter, Facebook or any other social media as I don’t like seeing those kinds of things pollute my timeline, so why would I make it easy to do that in my game? Just challenge your friends directly, you know who will or will not take up the challenge, so why annoy anyone else?

If you’re looking for something short and sweet to challenge friends with, you can purchase Missed Three from the App Store.

Older Posts