unhosted web apps
freedom from web 2.0's monopoly platforms
6. Controlling your server over a WebSocket
No Cookie Crew - Warning #4: This tutorial is the last time you still need to update your personal server using an ssh/scp client.
Web shell
Unhosted web apps are still a very new technology. That means that as a member of the No Cookie Crew, you are as much a mechanic as you are a driver. You will often be using unhosted web apps with the web console open in your browser, so that you can inspect variables and issue javascript commands to script tasks for which no feature exists. This is equally true of the services you run on your personal data server.
So far, we have already installed nodejs, "forever", a webserver with TLS support, as well as gateways to Twitter and to Facebook. And as we progress, even though we aim to keep the personal server as minimal and generic as possible, several more pieces of software and services will be added to this list. To install, configure, maintain, and (last but not least) develop these services, you will need to have a way to execute commands on your personal server.
In practice, it has even proven useful to run a second personal server within your LAN, for instance on your laptop, which you can then access from the same device or from for instance a smartphone within the same wifi network. Even though it cannot replace your public-facing personal server, because it is not always on, it can make certain server tasks, like for instance streaming music, more practical. We'll talk more about this in future posts, but for now suffice to say that if you choose to run two personal servers, then of course you need "web shell" access to both of them in order to administer them effectively from your browser.
There are several good software packages available which you can host on your server to give it a "web shell" interface. One that I can
recommend is GateOne. If you install it on Ubuntu, make sure you
apt-get install dtach, otherwise the login screen will crash repeatedly. But other than that, I have been using it as an
alternative to my laptop's device console (Ctrl-Shift-F1), with good results so far.
The GateOne service is not actually a webshell directly into the server it runs on, but rather an ssh client that allows you to ssh into
the same (or a different) server. This means you will also need to run ssh-server, so make sure all users on there have good
passwords.
Unhosted shell clients
You can host web shell software like this on your server(s), but this means the application you use (the web page in which you type your interactive shell commands) is chosen and determined at install-time. This might be OK for you if you are installing your personal server yourself and you are happy with the application you chose, but it is a restriction on your choice of application if this decision was made for you by your server hosting provider. This problem can be solved by using an unhosted shell client instead of a hosted one. The principle of separating unhosted web apps from minimal server-side gateways and services is of course the main topic of this blog series, so let's also try to apply it here.
A minimal way to implement a command shell service would be for instance by opening a WebSocket on the server, and executing commands that get sent to it as strings. You could put this WebSocket on a hard-to-guess URL, so that random people cannot get access to your server without your permission.
An unhosted web app could then provide an interface for typing a command and sending it, pretty much exactly in the same way as we did for the social dashboard we built last week.
But there are two things in that proposal that we can improve upon. First, we should document the "wire protocol", that is, the exact format of messages that the unhosted web app sends into the WebSocket, and that come back out of it in response. And second, if we open a new WebSocket for each service we add to our personal server, then this can become quite unwieldy. It would be nicer to have one clean interface that can accept various kinds of commands, and that dispatches them to one of several functionalities that run on the server.
Sockethub to the rescue
Nick Jennings, who was also at the the 2012 unhosted unconference, and who is now also here at Hacker Beach, recently received NLnet funding to start a project called Sockethub. It is a redis-based nodejs program that introduces exactly that: a WebSocket interface that lets unhosted web apps talk to your server. And through your server, they can effectively communicate with the rest of the world, both in a sending and a receiving capacity.
So let's have a look at how that would work. The command format for Sockethub is loosely based on ActivityStreams.
In that regard it is very similar to Evan Prodromou's new project, pump.io. A Sockethub command has three required fields:
rid, platform, and verb. Since multiple commands may be outstanding at the same time, the rid (request
identifier) helps to know which server response is a reply to which of the commands you sent. The platform determines which code module will handle
the request. That way, it's easy to write modules for Sockethub almost like writing a plugin for something. And finally, the verb determines which
other fields are required or optional. Some verbs may only exist for one specific platform, but others, like 'post', make sense for multiple platforms, so
their format is defined only once, in a platform-independent way wherever possible.
For executing shell commands, we will define a 'shell' platform that exposes an 'execute' verb. The way to do this in Sockethub is described in the instructions for adding a platform. The API has changed a little bit now, but at the time of writing, this meant adding a file called
shell.js into the lib/protocols/sockethub/platforms/ folder of the Sockethub source tree, with the following content:
var https = require('https'),
exec = require('child_process').exec;
module.exports = {
execute: function(job, session) {
exec(job.object, function(err, stdout, stderr) {
session.send({
rid: job.rid,
result: (err?'error':'completed'),
stdout: stdout,
stderr: stderr
});
});
}
};
We also need to add 'shell' the PLATFORMS variable in config.js, add the 'execute' verb and the 'shell' platform to
lib/protocols/sockethub/protocol.js, and add the 'execute' command to lib/protocols/sockethub/schema_commands.js with a required
string property for the actual shell command we want to have executed. We'll call this parameter 'object', in keeping with ActivityStreams custom:
...
"execute" : {
"title": "execute",
"type": "object",
"properties": {
"object": {
"type": "string",
"required" : true
}
}
},
...
Conclusion
Sockethub was still only a week old when I wrote this episode in January 2013, but it promises to be a very useful basis for many of the gateway functionalities we want to open up to unhosted web apps, such as access to email, social network platforms, news feeds, bittorrent, and any other platforms that expose server-to-server APIs, but are not directly accessible for unhosted web apps via a public cross-origin interface.
Especially the combination of Sockethub with remoteStorage and a web runtime like for instance Firefox OS looks like a promising all-round app platform. More next week!
