Erlang, Cowboy and Batman.js for Building Web Applications

I’ll have a complete walk of through using Cowboy and Batman.js to build the TodoMVC clone in a few days. For now I have the slides from my talk at the Chicago Erlang User Group:

Chicago Erlang User Group April, 4th 2012

I couldn’t get iframe embedding to work with WordPress… So if anyone knows what that is up with please comment.

Advertisements

Opa Plugin Development with a PubNub Example

This will be a two part series of posts on writing plugins for Opa. One of Opa’s greatest features is you write everything in the functional, staticly typed Opa language. This even includes the frontend code you’d usually do in Javascript. This means your code is less error-prone which significantly reduces the amount of time spent on debugging code.

But how do you use Javascript library X within your Opa code?

While you can include Javascript code in a resource, like we do here for Google Analytics:

do Resource.register_external_js("/resources/js/google_analytics.js")

This isn’t what we want to do if we want to call external Javascript functions from Opa code. For this we create an Opa plugin. In this post I’ll be showing how I built the Opa client side bindings for the PubNub Javascript API, fork it on github and see all the other language bindings. PubNub is a cloud hosted real-time publish and subscribe interface for all types of applications.

Opa allows binding to external functions through its Binding System Library (BSL). In order to use this to create a plugin for some external Javascript, we create a file pubnub.js that we will compile to a plugin using opa-plugin-builder. Below is an example of one of the external functions that needs to be defined to use PubNub’s publish.

##register publish : string, string -> void
##args(c, m)
{
    PUBNUB.publish({
                       channel : c, message : m
                   });

    return js_void;
}

The first line registers the function publish and specifies its arguments to be two strings and its return value to be void. The next line names the arguments. You can think of ##args like Javascript’s function keyword. In the function, we can use the PUBNUB library just like we would normally. Here we pass to publish the channel and the message to be published. Note that the message can be any Javascript object, even though its a string to the plugin. We simply must serialize the JSON before passing to publish, which we’ll see shortly.

Now after defining external functions for all PubNub api calls in the pubnub.js file we can compile it:

$ opa-plugin-builder src/plugin/pubnub.js -o pubnub

This creates the plugin pubnub.opp.

To make using the plugin easier, we create a package in pubnub.opa that will contain a module PubNub. Below is the contents of pubnub.opa for the publish function that will show how to register the external functions and create a module named PubNub around them.

package pubnub

pubnub_publish = %%pubnub.publish%%

@client publish_client = pubnub_publish

PubNub = {{
  publish(channel: string, j : RPC.Json.json) =
    publish_client(channel, Json.to_string(j))
}}

do Resource.register_external_js("http://cdn.pubnub.com/pubnub-3.1.min.js")

The first thing to notice is setting the pubnub_publish variable. The %% syntax is specifying that pubnub.publish is the key of an external primitive. pubnub is the name of the plugin (pubnub.opp) and publish is the function we registered in the plugin.

After this we create another variable publish_client that specifies that that this function (variables can contain functions) is to be run on the client side. Lastly, we define a module PubNub and a publish function inside the module to be used by our Opa programs.

At the end we make sure that the needed PubNub Javascript file is includes in all resources by registering it and giving the URL to the PubNub CDN.

Now we can compile it all together with the opa-plugin-builder command from above and use the plugin.

import pubnub

broadcast(author, msg) =
   username : RPC.Json.json = {String = author}
   message : RPC.Json.json = {String = msg}
   record_json : RPC.Json.json = {Record = [("username", username), ("message", message)]}
   do PubNub.publish("chat", record_json)
   Dom.clear_value(#entry)

The broadcast function is part of pubnub_chat.opa. broadcast takes 2 arguments, the username of the user sending the message and the message itself. We then construct three variables of Opa’s json type, username, message and record_json. username and message are simply Json strings and they are then combined into a Json Record type that is a list of string/Json pairs. Since our PubNub.publish function converts the Opa Json to a string before sending to the Javascript binding we can simply pass the Opa Json variable record_json.

Lastly, lets take a quick look at the subscribe function. Defining it in pubnub.js look like:

##register subscribe : string, (string -> void) -> void

Since it takes a callback function you’ll notice that the second argument’s type looks different from in publish. (string -> void) is defining that the function takes a variable that is a function which takes a single argument of type string and has a void return type.

Now in pubnub.opa we have a subscribe function that is hardly different from the publish function we described above.

subscribe(channel: string, callback) =
  subscribe_client(channel, callback)

But how do we define a function we can pass to subscribe? That is the great thing, it can be any Opa function!

PubNub.subscribe("chat", (x -> user_update(x)))

Above we are subscribing to the channel chat and passing an anonymous function that takes a single argument and sends it to the function user_update.

user_update(x) =
  match Json.deserialize(x) with
         | {some={Record = [("username", username), ("message", message)]}} ->
                    line = <div class="line"> <div class="user">{Json.to_string(username)}:</>
                             <div class="message">{Json.to_string(message)}</>
                            </>
                    do Dom.transform([#conversation +<- line ])
                    Dom.scroll_to_bottom(#conversation)
         | _ -> void

Now every time a message is posted to the chat channel, our callback, the anonymous function, will be run and call the function user_update which we see deserializes the Json string to get an Opa Json Record. The rest is basic Opa DOM manipulation to add the new message to the messages in the conversation element.

In my next post on Opa and PubNub, I’ll be describing an Opa server side API for PubNub.

Major OpaDo Speed-Up with @publish

I received a patch for OpaDo from Francois Regis Sinot that has made the hosted OpaDo http://opado.org on Dotcloud MUCH faster, adding and removing item is now instantaneous. He placed @publish around explicitly server side functions that caused adding todo items to go from 21 to 1 server calls. I thought this simple but maybe not obvious for new Opa developers (it certainly wasn’t to me!) needed a blog post so that beginners like my self may find it and be able to start using the @publish directive. Check out the Opa manual for some more information on @publish, specifically I’d look at Section 4.7.4 and 10.2

Below are the changes he made. Which you can also of see in the commit diff on Github and check out the entire source.

    
-   @async
-   db_make_done(username: string, id: string) =
+   @publish @async
+   db_make_done(id: string) =
+       username = User.get_username()
        items = /todo_items[username]
        item = Option.get(StringMap.get(id, items))
        /todo_items[username] <- StringMap.add(id, {item with done=true}, items)
 
-   @async
+   @publish @async
    db_remove_item(id: string) =
      username = User.get_username()
      items = /todo_items[username]
 
-   @async
+   @publish @async
    db_add_todo(id: string, x: string) =
      username = User.get_username()
      items = /todo_items[username]
      /todo_items[username] <- StringMap.add(id, { value=x done=false created_at="" }, items)
 
+   @publish
    add_todos() =
      username = User.get_username()
      items = /todo_items[username]

Adding JS To all Opa Resources: Use Case Google Analytics

I decided I wanted to add Google Analytics to OpaDo but had no idea how to easily tell each page to include the necessary Javascript. I asked on the Opa mailing list and got a quick and simple response. Frederic Ye pointed me to Resource.register_external_js

It couldn’t have been any easier. You simply place your google_analytics.js file in your project and use the Resource.register_external_js function to modify the default customization of all Resources. See code below or on the github repo.

package opado.main

import opado.user
import opado.admin
import opado.todo

urls : Parser.general_parser(http_request -> resource) =
  parser
  | {Rule.debug_parse_string(s -> Log.notice("URL",s))} Rule.fail -> error("")
  | "/todos" result={Todo.resource} -> result
  | "/user" result={User.resource} -> result
  | "/login" result={User.resource} -> result
  | "/admin" result={Admin.resource} -> result
  | (.*) result={Todo.resource} -> result

do Resource.register_external_js("/resources/js/google_analytics.js")
server = Server.of_bundle([@static_resource_directory("resources")])
server = Server.make(urls)

For a longer article/tutorial on dealing with external resources check out this blog post from the Opa team Dealing with External Resources.

OpaDo: Personal ToDo Lists

This is a continuation of two past posts (one, two) on my first application with Opa called OpaDo. You can try the live demo here and check out the full source code on Github

Updating OpaDo to add user accounts the project structure has been changed a bit and modularized. Below is the new project layout.

opado/
├── Makefile
├── README.md
├── dotcloud.yml
├── resources
│   ├── destroy.png
│   └── todos.css
└── src
    ├── main.opa
    ├── todo.opa
    └── user.opa

Now there is a main, todo and user module. The main module is the entry point for the app and looks like:

package opado.main

import opado.user
import opado.todo

urls : Parser.general_parser(http_request -> resource) =
  parser
  | {Rule.debug_parse_string(s -> Log.notice("URL",s))} Rule.fail -> error("")
  | "/todos" result={Todo.resource} -> result
  | "/user" result={User.resource} -> result
  | "/login" result={User.resource} -> result
  | (.*) result={Todo.resource} -> result

server = Server.of_bundle([@static_resource_directory("resources")])
server = Server.make(urls)

Here we define the name of this package and import the user and todo modules. Next is the url matching code. urls is a parser that takes an HTTP request and returns a resource. The matching is pretty straight forward. For example:

  | "/todos" result={Todo.resource} -> result

Here we are matching on URLs that begin with /todos but could have anything after that. What is contained after /todos is passed to the Todo.resource which the variable result is set to. And finally that result is returned.

The last two lines simple define the reource directory for the server and pass in the matching function for the HTTP requests.

The todo resource isn’t important to us in this post since its hardly changed. But there are a two important changes:

db /todo_items : stringmap(stringmap(todo_item))
db /todo_items[_][_]/done = false

Here we see that the /todo_items database is not longer simply a stringmap of todo_item‘s but a stringmap of a that. This is so we can reference the items by a user identifier. For example a user identified by the string “user01” who has a todo item identified by “aaa” would be read from the data base as /todo[“user01”][“aaa”].

There are a few other changes to the todo module so that items are properly inserted for the logged in user and deleting must be done in the second stringmap. But we’ll move on to the user module now.

Much of the user module was taken from Matthieu Guffroy’s OpaCMS code on github. But I’ve made a number of modification for my needs.

@abstract type User.password = string
@abstract type User.ref = string

type User.t =
  {
    username : string
    fullname : string
    password : User.password
  }

type User.status = { logged : User.ref } / { unlogged }
type User.info = UserContext.t(User.status)
type User.map('a) = ordered_map(User.ref, 'a, String.order)

db /users : User.map(User.t)

User_data = {{
  mk_ref( login : string ) : User.ref =
    String.to_lower(login)

  ref_to_string( login : User.ref ) : string =
    login

  save( ref : User.ref, user : User.t ) : void =
    /users[ref] <- user

  get( ref : User.ref ) : option(User.t) =
    ?/users[ref]
}}

Above we have the data, types and database definitions necessary to handle the users.

User.t provides the record for storing necessary user data. Next, we have types for checking the user status of if they are logged in or not.

UserContext is a module provided by Opa for dealing with associating the user values with the client — via cookies. And the data for that user can only accessed by the user that owns it.

User_data object provides functions for accessing and manipulating users.

Now we can look at the User module.

User = {{

  @private state = UserContext.make({ unlogged } : User.status)

  create(username, password) =
    do match ?/users[username] with
      | {none} ->
          user : User.t =
            { username=username ;
              fullname="" ;
              password = Crypto.Hash.sha2(password) }
          /users[username] <- user

      | _ -> void
    Client.goto("/login")

At the beginning of the User object we declare a UserContext and a function for creating new users. The function simply checks if the user exists already with the match statement and if not creates a new User.t record and inserts it to the users database.

If we wish to login we must also modify the UserContext

  login(login, password) =
    useref = User_data.mk_ref(login)
    user = User_data.get(useref)
    do match user with
     | {some = u} -> if u.password == Crypto.Hash.sha2(password) then
                       UserContext.change(( _ -> { logged = User_data.mk_ref(login) }), state)
     | _ -> void
    Client.goto("/todos")

The function attempts to read the user from the database and checks if the passwords match. If so, it will set the UserContext to logged in. The function then tells the client to go to /todos. If the login was unsuccessful, it doesn’t matter and will just redirect to the sign up page.

Obviously, better error handling and notification is the next step for the application.

The last interesting part for this I think is the request matching. The rest of the code is mostly just HTML and piecing together the functions I already described.

  resource : Parser.general_parser(http_request -> resource) =
    parser
    | "/new" ->
      _req -> Resource.styled_page("New User", ["/resources/todos.css"], new())
    | "/edit" ->
      _req -> edit()
    | "/view/" login=(.*) ->
      _req -> view(Text.to_string(login))
    | .* ->
      _req -> start()

The key match to look at is:

    | "/view/" login=(.*) ->
      _req -> view(Text.to_string(login))

This shows the request matching /view, which in this case comes after the main module matches ‘/user’ and routes to the User module resource. But then we have login=(.*), this is matching the variable login to the rest of the url. This variable login can then be used in view(Text.to_string(login)) to pass to the view function so it knows what user is being asked to be displayed.

There’ll be more to come. Next, I need to add some validation, an admin page and then the ability for users to have categories to organize their todo items under.

And let me know anything else people would like to see!

OpaDo Data Storage

OpaDo (a port of the TodoMVC app to Opa) now persists todo items to the Opa database. The new version is up on dotcloud, http://opado-tristan.sloughter.dotcloud.com/

I’ve added a todo_item type which stores the item’s value and two other attributes we won’t use until the next post when we have user accounts for their own todo_item stores.

type todo_item = { user_id : string
                 ; value : string
                 ; created_at : string
                 }

To tell Opa where to store the records we’ll create, we provide a path to the Opa db function and set its type. For our todo items we use a stringmap since currently the id’s are randomly generated strings (I know, I know, but its just an example!). We can then reference a record in the database with the path /todo_item[some_id_string].

db /todo_items : stringmap(todo_item)

Now we can insert todo_item‘s to this db path as so:

/todo_items[id] <- { value=x user_id="" created_at="" }

For now user_id and created_at are empty, but I’ll be updating that when I add user accounts.

Since we are storing each item, we need to populate the list on page load with whats already stored:

add_todos() =
  items = /todo_items
  StringMap.iter((x, y -> add_todo_to_page(x, y.value)), items)

The first line of the function sets the variable items to all the todo_item records in the database. We use StringMap.iter to take each todo_item and add it to the page. The first argument to the anonymous function is the id the item is stored in the database with (the id we will use in the HTML as well) and the second is the actual todo_item, so we take its value field and pass that to the add_todo_to_page function along with the id.

To have the add_todos function when the list element is ready we add an on_ready attribute that will call add_todos:

<ul id=#todo_list onready={_ -> add_todos() } ></ul>

Lastly, we want to be able to delete a todo_item from the database:

remove_item(id: string) =
  do Dom.remove(Dom.select_parent_one(#{id}))
  do Db.remove(@/todo_items[id])
  update_counts()

remove_all_done() =
  Dom.iter(x -> remove_item(Dom.get_id(x)), Dom.select_class("done"))

The main piece to notice here is @/todo_items[id] in Db.remove(). The @ is saying that we are passing the path itself to remove() and not the value at that path.

Nice and easy! No database to setup or deploy, just Opa. Next time we’ll add user accounts, so we don’t have to all share the same todo list.

TodoMVC in Opa

Edit: I just learned that dotcloud supports Opa! So I’ve pushed OpaDo and you can see a demo here http://opado-tristan.sloughter.dotcloud.com/

I wanted something quick and simple to do in Opa to give it a try so I decided to implement the TodoMVC example that has been redone in almost all Javascript frameworks, https://github.com/addyosmani/todomvc.

The code can be found on GitHub here: https://github.com/tsloughter/OpaDo

Opa is unique in that it is not only a new language but also a new web server and database. While Opa’s page pushes the idea that its for the cloud and its easy distribution, I found the nicest part being the static typing and no need for Javascript.

The functions below handle interactions with the Todo items. It somewhat reminds me of Lift but taken even farther.

/** * {1 User interface} */
update_counts() =
  num_done = Dom.length(Dom.select_class("done"))
  total = Dom.length(Dom.select_class("todo"))
  do Dom.set_text(#number_done, Int.to_string(num_done))
  Dom.set_text(#number_left, Int.to_string(total - num_done))

make_done(id: string) =
  do if Dom.is_checked(Dom.select_inside(#{id}, Dom.select_raw("input"))) then Dom.add_class(#{id}, "done")
  else
    Dom.remove_class(#{id}, "done")

  update_counts()

remove_item(id: string) =
  do Dom.remove(#{id})
  update_counts()

remove_all_done() =
  do Dom.remove(Dom.select_parent_one(Dom.select_class("done")))
  update_counts()

add_todo(x: string) =
  id = Random.string(8)
  li_id = Random.string(8)
  line = <li id={ li_id }><div class="todo" id={ id }> <div class="display"> <input class="check" type="checkbox" onclick={_ -> make_done(id) } /> <div class="todo_content">{ x }</div> <span class="todo_destroy" onclick={_ -> remove_item(li_id) }></span> </div> <div class="edit"> <input class="todo-input" type="text" value="" /> </div> </div></li>
  do Dom.transform([#todo_list +<- line ])
  do Dom.scroll_to_bottom(#todo_list)
  do Dom.set_value(#new_todo, "")
  update_counts()

It is unique in combining the HTML into the language itself. Some have argued against this but when it works well it makes perfect sense. I don’t want to have to convert a designers HTML into some other representation! And being able to have type checked dynamic functionality within the HTML is a boon. Even with just this simple program, I found the usefulness of the type checker outstanding.

Next we have the main outline of the page and the entry part for the program.

start() =
  <div id="todoapp"> <div class="title"> <h1>Todos</h1> </div>
    <div class="content"> <div id=#create_todo> <input id=#new_todo placeholder="What needs to be done?" type="text" onnewline={_ -> add_todo(Dom.get_value(#new_todo)) } /> </div>
      <div id=#todos> <ul id=#todo_list></ul> </div>

      <div id="todo_stats"> <span class="todo_count"> <span id=#number_left class="number">0</span> <span class="word">items</span> left. </span> <span class="todo_clear"> <a href="#" onclick={_ -> remove_all_done() }> Clear <span id=#number_done class="number-done">0</span> completed <span class="word-done">items</span> </a> </span> </div>
    </div>
  </div>

/** * {1 Application} */

/** * Main entry point. */
server = Server.one_page_bundle("Todo",
       [@static_resource_directory("resources")],
       ["resources/todos.css"], start)

It won’t be able to replace my use of Erlang for the backend and Coffeescript for the frontend, but it looks very promising.

I’ll be extending this example to include persistence, sessions and users and will add posts as I complete those.

Erlang PubNub Client and Chat

I was thoroughly impressed with PubNub, a publish/subscribe service, when I first read their articles and played around with it some in Javascript. But obviously I need an Erlang API if I’m going to really use it! So I’ve created ePubNub.

In the ePubNub README you’ll find information on some basic usage of the application. You don’t have to do anything more than use the epubnub.erl module to publish and subscribe (by either providing a PID to send messages to or a function handler to process each).

Here I’ve built a little more complicated app/release called epubnub_chat, and the source is also on github.

The first thing we need is the epubnub app as a dependency in the epubnub_chat.app file:

   {applications, [kernel, stdlib, epubnub]},

We’ll use a simple_one_for_one for supervising channel subscribed processes. In epubnub_chat_sup we have 3 API functions for the user to use (start_link is run by the _app.erl module on startup): connect/1, connect/2, disconnect/1:

connect(Channel) ->
    supervisor:start_child(?SERVER, [Channel]).

connect(EPN, Channel) ->
    supervisor:start_child(?SERVER, [EPN, Channel]).

disconnect(PID) ->
    epubnub_chat:stop(PID).

EPN is a record containing the necessary url and keys for talking to the PubNub service and is created with the new functions in epubnub:

-spec new() -> record(epn).
new() ->
    #epn{}.

-spec new(string()) -> record(epn).
new(Origin) ->
    #epn{origin=Origin}.

-spec new(string(), string()) -> record(epn).
new(PubKey, SubKey) ->
    #epn{pubkey=PubKey, subkey=SubKey}.

-spec new(string(), string(), string()) -> record(epn).
new(Origin, PubKey, SubKey) ->
    #epn{origin=Origin, pubkey=PubKey, subkey=SubKey}.

You can pass none to connect/1 and it will send none to the chat gen_server and it will use defaults of pubsub.pubnub.com, demo and demo, for the url, publish key and subscribe key respectively.

Now in the epubnub_chat gen_server we need the following API functions:

start_link(Channel) ->
    start_link(epubnub:new(), Channel).

start_link(EPN, Channel) ->
    gen_server:start_link(?MODULE, [EPN, Channel], []).

message(Server, Msg) ->
    gen_server:cast(Server, {message, Msg}).

stop(Server) ->
    gen_server:cast(Server, stop).

The start_link functions are run when the supervisor spawns a simple_one_for_one supervisor for the process. This returns {ok, PID}. This PID must be remembered so we can talk to the process we have started thats subscribed to a specific channel. We pass this PID to the message/2 and stop/1 functions, which we’ll see at the and when we use the program.

start_link/1 and /2 call init/1 with the provided arguments:

init([EPN, Channel]) ->
    {ok, PID} = epubnub_sup:subscribe(EPN, Channel, self()),
    {ok, #state{epn=EPN, pid=PID, channel=Channel}}.

Here we use the epubnub_sup subscribe/3 function and not epubnub:subscribe because we want it to be supervised. We store the PID for this process so we can terminate it later.

The epubnub subscribe process was given the PID, returned by self/1, of the current process which is the gen_server process and will send messages that are published to the channel to that process. We handle these messages in handle_info/2:

handle_info({message, Message}, State) ->
    io:format("~p~n", [Message]),
    {noreply, State}.

Lastly, we have to handle the messages from message/2 and stop/1:

handle_cast({message, Msg}, State=#state{epn=EPN, channel=Channel}) ->
    epubnub:publish(EPN, Channel, Msg),
    {noreply, State}.

terminate(_Reason, #state{pid=PID}) ->
    epubnub:unsubscribe(PID),
    ok.

The handle_cast/2 function published your message to the channel this process subscribed to with epubnub:publish/3 and terminate calls epubnub:unsubscribe/1 before this process ending which sends a terminate message to the subscibred process.

Now lets see this program in action:

[tristan@marx ~/Devel/epubnub_chat]
09:55 (master)$ sinan dist
starting: depends
starting: build
starting: release
starting: dist
[tristan@marx ~/Devel/epubnub_chat]
09:55 (master)$ sudo faxien ir
Password:
Do you want to install the release: /Users/tristan/Devel/epubnub_chat/_build/development/tar/epubnub_chat-0.0.1.tar.gz
Enter (y)es, (n)o, or yes to (a)ll? > ? y
Replacing existing executable file at: /usr/local/lib/erlang/bin/epubnub_chat
Replacing existing executable file at: /usr/local/lib/erlang/bin/5.8.2/epubnub_chat
Replacing existing executable file at: /usr/local/lib/erlang/bin/erlware_release_start_helper
Replacing existing executable file at: /usr/local/lib/erlang/bin/5.8.2/erlware_release_start_helper
ok
[tristan@marx ~/Devel/epubnub_chat]
09:56 (master)$ epubnub_chat
....
Eshell V5.8.2  (abort with ^G)
1> {ok, Server} = epubnub_chat_sup:connect("chat").
{ok,}
2>
=PROGRESS REPORT==== 10-Apr-2011::09:57:14 ===
          supervisor: {local,inet_gethost_native_sup}
             started: [{pid,},{mfa,{inet_gethost_native,init,[[]]}}]

=PROGRESS REPORT==== 10-Apr-2011::09:57:14 ===
          supervisor: {local,kernel_safe_sup}
             started: [{pid,},
                       {name,inet_gethost_native_sup},
                       {mfargs,{inet_gethost_native,start_link,[]}},
                       {restart_type,temporary},
                       {shutdown,1000},
                       {child_type,worker}]

2> epubnub_chat:message(Server, <"hello there!">).
ok
3> <"hello there!">
<"I'm from the webapp!">

3> q().
ok

You can go to the PubNub tutorial page to chat between yourself, or get someone else to join!

That’s it! Simple and quick global communication that scales for you!

I’ve really enjoyed playing with PubNub and hope I get to use it for a real project soon.

eCloudEdit Part 2: CouchDB

In my last post I showed the Webmachine backend to James Yu’s CloudEdit app in Backbone.js. What was left out was, where are the documents stored? Here I’ll show how this is done with CouchDB. And you can give the app a try at http://erlware.org:8080

First, a new Erlang app is needed that we’ll call ece_db.

ece_db/
├── doc
├── ebin
│   ├── ece_db.app
│   └── overview.edoc
├── include
└── src
    ├── ece_db.erl
    ├── ece_db_app.erl
    └── ece_db_sup.erl

Three modules are implemented, one that starts the app by calling the supervisor’s start function, the supervisor itself that sets up a simple_one_for_one and the gen_server that handles the frontends requests for creating and modifying documents in CouchDB.

We’ll ignore the ece_db_app.erl module for this post and start with ece_db_sup.erl. On requests from the Webmachine resource module we don’t want one to wait on the other, requests should be handled in parrallel. One option is to not create a process for the database backend and instead have all of the functions in the database interface run in the process of the Webmachine resource. However, two reasons to not do this are that there are multiple pieces of data that must be read out of a configuration file and used to setup what is needed to talk to CouchDB. We do not want to be doing this for every request! Furthermore a supervised gen_server allows us the possibility of retrying requests with no extra code and crashing without retrying but not taking down the resource’s process that is handling the user’s HTTP request. It is a lot nicer and easier to just let things fail when we can!

  {ece_db, [{server, "HOSTNAME"},
            {port, 80},
            {database, "ecloudedit"}]}
-spec init(list()) -> {ok, {SupFlags::any(), [ChildSpec::any()]}} |
                       ignore | {error, Reason::any()}.
init([]) ->
    RestartStrategy = simple_one_for_one,
    MaxRestarts = 0,
    MaxSecondsBetweenRestarts = 1,

    SupFlags = {RestartStrategy, MaxRestarts, MaxSecondsBetweenRestarts},

    Restart = temporary,
    Shutdown = 2000,
    Type = worker,

    {ok, Server} = application:get_env(server),
    {ok, Port} = application:get_env(port),
    {ok, DB} = application:get_env(database),

    AChild = {ece_db, {ece_db, start_link, [Server, Port, DB]},
              Restart, Shutdown, Type, [ece_db]},

    {ok, {SupFlags, [AChild]}}.

First, we have the config entries in config/sys.config that will be read in by application:gen_env. Here we set the server URL, port number and name of the database to use. CouchDB can easily be run locally but for my running copy I use a database hosted by the great service Cloudant. Next is the init function for ece_db_sup.erl. A key thing to note for a simple_one_for_one is that NO process is started after the supervisor’s init function returns like with other types of supervisor children. Instead we must explicitly call start_child. This is how we are able to create a supervised gen_server process for every HTTP request. Below is the code in ece_db_sup for starting and stopping the process:

start_child() ->
    supervisor:start_child(?SERVER, []).

terminate_child(PID) ->
    ece_db:terminate_child(PID).

In the last post I showed the functions that start and stop the ece_db process. Here they are again:

init([]) ->
    {ok, PID} = ece_db_sup:start_child(),
    {ok, #ctx{db=PID}}.
finish_request(ReqData, Ctx) ->
    ece_db_sup:terminate_child(Ctx#ctx.db),
    {true, ReqData, Ctx}.

On each request Webmachine calls the init function for the resource that is matched in the dispatch table. In that init function we call start_child in ece_db_sup which returns {ok, PID}. The PID is the process id we’ll be sending messages. Now in ece_db we implement the API functions needed to start the process and to interact with it.

start_link(Server, Port, DB) ->
    gen_server:start_link(?MODULE, [Server, Port, DB], []).

all(PID) ->
    gen_server:call(PID, all).

find(PID, ID) ->
    gen_server:call(PID, {find, ID}).

create(PID, Doc) ->
    gen_server:call(PID, {create, Doc}).

update(PID, ID, JsonDoc) ->
    gen_server:call(PID, {update, ID, JsonDoc}).

terminate(PID) ->
    gen_server:call(PID, terminate).

Each function, besides the one for starting the process, takes a PID. This PID is the process gen_server:call to which it sends its message. To handle these messages we have the gen_server callbacks.

%%%===================================================================
%%% gen_server callbacks
%%%===================================================================

%% @private
init([Server, Port, DB]) ->
    CouchServer = couchbeam:server_connection(Server, Port, "", []),
    {ok, CouchDB} = couchbeam:open_db(CouchServer, DB),
    {ok, #state{db=CouchDB}}.

%% @private
handle_call(all, _From, #state{db=DB}=State) ->
    Docs = get_docs(DB, [{descending, true}]),
    {reply, mochijson2:encode(Docs), State};
handle_call({find, ID}, _From, #state{db=DB}=State) ->
    [Doc] = get_docs(DB, [{key, list_to_binary(ID)}]),
    {reply, mochijson2:encode(Doc), State};
handle_call({create, Doc}, _From, #state{db=DB}=State) ->
    {ok, Doc1} = couchbeam:save_doc(DB, Doc),
    {NewDoc} = couchbeam_doc:set_value(<<"id">>, couchbeam_doc:get_id(Doc1), Doc1),
    {reply, mochijson2:encode(NewDoc), State};
handle_call({update, ID, NewDoc}, _From, #state{db=DB}=State) ->
    IDBinary = list_to_binary(ID),
    {ok, Doc} = couchbeam:open_doc(DB, IDBinary),
    NewDoc2 = couchbeam_doc:set_value(<<"_id">>, IDBinary, {NewDoc}),
    NewDoc3 = couchbeam_doc:set_value(<<"_rev">>, couchbeam_doc:get_rev(Doc), NewDoc2),
    {ok, {Doc1}} = couchbeam:save_doc(DB, NewDoc3),
    {reply, mochijson2:encode(Doc1), State;
handle_call(terminate, _From, State) ->
    {stop, normal, State}.

init takes the server URL, the port number and the name of the database to store the documents as arguments and creates a CouchBeam database record. To handle the other messages we have the handle_call function to match on the different tuples sent to gen_server:call for the different function calls. all uses the internal function get_docs with the option to have the function in descending order (so the newest is first) and returns those after encoding to JSON. find uses the same function but only wants a certain document which in the case of CouchDB is specified with a key value. create first saves the new document which returns the document that is actually stored in CouchDB. CouchDB adds the _id and _rev key/value pairs it generates. Since our frontend expects the id to be the value of a key id and not _id we use couchbeam_doc:set_value to set a value of the new key id with the value of the documents id retrieved from the document with couchbeam_doc:get_id. For update we first open the document corresponding to the id passed in as an argument to get the _rev value. _rev needs to be set in the document we send to save_doc so CouchDB knows this is a valid update based on the previous version of the document. Thus we set the new documents _id and _rev values and send to save_doc.

Lastly, we have the get_docs function. Since the frontend expects the key id to exist and CouchDB stores the id with the key _id we use a simple view to return the documents with id containing the value of the documents _id. Views are defined with map/reduce functions that CouchDB runs across the databases documents and builds an index from. These indexes are then queried based on keys to find documents. Thus access is very quick, indexes are built with BTrees, and a map/reduce does not have to be run on the documents for every request. Additionally, when a new document is created the view’s map/reduce functions only need to run over the new documents to update the BTree. Updates to the view’s indexes are either done on each use of the view or can be manually told to run.

function (doc)
{
  emit(doc._id, {id : doc._id, title : doc.title, created_at : doc.created_at, body : doc.body});
}
get_docs(DB, Options) ->
    {ok, AllDocs} = couchbeam:view(DB, {"all", "find"}, Options),
    {ok, Results} = couchbeam_view:fetch(AllDocs),

    {[{<<"total_rows">>, _Total},
      {<<"offset">>, _Offset},
      {<<"rows">>, Rows}]} = Results,

    lists:map(fun({Row}) ->
                      {<<"value">>, {Value}} = lists:keyfind(<<"value">>, 1, Row),
                      Value
              end, Rows).

In get_docs the couchbeam:view function is used to construct a view request to be sent to the CouchDB server. couchbeam_view:fetch sends the view request and returns the results as Erlang terms. After this we extract the rows from the results and use lists:map to extract out just the value of each row to be returned.

That’s it! There are places that could use improvement. One being the use of lists:map over every returned row to construct documents the frontend can deal with. Additionally, the need to duplicate _id as id for the frontends use could be removed through modifications on the frontend.

In the next installment I’ll be updating the code — and maybe making some of these performance enhancements — with James Yu’s recent changes in his Part 2.

eCloudEdit: Erlang, Webmachine and Backbone.js

To experiment with using a pure client-side rendering talking to an Erlang backend I’ve taken James Yu’s CloudEdit tutorial an app written with Backbone.js and Rails and converted the backend to use Webmachine and CouchDB. You can see eCloudEdit in action here. The Backbone.js code is the same so to understand that please see James’ post, here I’ll describe the Erlang backend.

To begin with we setup two applications, one for handling the web interaction and a second for handling the database interaction. We end up with this directory layout:

  |-eCloudEdit
   |-bin
   |-config
   |-doc
   |-lib
   |---ece_db
   |-----doc
   |-----ebin
   |-----include
   |-----src
   |---ece_web
   |-----doc
   |-----ebin
   |-----include
   |-----priv
   |-------images
   |-------javascripts
   |---------controllers
   |---------models
   |---------views
   |-------stylesheets
   |-----src

Under ece_web/priv is where all the html and Javascript from CloudEdit is placed, with nothing changed. To serve up the static content through Webmachine we use this resource which we won’t worry about in this article. Lets look at how webmachine deciced where to route requests. This is handled by a set of dispatch rules that are passed into Webmachine on startup. We can see how this is done by looking at ece_web_sup and how the supervisor loads Webmachine:

-spec init(list()) -> {ok, {SupFlags::any(), [ChildSpec::any()]}} |
                          ignore | {error, Reason::any()}.
init([]) ->
    WebChild = {webmachine_mochiweb,
                {webmachine_mochiweb, start, [config()]},
                permanent, 5000, worker, dynamic},

    RestartStrategy = one_for_one,
    MaxRestarts = 3,
    MaxSecondsBetweenRestarts = 10,
    SupFlags = {RestartStrategy, MaxRestarts, MaxSecondsBetweenRestarts},

    {ok, {SupFlags , [WebChild]}}.

config() ->
    {ok, IP} = application:get_env(webmachine_ip),
    {ok, Port} = application:get_env(webmachine_port),
    {ok, App}= application:get_application(),
    LogDir = code:priv_dir(App) ++ "/logs",
    {ok, Dispatch} = file:consult(filename:join([code:priv_dir(App), "dispatch"])),

    [{ip, IP},
     {port, Port},
     {log_dir, LogDir},
     {backlog, 128},
     {dispatch, Dispatch}].

The dispatch terms are loaded from a file dispatch in the application’s priv directory. For this app the dispatch file contains:

{["documents", id], ece_resource_documents, []}.
{["documents"], ece_resource_documents, []}.
{['*'], ece_resource_static, []}.

There are two resources, one for handling the requests for creating, updating and viewing documents and one for serving up all other requests (static html, js and css files). The first rule matches paths like, /documents/foo, id is set to foo and the request is sent to the documents resource. If there is nothing after /documents it is still sent to the documents resource but there is no id.

Webmachine is essentially a REST toolkit. You build resources by defining functions that handle the different possible HTTP requests. For this webapp we’ll only be dealing with GET, POST and PUT. GET is used if we’d like to retrieve information on documents, POST is for creating a new document and PUT is for updating a document.

-record(ctx, {db}).

init([]) ->
    {ok, PID} = ece_db_sup:start_child(),
    {ok, #ctx{db=PID}}.

allowed_methods(ReqData, Ctx) ->
    {['HEAD', 'GET', 'POST', 'PUT'], ReqData, Ctx}.

content_types_accepted(ReqData, Ctx) ->
    {[{"application/json", from_json}], ReqData, Ctx}.

content_types_provided(ReqData, Ctx) ->
    {[{"application/json", to_json}], ReqData, Ctx}.

finish_request(ReqData, Ctx) ->
    ece_db_sup:terminate_child(Ctx#ctx.db),
    {true, ReqData, Ctx}.

To handle GET requests we implemented the to_json function we specified in content_types_provided to handle GET requests for json data.

to_json(ReqData, Ctx) ->
    case wrq:path_info(id, ReqData) of
        undefined ->
            All = ece_db:all(Ctx#ctx.db),
            {All, ReqData, Ctx};
        ID ->
            JsonDoc = ece_db:find(Ctx#ctx.db, ID),
            {JsonDoc, ReqData, Ctx}
    end.

wrq:path_info is used to find the value of id, which we set in the dispatch file, if it is undefined we know the request is for all documents, while if it has a value we know to find the contents of the document with that id and return its contents. We’ll see the content of ece_db:all/1 and ece_db:find/2 in the next article. Just know they both return JSON data structures.

Now we must support POST for creating documents or we have nothing to return to a GET request.

process_post(ReqData, Ctx) ->
    [{JsonDoc, _}] = mochiweb_util:parse_qs(wrq:req_body(ReqData)),
    {struct, Doc} = mochijson2:decode(JsonDoc),
    NewDoc = ece_db:create(Ctx#ctx.db, {Doc}),
    ReqData2 = wrq:set_resp_body(NewDoc, ReqData),
    {true, ReqData2, Ctx}.

wrq:req_body/1 returns the contents of the body sent in the HTTP request. Here it is the conents of the document to store. We decode it to an Erlang data structure and pass it to the ece_db app for inserting into the database. After inserting to the database the create/2 function returns the new document with a new element id (in this case generated by CouchDB). This is required so we know the document’s id which is used by the Backbone.js frontend. In order to return it from the POST request we must set response body to the contents of the document with wrq:set_resp_body/2

Lastly, updating documents requires support for PUT. In contents_type_accepted/2 we’ve specified that PUT requests with JSON content is sent to the function from_json/2:

from_json(ReqData, Ctx) ->
    case wrq:path_info(id, ReqData) of
        undefined ->
            {false, ReqData, Ctx};
        ID ->
            JsonDoc = wrq:req_body(ReqData),
            {struct, Doc} = mochijson2:decode(JsonDoc),
            NewDoc = ece_db:update(Ctx#ctx.db, ID, Doc),
            ReqData2 = wrq:set_resp_body(NewDoc, ReqData),
            {true, ReqData2, Ctx}
    end.

If this request was not routed through the first rule in our dispatch file it does not have an id and thus can not be an update. When this happens we return false so the frontend is aware something has gone wrong. For requests containing an id we pass the contents of the requests body to ece_db‘s update/2 function.

In the next post I’ll show how ece_db is implemented with Couchbeam for reading and writing the documents to CouchDB on Cloudant.