Recently in Dev Category

After years of dislike, I may have to admit that I'm not entirely filled with hatred for Python. This did not come voluntarily, mind you. My love of laziness naturally breeds a lot of small projects; why repeat a task over and over when I can code something once and not have to think about it again?

My love of laziness was thrilled when I introduced it to web services some years ago. Now there was a place for my apathetic side as well - why bother learning what over people have coded, when all I have to do it point my code to a web service (ReST was a natural choice *rimshot*)?

A situation arose where there only real way to get something done without reinventing the wheel (avoiding a joke about things being chowned and chgrped to 'wheel') was to consume a web service. Finding a chunk of code where the consumption was already written was easy, but it happened to be in Python. I swallowed hard and decided to use it anyway, creating a Frankensteinian script wherein PHP passed arguments to a Python script which made the web service call and passed the results to a Ruby script (I never managed to work in Erlang).

As the benefit of that particular script became apparent, I made use of the Python section again and again. I finally realized that it was impossible to do any sort of sane version control since there were at least three script files to be updated per 'script'.

On one particularly slow day I decided to try to rewrite one of the aging scripts in pure Python. It worked and was a lot cleaner. I've written some subsequent scripts in pure Python as well, and a lot of my dislikes remain:

• I hate converting every single number to a string before concatenating it to another string
• Python, your implementation of lists is hideous
• I don't want to have to track eleventy billion import statements
• The whitespace is just annoying. I know how to write clean-looking code. Sometimes I just don't want to.
• Get a damn case command. Virtually every other language in common use has one.
• As I've said before: Twisted Python (which admittedly is not part of the Python core) tries way too hard to behave like Erlang. Just learn Erlang and be done with it.
• String manipulation is the most counter-intuitive crap I've ever seen. x86 assembly makes more sense

That being said, it's just possible that perhaps Python isn't quite as evil as I once thought. Python, at least you're not Java.

• FINALLY a language (that I'm aware of) where True !== 1 and False !== 0. I'd have to take off YOUR shoes to count the number of times that's bitten me in the ass. For people who don't know me, I wear sandals so no shoe-taking-offing is ever needed.


• The it primitive is pretty neat and has possibilities:
  

  True || False
  >>True
  it
  >>True
  not it
  >>False
  



• I like the /= representation of !=


• Hmmm...lists are interesting.
  let x = [1,2]
  [1,2]
  0:x
  [0,1,2]
  
  So clearly the : primitive appends elements to the beginning of a list. I thought that perhaps [0,1]:x might produce [[0,1],[1,2]]. It turns out this is not the case, and instead throws an error that essentially amount to a data type mismatch. To achieve the desired result, one must type:
  
  [0,1]:[x]
  
  ...such that both arguments are lists, I assume. Weird.


• putStrLn "Hi"
  Hi
  putStrLn 'Hi'
  
  :1:11:
  lexical error in string/character literal at character 'i'
  
  D'oh. I prefer to use single quotes with my strings. Not a deal breaker though.



Well, as thrilling as I'm sure this has been for you, I'm going to stop and actually get to coding.

So after a full reinstall of Leopard, my Mac seems to be behaving a bit more like it should. I wonder what the odds are of getting a corrupt OS from the factory.

MacPorts seems like a nice idea, and the list of software is decent. My two complaints (what, you thought I wouldn't have any?):

• 1) It's slow as hell. I've been installing Haskell for the last twenty minutes.
• The .dmg-based installer forgets one minor detail, namely changing the $PATH variable. Looking through the instructions for compiling from source, it seems that adding

export PATH=/opt/local/bin:/opt/local/sbin:$PATH

to .profile or .bashrc fixes the issue. I'm sure there's a more elegant approach, but this allowed me to get started on the lengthy process of installing something that would be a single command on Ubuntu.

I can't remember if I mentioned this earlier or not, and quite frankly I'm too lazy to look. So here's the idea. My servers do their internal communication on separate gigabit backbone, leaving the 100Mb NICs free to handle the rest of the traffic. I'd love to have a 10Gb backbone, but I can't afford a Cisco 6500 at the moment. Or possibly ever.

Anyway, the spine takes advantage of Erlang's ability to interconnect nodes running on different machines virtually transparently. The code is nothing more than a complicated wrapper around Erlang's '!' primitive, which is used to send messages between nodes.

My initial use for it is to send relevant entries from the BIND logs on the DNS server to the web server for easy display on the secondary DNS hosting site that's nearing completion. While this is a very simple application, there are plenty of other uses. After a few security-related tweaks, a node will be placed on our back-up cluster in Dallas/Fort Worth for doing backups.

The nice thing about having an intelligent spine is the each node can report back on the status of it's host machine. If, for example, a web server was experiencing difficulties, it could communicate that to a fail-over server which could take over until the problem could be resolved. It's sort of like a cloud concept - there are merely servers that behave however they have to.

Since Erlang nodes can pass instructions to other nodes, the network could essentially re-structure itself if needed.

To get a better idea of what it's like, Kevin Smith has a fabulous series of screencasts on the Pragmattic Programmers site. Go and buy them.

Kevin, if you're reading this and would like to team up on a screencast, let me know.

I mentioned in the last entry that Jan Lehnardt was also talking about gen_server, and his article is now up:
Read it here.

Those of you playing along at home may have gotten the impression that I kinda like this whole Erlang thing. Correct. Those of you playing along at home (and wayward surfers as well) from outside of Europe may be wondering why there's a bothersome 'u' lurking in the spelling of 'behaviour'. Erlang came from Ericsson (who denies the connection between 'erlang' and 'ERicsson LANGuage'. They claim it was named after a mathematician of the same name.) and Ericsson just so happens to reside in Scandanavia.

Linguists theorize that they are allowed to put the letter 'u' wherever they please as a trade-off for having to use that weird o-with-a-line-through-it letter. Or maybe they just want to show the word how much they love empty sets. Regardless, that being said let's hope into gen_server.

gen_server is a component of OTP, the Open Telecom Platform. OTP can be thought of as a sort of application framework for Erlang, much as Python has Pylons, Ruby has Rails, and Windows has Problems Crashing. OTP is hilariously powerful, and being the relative new-comer that I am, I've only just started to really grasp just how powerful it is. To quote Joe Armstrong, author of the fantastic book "Programming Erlang":

    The power of OTP comes from the fact that properties such as fault tolerance, scalability, dynamic-code upgrade, and so on, can be provided by the behavior itself. In other words, the writer of the callback does not have to worry about [those things] because this is provided by the behavior.

Okay Philip, what's a behaviour? A behaviour is just what it sounds like - a bucket containing all the common behaviors of a certain kind of system or platform.

In the great tradition of contrived examples, imagine a movie store that would only let you check out one movie at a time. The code for that might look something like what I've included below. Just like last time, first the code and then the explanation. My eternal thanks go out to Jan Lehnardt of the CouchDB project for his assistance, patience, and friendship. In the near future, he's going to be posting a plok entry on this same subject, so by all means go see if that's there. And give him money.

  -module(movie_store).
  -behaviour(gen_server).
 
  -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]).
 
  -export([checkout/2, lookup/1, start_link/0]).
 
  start_link() -> gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
  checkout(Customer, Movie) -> gen_server:call(?MODULE, {checkout, Customer, Movie}).
  lookup(Customer) -> gen_server:call(?MODULE, {lookup, Customer}).

  init([]) ->
       Tab = ets:new(?MODULE, []),
       {ok, Tab}.
 
  handle_call({checkout, Customer, Movie}, _From, Tab) ->
       Response = case ets:lookup(Tab, Customer) of
         [] -> 
               ets:insert(Tab, {Customer, Movie}),
               {ok, Movie};
         [{Customer, OtherMovie}] ->
               {already_checked_out, OtherMovie}
       end,
       {reply, Response, Tab};
 
  handle_call({lookup, Customer}, _From, Tab) ->
        Reply = case ets:lookup(Tab, Customer) of
               [{Customer, Movie}] ->
                   Movie;
               [] ->
                   none
        end,
        {reply, Reply, Tab}.
 
  handle_cast(_Msg, State) -> {noreply, State}.
  handle_info(_Msg, State) -> {noreply, State}.
  terminate(_Reason, _State) -> ok.
  code_change(_OldVersion, State, _Extra) -> {ok, State}.

Okay - it looks terrifying (which incidentally, is Erlang's super power) but it's not. The first line declares the module name (and 'movie_store.erl' should also be the filename) and is a standard part of every Erlang program.

-behaviour(gen_server).can be thought of a line that declares which 'template' we'll be using for this program. The gen_server behaviour mandates that we define the functions specified in the first set of exports.

  init([]) ->
       Tab = ets:new(?MODULE, []),
       {ok, Tab}.

  handle_call({checkout, Customer, Movie}, _From, Tab) ->
       Response = case ets:lookup(Tab, Customer) of
         [] -> 
               ets:insert(Tab, {Customer, Movie}),
               {ok, Movie};
         [{Customer, OtherMovie}] ->
               {already_checked_out, OtherMovie}
       end,
       {reply, Response, Tab};

  handle_call({lookup, Customer}, _From, Tab) ->
        Reply = case ets:lookup(Tab, Customer) of
               [{Customer, Movie}] ->
                   Movie;
               [] ->
                   none
        end,
        {reply, Reply, Tab}.

  handle_cast(_Msg, State) -> {noreply, State}.
  handle_info(_Msg, State) -> {noreply, State}.
  terminate(_Reason, _State) -> ok.
  code_change(_OldVersion, State, _Extra) -> {ok, State}.

One thing I hate doing in Ruby (or any language for that matter) is setting up a web service or anything that requires a read/write socket. That's one of the reasons I like Erlang so much - the messaging takes a lot of the hassle out of the equation.

You build a module, register it (so you can address the module by name instead of PID), and send messages (see my tutorial a few entries ago). I've also become interested in the message bus idea. All concerned processes are tied into a communication circuit wherein they listen for messages addressed to them and send replies as appropriate.

Erlang offers fantastic stability (one project that involved 2 million lines of Erlang code achieved a nine nines reliability, which is absolutely unheard of) but is somewhat counter-intuitive when you're getting started. Ruby (or Python, PHP, etc) is very fast to write and generally makes sense but I find it's a pain in the ass to do network I/O. God help you people who do it in C or Java.

So I got to thinking that having an Erlang-powered 'spine' may be just the thing for a SOA. If one bites the bullet and writes an Erlang back-bone to handle the interconnectivity between machines and act as a central nerve fiber between interpreted language processes.

The hard part in my mind would be adding a smooth interface between Ruby and Erlang but once that's done it's trivial to hang another Erlang endpoint off the bus to do the talking.

I'm probably not describing this well and as such it may come out sounding really, really stupid. But I think it will at the very least be a good learning experience. The more I work with Erlang the more I like it and the more potential I see in it. If you haven't looked at it, I strongly suggest you at least give it a try. It can look very strange and will challenge some of the basic ways you think about coding, but it is very worth it.

Erlang is a language that has fascinated me for lo, these last few weeks. For the unfamiliar, it was developed to implement high-availability, distributed, fault-tolerant systems for the telecom world decades ago and in recent years has witnessed a resurgence as multi-core processors become the norm, rather than the exception.

At first, Erlang is bit daunting. Variables aren't, line endings vary by location, and recursion is common-place, among others. After a few hours of playing with the language however, these 'problems' turn into 'features' and you begin to see the benefits of the language.

A key component of Erlang are processes. Processes, unlike threads in other languages, have no shared environment or data. The only manner of communicating things between processes is through messaging. Processes listen for messages addressed to them and take appropriate action. I've been playing with inter-processes messaging and wanted to share a simple example. First, I'll list the code and then walk you through it. The code is based on the 'Getting Started' tutorial at http://www.erlang.org

The Code:

-module(messagetest).
-export([start/0, first/2, second/0]).

first(N, SecondPID) ->
  SecondPID ! {N + 1, self()},
  receive
    continue ->
      io:format("First Continuing... ~n", []),
      first(N + 1, SecondPID);
    stop ->
      io:format("First Received 'Stop' command.  Stopping.~n", []),
      SecondPID ! done
  end.

second() ->
  receive
    done ->
      io:format("Second Received 'done'~n", []);
    {N, FirstPID} ->
      case (N < 10) of
        false -> FirstPID ! stop;
        true -> FirstPID ! continue
      end,
      io:format("Second got ~p ~n", [N]),
      second()
  end.

start() ->
  SecondPID = spawn(messagetest, second, []),
  spawn(messagetest, first, [0, SecondPID]).

The Explanation:

The first two lines define the module name and give a directory of the functions that are externally callable along with their argument counts. The last three lines are where the fun begins. Save the code in a file named 'messagetest.erl' and from the Erlang interpreter, compile it by typing:

c(messagetest). 

Next, run the 'start' function by typing:

messagetest:start().

You'll see output similar to:

<0.71.0>Second got 1 

First Continuing... 
Second got 2 
First Continuing... 
Second got 3 
First Continuing... 
Second got 4 
First Continuing... 
Second got 5 
First Continuing... 
Second got 6 
First Continuing... 
Second got 7 
First Continuing... 
Second got 8 
First Continuing... 
Second got 9 
First Continuing... 
Second got 10
First Received 'Stop' command.  Stopping. 
Second Received 'done'

The <0.71.0> is the process id (not to be confused with a *nix PID) and will very likely be different on your machine. Let's look at the 'start' function.

start() ->
  SecondPID = spawn(messagetest, second, []),
  spawn(messagetest, first, [0, SecondPID])

The first line is the function definition. Simple enough. The second line executes the spawn command which kicks off an Erlang process that executes the function called 'second' in the 'messagetest' module. No parameters are passed ('[]'). The return value of spawn is the process ID of the started process.

The second line also spawns another Erlang process, passing it a value of 0 as well as the process ID of the process that was just spawned.

Ironically enough, we'll examine the function called 'second' first as it's a bit easier to understand, I think. 'second' consists entirely of a receive/end block who's job is to receive messages passed to the function. Inter-process messages are sent via:

process_id_of_recipient ! message

When a process receives a message, it is placed in a FIFO queue and the next message is examined when a receive/end is encountered. Message processing is very similar to a case/switch in other languages in that the message is compared against a list of expected values and when a match is found the corresponding code is evaluated. If no match is found, the next message is examined while keeping the first message in the queue. If the second doesn't match, the third is examined and so on until either a message matches at which point it is removed from the queue, or there are no more messages. In the case of the latter, the process blocks and waits for a message it knows how to process.

second() ->
  receive
    done ->
      io:format("Second Received 'done'~n", []);
    {N, FirstPID} ->
      case (N < 10) of
        false -> FirstPID ! stop;
        true -> FirstPID ! continue
      end,
      io:format("Second got ~p ~n", [N]),
      second()
  end.

Here we can see that the function is written to respond to either the word 'done' or a tuple matching a {x,y} pattern. In the case of the word 'done' for example, a notice is printed and the function ends.

In the event the second pattern matches the message, the value of 'N' is evaluated to see if it's less than 10. If it isn't, a 'stop' message is sent to whatever process has a PID of 'FirstPID' - I'll show where this comes from in a moment. If it is, the message 'continue' is sent instead.

In either case, the number it received in the variable 'N' is printed and the function calls itself to begin listening again. Now over to the 'first' function.

first(N, SecondPID) ->
  SecondPID ! {N + 1, self()},
  receive
    continue ->
      io:format("First Continuing... ~n", []),
      first(N + 1, SecondPID);
    stop ->
      io:format("First Received 'Stop' command.  Stopping.~n", []),
      SecondPID ! done
  end.

'first' expects two parameters, a number and a PID. As you recall from the 'start' function, this information was supplied by calling:

spawn(messagetest, first, [0, SecondPID]).

'first' uses:

SecondPID ! {N + 1, self()},

This matches the

continue ->
  io:format("First Continuing... ~n", []),
  first(N + 1, SecondPID);

block, so a message is printed and 'first' calls itself passing N + 1 and the PID of the 'second' function's process. This cycle repeats until 'first' tells 'second' it's reached the number 10.

At this point, 'second' send a message consisting of 'stop' back to 'first'.

stop ->
  io:format("First Received 'Stop' command.  Stopping.~n", []),
  SecondPID ! done

'first' processes the message and prints that it is shutting down. Finally, the message 'done' is sent back to 'second'. 'first' shuts itself down as it has nothing left to do.

done ->
  io:format("Second Received 'done'~n", []);

'second' receives the 'done' message from 'first', prints an acknowledgment, and decides it's going out for a beer or something so it shuts down. The program terminates.

There's a lot going on and it can take some time to wrap your head around it, but probably significantly less time than you might think. I find Erlang to be really enjoyable to write, although I think the error messages could use some work. The people who wrote Erlang apparently founded the 'Obscure Error Message College' (note: in the 1990s, it was renamed to 'The Javascript "Has No Properties" Academy').

I hope this was useful to someone and that you enjoy working in Erlang.

Finally got all the zones configured properly and transferred to my DNS server. In celebration, here are some additions to the Vector class. Enjoy.

require 'matrix'

class Vector
 
  def include?(search_term)
    self.to_a.include?(search_term)
  end
   
  def to_float
    new_elements = []
    self.to_a.each do |element|
        element = element.to_f if element.class == Fixnum
        new_elements << element
      end
          
      Vector.elements(new_elements)
  end

  def pretty  
    counter = 0 
    self.map do |element|
      puts "[#{counter}] = #{element}"
      counter += 1
    end
  end

  def empty?
    return true if self.nil?  
    empty = true
    self.map do |element|
      if case(element)
           when Fixnum
             element == 0
           when nil
             true
           when String
             element.empty? or element == ' '
           when Float
             element == 0.0
           end
        next
      else
        empty = false
        break
      end
    end

    empty
  end

  def self.random(size, max_value = 50)
    elements = []
    (1..size).each do |i|
      elements << rand(max_value) + 1
    end

    Vector.elements(elements)
  end

end