Translation for all the non-italian readers: in the June issue of Computer Programming, the leading italian programming magazine, you can find the first of three articles I wrote about Erlang. It’s a great introduction to the language, so if you have an italian friend who wants to know more about this language, tell him to buy the magazine!

Update: the battle continues here.

Unfortunately I’m blocked by an issue I’m still trying to solve. (update: thanks to your suggestions I’ve solved the “folding” issue)

Here’s what I did until now:

(Note: I followed Peter Norvig’s naming for functions as much as possible.)

The words/1 function takes a binary representation of the reference text file and build a list of words:

words(Bin) ->
    {ok, Words} = regexp:split(binary_to_list(Bin), "[^a-zA-Z]"),
    lists:map(fun(X) -> string:to_lower(X) end, Words).

The train/1 function takes a list of words and creates a Set of lower case words with associated frequency:

train(Words) ->
    Dict = ets:new(dictionary, [set]),
    lists:foreach(fun(X) ->
        case ets:insert_new(Dict, {list_to_binary(X), 1}) of
            false -> ets:update_counter(Dict, list_to_binary(X), 1);
            true -> true
        end
    end, Words),
    Dict.

And finally the build_dict/1 function takes a filename and returns the associated word Set:

build_dict(Filename) ->
    {ok, Bin} = file:read_file(Filename),
    train(words(Bin)).

Then there’s the function that generates all the possible “mistakes” from a word. This function is actually made of four different functions that generates different kind of errors:

alphabet() ->
    "abcdefghijklmnopqrstuvwxyz".

deletion_edits(Word) ->
    deletion_edits([], Word, []).
deletion_edits(_, [], Edits) ->
    Edits;
deletion_edits(Before, [Current | After], Edits) ->
	deletion_edits([Current | Before], After,
		[lists:reverse(Before) ++ After | Edits]).

transposition_edits(Word) ->
    transposition_edits([], Word, []).
transposition_edits(_, [], Edits) ->
    Edits;
transposition_edits(_, [_], Edits) ->
    Edits;
transposition_edits(Before, [Current, Next | After], Edits) ->
	transposition_edits([Current | Before], [Next | After],
		[ lists:reverse(Before) ++ [Next, Current] ++ After | Edits]).

alteration_edits(Word) ->
    alteration_edits([], Word, []).
alteration_edits(_, [], Edits) ->
    Edits;
alteration_edits(Before, [Current | After], Edits) ->
	BeforeR = lists:reverse(Before),
	alteration_edits([Current | Before], After,
		[BeforeR ++ [X] ++ After || X <- alphabet()] ++ Edits).

insertion_edits(Word) ->
    insertion_edits([], Word, [[X] ++ Word || X <- alphabet()]).
insertion_edits(_, [], Edits) ->
    Edits;
insertion_edits(Before, [Current | After], Edits) ->
	BeforeR = lists:reverse(Before),
	insertion_edits([Current | Before], After,
		[BeforeR ++ [Current, X] ++ After || X <- alphabet()] ++
		Edits).

We have edits1/1 that generates the 1-level error words and edits1/2 that generates the 2-level (errors of errors) words:

edits1(Word) ->
    lists:usort(deletion_edits(Word) ++ transposition_edits(Word) ++ alteration_edits(Word) ++ insertion_edits(Word)).

edits2(Word) ->
	lists:usort(lists:foldr(fun(A, AccIn) ->
		AccIn ++ edits1(A)
	end, [], edits1(Word))).

The problem is on the edits2/1 function, it generates a list made of sublists but I want it to generate a single flat list instead. I've tryed lists:flatten/1 but it flattens strings too, I need time to find the correct function...

Here's the known/2 function that filters a list of words by returning only the words present in the dictionary:

known(Words, Dict) ->
	lists:filter(fun(Word) ->
		ets:member(Dict, list_to_binary(Word))
	end, Words).

And then there's the gen_candidates/2 function that generates the possible correct words from the mispelled one:

gen_candidates(Word, Dict) ->
	C1 = known([Word], Dict),
	case (length(C1) > 0) of
		true ->
			[Word];
		false ->
			C2 = known(edits1(Word), Dict),
			case (length(C2) > 0) of
				true ->
					C2;
				false ->
					C3 = known(edits2(Word), Dict),
					case (length(C3) > 0) of
						true -> C3;
						false -> [Word]
					end
			end
	end.

And finally the correct/2 function that takes a mispelled word and returns the most probable correct one:

correct(Word, Dict) ->
	lists:foldl(fun(W, {CN, CW}) ->
		case ets:lookup(Dict, list_to_binary(W)) of
			[{LW, LN}] ->
				case LN > CN of
					true -> {LN, LW};
					false -> {CN, CW}
				end;
			_ ->
				{CN, CW}
		end
	end, {0, Word}, gen_candidates(Word, Dict)).

At this point I started trying some words on the shell:

5> Dict = spellcheck:build_dict("big.txt").
14
6> spellcheck:known(["hello", "helo"], Dict).
["hello"]
7> spellcheck:known(["hello", "helo", "true"], Dict).
["hello","true"]
8> spellcheck:gen_candidates("hello", Dict).
["hello"]
9> spellcheck:gen_candidates("helo", Dict).
["felo","halo","held","hell","hello","helm","help","hero"]
10> spellcheck:correct("helo", Dict).
{287,<<"held">>}
11> spellcheck:correct("correctz", Dict).
{38,<<"correct">>}
12> spellcheck:correct("correc", Dict).
{38,<<"correct">>}
13> spellcheck:correct("gues", Dict).
{112,<<"guns">>}
14> spellcheck:correct("guess", Dict).
{17,<<"guess">>}
15> spellcheck:correct("gues", Dict).
{112,<<"guns">>}
16> spellcheck:correct("gue", Dict).
{249,<<"due">>}

It seems to work!

In the next iteration I'll try to optimize and refactor the code, in special way for the edits generators that are too long for my tastes... Suggestions are welcome!

Numbers

Erlang supports integer and floating point numbers. Integer numbers can have almost unlimited size. Try this on the Erlang shell (erl):

2> 4534535*543636345*3445343*6564574523432.
55754546761558695184405188371500200

You can also have floating point numbers, but they have limited precision and size:

3> 3 / 2.
1.50000
4> 2 * 1.5.
3.00000


Strings

Strings are made of quoted text like “This is a String!”. Erlang deals with strings as they were lists of characters. Unfortunately when Erlang was born, his father didn’t knew about Unicode, so we can just have Latin-1 characters (ASCII 0-255) so far.

Atoms

Atoms are names, constant text strings that you can use like C’s enums or Ruby’s symbols.
Atoms must be unquoted alphanumeric strings that begins with a lower case letter. You can also have non alphanumeric atoms that begins with any letter by quoting the string with a single quote.
Examples: apple, mice, dog, banana, ‘Valentina’, ‘This is an Atom!’.

Tuples

Tuples are fixed sets of data. You cannot add anything to a tuple once you created it. Tuples can contain other tuples, atom, strings and lists.
Examples: {this, is, “a tuple”, with, 6, ‘Elements!’}, {1 + 2, milk, banana, [ 1, 2, 3 ]}.

Lists

Lists are similar to tuples but they can contain an undefined amount of items of any type:

10> [1, 2, 3, {this, is, a, tuple}, anatom, [a, list]].
[1,2,3,{this,is,a,tuple},anatom,[a,list]]

It’s also very easy to split a list into his head and his tail, and that’s also the only way to access his elements (you cannot extract the nth element before the nth-1 elements):

11> [Head | Tail] = [a, b, c, d].
[a,b,c,d]
12> Head.
a
13> Tail.
[b,c,d]

You can also create new lists by adding elements in front or at the end of an existing one:

14> [1, 2, 3 | Tail].
[1,2,3,b,c,d]
15> [Head|[1,2,3]].
[a,1,2,3]


Funs

Last but not least we have funs that are anonymous functions. Erlang deals with funs as they were like any other data type. You can assign funs to variables, pass funs to as parameter to function calls and return funs as function results:

15> MyFun = fun(X) -> X * 2 end.
#Fun
16> MyFun(5).
10
17> lists:map(MyFun, [1, 2, 3]).
[2,4,6]


That’s all for now! I hope to have made less mistakes than the last time!

Here’s the program itself:

-module(hello).
-export([hello_world/0]).

hello_world()->
    io:format("Hello World ~n").


What does that means? Briefly:

-module(hello).

Create a module for the following functions, think of it like a namespace declaration.

-export([hello_world/0]).

This tells the compiler that we want to export the hello_world/0 function to the outer world. It means the other modules can call this function. What does the “/0″ means? It means the “hello_world” function with 0 parameters. In Erlang, you can have different functions with the same name and different number of parameters.

hello_world()->

Declare a new function named “hello_world” that accept no parameters, the body will follow the arrow.

    io:format(“Hello World ~n”).

This calls the “format” function from the module “io” that prints out the Hello World message. The “~n” characters prints the Carriage Return, the same as “/n” in other languages.

To run your program you must start the Erlang shell first:


$ erl
Erlang (BEAM) emulator version 5.5.4 [async-threads:0]
Eshell V5.5.4 (abort with ^G)
1> c(hello).
{ok,hello}
2> hello:hello_world().
Hello World
ok
3>


Try it, and understand it. Next time will be a little bit more complex!

Note: if you guess why by copying and pasting the code you get a lot of errors, that’s because SciTe, the editor I use, exports the code with fancy double quotes. That’s a good reason to not being lazy and write the program by yourself! Writing it’s a nice way of learning!

Nobody can predict the future, but I’m going to make a few informed guesses.

Let’s suppose Intel is right: let’s suppose that the Keifer project succeeds. If this happens, then 32 core processors will appear on the market as soon as 2009/2010.

This comes as no surprise; Sun already ships the Niagara with 8 cores running 4 hyperthreads per core (which is equivalent to 32 cores).

This is a development that makes Erlang programmers very happy. They have been waiting 20 years for this to happen, and now it’s payback time.

Here’s the good news for Erlang programmers:

Your Erlang program should just run N times faster on an N core processor