From quite a year I’m mainly working with Ember. Recently, due to recent changes on ember data about embedded records i thought it could fit my needs.

But after a bit of digging, i’ve easily found that embedded records do still want an id key to be present to work. As of my API, i have repeated data structure in different API calls, so i wanted to share the capability of a model in many context. Embedded models were great but unfortunately the problem to have an id made me unable to use them… as they are.

Digging i’ve found that modifying indeed ember-data to do what i wanted was really easy: a mixin that properly configured can make ember-data models get the id automatically client side.

Here is the code I’ve ended up with. I started from embedded record mixin and modified its behavior to support noKey option in attrs structure.

This is an example in how to use it. With this post and maybe some merge request on github i’ll try to make ember-data people aware of the problem and maybe evaluate this solution.

Looks like I’m not alone in this problem. Here is a list of documents i’ve read/tried before writing my own mixin. Of course none of them worked in my case  but led me the way to the code i wrote.

Ember data serializer that normalizes embedded records without ids

Jsfiddle demo

Serializing Embedded Relationships with Ember Data 1.0.0 beta

How to make embedded hasMany relationship work with ember-data

One very good resources to find regexp or base for your regexp is for sure RegexLib in which a huge collection of regexps is exposed. 90% of the time I’ve to write a regexp I start browsing that site for ideas or pattern to use in my regexp. From this base i tend then to build my own regexp like in this case:

portsList = /^$|^(6[0-5]?[0-9]{0,2}[0-5]?|6[0-4]?[0-9]{0,3}|[1-5][0-9]{0,4}|[1-9][0-9]{0,3})(-(6[0-5]?[0-9]{0,2}[0-5]?|6[0-4]?[0-9]{0,3}|[1-5][0-9]{0,4}|[1-9][0-9]{0,3}))?((\s*,\s*)(6[0-5]?[0-9]{0,2}[0-5]?|6[0-4]?[0-9]{0,3}|[1-5][0-9]{0,4}|[1-9][0-9]{0,3})(-(6[0-5]?[0-9]{0,2}[0-5]?|6[0-4]?[0-9]{0,3}|[1-5][0-9]{0,4}|[1-9][0-9]{0,3}))?)*$/;

This *huge and hard to maintain* regexp recognizes a valid TCP/UDP port or port range which is a number between 0 and 65535 for port and a port-port for port range (there is an error in that regexp, have fun in finding it ). Never to say, that code is unmainteable, that’s why I usually try to describe regexps in a EBNF way. In EBNF you split your regexp in smaller regexp in variables such that you can reuse them.  For example we can define:

• port = […….]
• portList = port || port-port

This is usually easy in programming languages (like in python with its re module) but I found a little bit more tricky in javascript. So here are my rules:

1. Always use the RegExp object
2. Always compose regexp from strings
3. Always isolate regexp string between parenthesis.

In javascript you can create regexp using the RegExp object or using the / / operator. The advantage of the former is that you can compose the regexp using  strings. Look at this example:

var my_regexp = new RegExp("^hello world$");

var my_regexp = /^hello world$/;

The two statements are equivalent. The only difference is that the former is constructed with a string and we can thus put the string into a variable and compose those variables to a valid regexp string.

Isolation of the regexp between parenthesis may look verbose but makes the regexp less error prone and easy to debug. Applying all the rules you would be able to do something like:

var port = "(6553[0-5]|655[0-2]\\d|65[0-4]\\d\\d|6[0-4]\\d{3}|[1-5]\\d{4}|[1-9]\\d{0,3}|0)";

var port_or_port_range = "("+port+"(-"+port+")?)";

var my_regexp = new RegExp("^("+port_or_port_range+")((\\s*,\\s*)("+port_or_port_range+"))*$");

With these three lines we accomplished the same job of the first long regexp I posted. Much easier to maintain and use. Note the leading ( and trailing ) to the port regexp string and the my_regexp that uses the regexp object.

ExtJs has that feature: the very well designed Ext.Loader can load arbitrary javascript files just looking on the configured paths of the different resources. On the other side of the most common libraries, Jquery has the $.getScript() function that does exactly the same thing, in a more dummy way. It just loads the javascript in the path passed as argument.

None of them are able to handle css, tho. So I wrote a small, library independent function to load a javascript or css dynamically. Here is the code:

var loadfile = (function(){
    //our hash table to keep record of already loaded files
    var loadedfiles = {};
    //TODO add callback
    return function(filenames, callback){

        if(!(filenames instanceof Array)){
            if(typeof(filenames) == 'string'){
                filenames = [filenames];
            }else{
                throw 'filenames is not an Array or a string';
            }
        }

        var map = {
            js: function(filename){
                var tag=document.createElement('script');
                tag.setAttribute("type","text/javascript");
                tag.setAttribute("src", filename);
                return tag;
            },
            css: function(filename){
                var tag=document.createElement("link");
                tag.setAttribute("rel", "stylesheet");
                tag.setAttribute("type", "text/css");
                tag.setAttribute("href", filename);
                return tag;
            }
        },i,length,type,tag,filename, ext;

        length = filenames.length;
        if(length>0){
            for(i=0; i<length; i++){
                filename = filenames[i];

                if(loadedfiles[filename])
                    //continue if we have already loaded this file
                    continue;

                type = filename.lastIndexOf('.');

                if(type === -1)
                    throw filename+' is not a valid filename with extension';

                ext = filename.slice(type+1);

                //create correct tag
                tag = map[ext](filename);

                //append to head
                document.getElementsByTagName("head")[0].appendChild(tag);

                //update hash table for files we've already loaded
                loadedfiles[filename] = true;

                //call callback
                if(callback && typeof(callback) === 'function')
                    callback(filename);
            }
        }
    };
})();

The loadfile function uses closure to store an hashmap of the loaded file and accepts an single string or an array of strings representing the paths to load. It also accepts an optional callback function that will be called if and when the script has been loaded into the page.

Feel free to use it as you wish. And if you like, give me some credit!

• John D. Cook from 2009, a bit out of date, about the limits of IronPython http://www.johndcook.com/blog/2009/02/26/ironpython-is-a-one-way-gate/
• IronClad about patching some of the IronPython limitations, mainly the ability to run python C modules http://code.google.com/p/ironclad/
• Zenon Ochal about calling python code from C# and vice versa http://www.simple-talk.com/dotnet/.net-framework/dynamic-language-integration-in-a-c-world/
• Microsoft on IronPython architecture and differencies with CPython http://msdn.microsoft.com/en-us/magazine/cc300810.aspx

Run python from C# and vice versa

But thanks to .Net 4.0 It seems that a lot of troubles have been resolved introducing the new dynamic type in C#. With that type I can finally do some work in python then export any class to my C# world.

There are still some issues in using any python module: Ironpython is not compatible with the normal python interpreter, or CPython. You can use Ironclad to make Ironpython run C CPython modules but they don’t support Python 2.7 yet and the last update from their release is from  07/2010.

Install IronPython

After installing IronPath you have to modify windows PATH system variable such that you can execute ipy in a shell and get a IronPython console. To do so you have to add the following two lines at the end of your PATH variable on Windows:

;c:Program Files (x86)IronPython 2.7;c:Program Files (x86)IronPython 2.7Sciprts

Install Setuptools

Tried to install distribute. Of course it didn’t work. It crashed. I’ve successfully managed to setup setuptools but their support is broken and i’m unable to install simple python eggs due to multiple error.   These are the steps that I’ve followed

c:UsersvinzDesktop>ipy distribute_setup.py
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.2
7.tar.gz
Extracting in c:usersvinzappdatalocaltemptmpanru2x
Now working in c:usersvinzappdatalocaltemptmpanru2xdistribute-0.6.27
Installing Distribute

Process is terminated due to StackOverflowException.
Something went wrong during the installation.
See the error message above.

Use setuptools instead http://peak.telecommunity.com/dist/ez_setup.py

Open a priviledged shell http://www.howtogeek.com/howto/windows-vista/run-a-command-as-administrator-from-the-windows-vista-run-box/

ipy ez_setup.py

Since there is no support to zlib in ironpython you cannot install any egg anyway even if you installed setuptools.

Install an egg

I’ve managed to install setuptools, however installing a simple package fails because .pth files are not managed correctly. So the best way of installing eggs is just to take their source, maybe unpacking an egg since it’s just a zip file, and copy directly on the IronPython site-packages directory. I’ll still describe the process I did just in case some reader sees that I’ve done a mistake.

To install an egg, just unzip it and run:

ipy -X:Frames setup.py install --user

The –user will install the package for user only. To install the package system-wide you have to install from an administrator shell.

The best thing to do to install an egg is just to unpack it, it’s a zip file, then copy the directory of your library in C:Program Files (x86)IronPython 2.7Libsite-packages . That directory will be included in sys.path of every ipy interpreter.

First problems

Just run a first example which uses ElementTree and i got:

NotImplementedError: iterparse is not supported on IronPython. (CP #31923)

Surfing a bit the web I found that It’s a bug from Dec 2011 http://ironpython.codeplex.com/workitem/31579

So I’ve downloaded and installed elementTree, It will install on normal Python’s site-packages, so you have to move the folder unver IronPython’s site-packages. After doing so, when you need to import etree just do:

import platform
if platform.python_implementation() == 'IronPython':
from elementtree import ElementTree
else:
import xml.etree.ElementTree

This code will import the just installed elementtree library or the system library one depending if you are on normal Python interpreter (CPython) or IronPython.

Conclusions

From this experience I’ve learnt many things:

• IronPython is not a full python interpreter: it lacks many basic support like the one for setuptool and distribute
• IronPython is very bugged, even the standard library is bugger. Thus import as external module everything you need.
• IronPython cannot use the vast majority of python modules because to support .NET jit it breaks CPython compatibility

Thus. Keep away from it! It’s just crap. Microsoft’s dream of .Net simply doesn’t work for python. Yeah, you can access Python functions and class from C# but you miss the goal that makes people use python: a very huge amount of libraries and packages that are easy to install and integrate. Sorry Microsoft, it’s yet another #Fail

pip install mysql-python

and you end up with an error which is the following (or similar):

(env)goshawk@earth:~/Projects/bvisible$ pip install mysql-python
Downloading/unpacking mysql-python
  Running setup.py egg_info for package mysql-python
    sh: mysql_config: not found
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/home/goshawk/Projects/bvisible/env/build/mysql-python/setup.py", line 15, in <module>
        metadata, options = get_config()
      File "setup_posix.py", line 43, in get_config
        libs = mysql_config("libs_r")
      File "setup_posix.py", line 24, in mysql_config
        raise EnvironmentError("%s not found" % (mysql_config.path,))
    EnvironmentError: mysql_config not found
    Complete output from command python setup.py egg_info:
    sh: mysql_config: not found

Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "/home/goshawk/Projects/bvisible/env/build/mysql-python/setup.py", line 15, in <module>

    metadata, options = get_config()

  File "setup_posix.py", line 43, in get_config

    libs = mysql_config("libs_r")

  File "setup_posix.py", line 24, in mysql_config

    raise EnvironmentError("%s not found" % (mysql_config.path,))

EnvironmentError: mysql_config not found

----------------------------------------
Command python setup.py egg_info failed with error code 1
Storing complete log in /home/goshawk/.pip/pip.log

The solution to the problem is to install the libmysqlclient-dev package in this way:

sudo apt-get install libmysqlclient-dev

You can now run pip install mysql-python without any further problem.

My solution was to add a .flat() method such that when called it serializes as python dict the instance, expanding relationships using the most significant fields. Who is the one which sets that significant fields ? It’s you, straight in the model definition. In this way the configuration is distributed within models and you obtain exactly what you wanna have for that relationships. You can also get more than one field for each relation and just join them using a separator. But let’s look at the code:

class CommonObj(object):
    def __repr__(self):
        return str(self.__dict__)

    def __str__(self):
        return self.__repr__()

    def flat(self, mapping={}, func=lambda x:x, separator=' '):
        '''
            Emit a dictionary representing the class. It's able to expand the relationship between classes
            using the special __config__ class you can define in your models. When representation will occour
            it will expand the relations and will return a single field value which is the result of the join
            using the fields_separator of the main_fields list defined in __config__ for each related class.
            If the __config__ is missed it will try to use the name of the relation as the value for the emitted
            dictionary.

            mapping defines how to map the results. it's a dictionary composed by {existent_key:rep_key ...} which
            changes all the existent methods of the object (the existent_key) with the rep_key
        '''

        def get_wanted_rep(key):
            if key in mapping:
                return mapping[key]
            return key

        res = dict()
        iter = dict(self.__dict__)
        for k,v in iter.iteritems():
            if k[0] == '_':
                continue
            elif k[-3:] == '_id' and k != 'id':
                ist = getattr(self, k[:-3])
                if ist:
                    try:
                        attrs = ist.__class__.__config__.main_fields
                    except AttributeError:
                        attrs = [k[:-3]]

                    fields = list()
                    for attr in attrs:
                        fields.append(func(unicode(getattr(ist, attr))))

                    try:
                        sep = ist.__class__.__config__.fields_separator
                        if sep:
                            separator = sep
                    except AttributeError:
                        pass

                    res[get_wanted_rep(k[:-3])] = separator.join(fields)
            else:
                try:
                    res[get_wanted_rep(k)] = func(v)
                except AttributeError:
                    #needed to print numbers
                    res[get_wanted_rep(k)] = v
        return res

class Persona(CommonObj, models.Model):
    nome = models.CharField(max_length=30)
    cognome = models.CharField(max_length=30)
    secondo_cognome = models.CharField(max_length=30, blank=True, null=True)
    indirizzo = models.CharField(max_length=100, blank=True, null=True)
    citta = models.CharField(max_length=15, blank=True, null=True)
    provincia = models.CharField(max_length=30, blank=True, null=True)
    cap = models.IntegerField(blank=True, null=True)
    telefono = models.CharField(max_length=15, blank=True, null=True)
    cellulare = models.CharField(max_length=10, blank=True, null=True)
    email = models.EmailField(blank=True, null=True)
    cliente = models.BooleanField()
    problemi_udito = models.BooleanField()
    porta_apparecchio = models.BooleanField()
    venuto = models.BooleanField()
    spontaneo = models.BooleanField()
    perdita_media = models.CharField(max_length=5, blank=True, null=True, choices=perdita_media_choices)
    tipo_apparecchio = models.CharField(max_length=30, blank=True, null=True)
    lato_apparecchio = models.IntegerField(blank=True, null=True, choices=lato_choices)
    pila = models.IntegerField(blank=True, null=True, choices=pila_choices)
    note = models.TextField(blank=True, null=True)
    preventivo = models.TextField(blank=True, null=True)

We have defined a model which inherits from the CommonObj class which adds the .flat() method. It’s a smart and configurable serializer that expands relations trying to guess the value (it gets the value which has the same name of the relation) it’s able to perform operations thanks to the lambda function parameter to the returned values. Here is an example of this powerful serializer.

goshawk@earth:~/Projects/cacerp/cacerp$ python manage.py shell
Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56)
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: from  callcenter.models import Persona

In [2]: p = Persona.objects.get(pk=1)

In [4]: p.flat()
Out[4]:
{'cap': 20151L,
 'cellulare': u'',
 'citta': u'MILANO',
 'cliente': False,
 'cognome': u'XXXXX',
 'email': u'',
 'id': 1L,
 'indirizzo': u'VIA MARIO BORSA 24',
 'lato_apparecchio': None,
 'nome': u'XXXXXXX',
 'note': u'AS ',
 'perdita_media': None,
 'pila': None,
 'porta_apparecchio': False,
 'preventivo': u'',
 'problemi_udito': False,
 'provincia': u'MI',
 'secondo_cognome': u'',
 'spontaneo': False,
 'telefono': u'XXXXXXX',
 'tipo_apparecchio': u'',
 'venuto': False}

It’s possible do define a mapping between the model names and the wanted result. Let’s assume we want CAP instead of cap as result and we want city instead of citta. It’s very easy to accomplish it:

In [9]: p.flat(mapping={'cap':'CAP', 'citta':'city'})
Out[9]:
{'CAP': 20151L,
 'cellulare': u'',
 'city': u'MILANO',
 'cliente': False,
 'cognome': u'XXXXX',
 'email': u'',
 'id': 1L,
 'indirizzo': u'VIA MARIO BORSA 24',
 'lato_apparecchio': None,
 'nome': u'XXXXXX',
 'note': u'AS ',
 'perdita_media': None,
 'pila': None,
 'porta_apparecchio': False,
 'preventivo': u'',
 'problemi_udito': False,
 'provincia': u'MI',
 'secondo_cognome': u'',
 'spontaneo': False,
 'telefono': u'XXXXXX',
 'tipo_apparecchio': u'',
 'venuto': False}

As you can see it’s changed as our defined mapping. Another powerful feature is to do an action on the values thanks to the func function parameter. Here is what we can do:

In [30]: p.flat(func=lambda x: x.capitalize())
Out[30]:
{'cap': 20151L,
 'cellulare': u'',
 'citta': u'Milano',
 'cliente': False,
 'cognome': u'Xxxxxx',
 'email': u'',
 'id': 1L,
 'indirizzo': u'Via mario borsa 24',
 'lato_apparecchio': None,
 'nome': u'Xxxxx',
 'note': u'As ',
 'perdita_media': None,
 'pila': None,
 'porta_apparecchio': False,
 'preventivo': u'',
 'problemi_udito': False,
 'provincia': u'Mi',
 'secondo_cognome': u'',
 'spontaneo': False,
 'telefono': u'xxxxxxxx',
 'tipo_apparecchio': u'',
 'venuto': False}

As you can verify, the values have been capitalized (first character with maiusc on and with maiusc off all the others).
Hope you like it and helps!

In these days i’m experimenting with Cherokee webserver and pylons applications deployed via wsgi. I’ve seen that uwsgi which is the preferred way to deploy a python app on cherokee is not packaged on Ubuntu 10.10. Here is how you can get the latest uwsgi on your server.

sudo apt-get install libxml2-dev python-pip python2.6-dev

sudo pip install http://projects.unbit.it/downloads/uwsgi-latest.tar.gz

For ubuntu 11.10 oneiric do:

sudo apt-get install libxml2-dev python-pip python2.7-dev

sudo pip install http://projects.unbit.it/downloads/uwsgi-latest.tar.gz

That’s all. you have now uwsgi in your path ready to be used by cherokee or any other webserver (ngnix maybe) ?

Hope this helps

Today on Planet Ubuntu i found a nice post about git. Well. It’s the revision control system developed by Linus Torvalds itself after the BitKeeper debacle. As this post says, git sucks in a lot of ways. i completely agree with Scott when he says so and he shows how git is cryptic and hard to use.

Instead of using git, i use mercurial which does exactly what i want as i want and it’s really well documented. And it’s distribuited of course