At first word, Metaprogramming seems a very funky and alien thing but if you have ever worked with decorators or metaclasses, you were doing metaprogramming there. In nutshell, we can say metaprogramming is the code that manipulates code.

In this article, we are going to discuss Metaclasses, why and when we should use them, and what are the alternatives. This is a fairly advanced Python topic and the following prerequisite are expected:

• OOP concept in Python
• Decorators in Python

Note: This article considers Python 3.3 and above

Metaclasses

In Python, everything has some type associated with it. For example, if we have a variable having an integer value then its type is int. You can get the type of anything using type() function.

num = 23
print("Type of num is:", type(num)) 
  
lst = [1, 2, 4] 
print("Type of lst is:", type(lst)) 
  
name = "Atul"
print("Type of name is:", type(name))

Output:

Type of num is: <class 'int'>
Type of lst is: <class 'list'>
Type of name is: <class 'str'>

Every type in Python is defined by Class. So in the above example, unlike C or Java where int, char, and float are primary data types, in Python they are objects of int class or str class. So we can make a new type by creating a class of that type. For example, we can create a new type of Student by creating a Student class.

class Student: 
    pass
stu_obj = Student() 
  
# Print type of object of Student class 
print("Type of stu_obj is:", type(stu_obj)) 

Output:

Type of stu_obj is: <class '__main__.Student'>

A Class is also an object, and just like any other object, it’s an instance of something called Metaclass. A special class type creates these Class objects. The type class is default metaclass which is responsible for making classes. For example, in the above example, if we try to find out the type of Student class, it comes out to be a type.

class Student: 
    pass
  
# Print type of Student class 
print("Type of Student class is:", type(Student)) 

Output:

Type of Student class is: <class 'type'>

Because Classes are also an object, they can be modified in the same way. We can add or subtract fields or methods in class in the same way we did with other objects. For example:

# Defined class without any 
# class methods and variables 
class test:pass
  
# Defining method variables 
test.x = 45
  
# Defining class methods 
test.foo = lambda self: print('Hello') 
  
# creating object 
myobj = test() 
  
print(myobj.x) 
myobj.foo() 

Output:

45
Hello

This whole meta thing can be summarized as – Metaclass creates Classes and Classes creates objects

A metaclass is responsible for the generation of classes, so we can write our own custom metaclasses to modify the way classes are generated by performing extra actions or injecting code. Usually, we do not need custom metaclasses but sometimes it’s necessary.
There are problems for which metaclass and non-metaclass based solutions are available (often simpler) but in some cases, only metaclass can solve the problem. We will discuss such a problem in this article.

Creating custom Metaclass

To create our custom metaclass, our custom metaclass have to inherit type metaclass and usually override:

• __new__(): It’s a method which is called before __init__(). It creates the object and returns it. We can override this method to control how the objects are created.
• __init__(): This method just initializes the created object passed as a parameter

We can create classes using type() function directly. It can be called in the following way:

1. When called with only one argument, it returns the type. We have seen it before in the above examples.
2. When called with three parameters, it creates a class. The following arguments are passed to it:
   1. Class name
   2. Tuple having base classes inherited by class
   3. Class Dictionary: It serves as a local namespace for the class, populated with class methods and variables

Consider this example:

def test_method(self): 
    print("This is Test class method!") 
  
# creating a base class  
class Base: 
    def myfun(self): 
        print("This is inherited method!") 
  
# Creating Test class dynamically using 
# type() method directly 
Test = type('Test', (Base, ), dict(x="atul", my_method=test_method)) 
  
# Print type of Test  
print("Type of Test class: ", type(Test)) 
  
# Creating instance of Test class 
test_obj = Test() 
print("Type of test_obj: ", type(test_obj)) 
  
# calling inherited method 
test_obj.myfun() 
  
# calling Test class method 
test_obj.my_method() 
  
# printing variable 
print(test_obj.x) 

Output:

Type of Test class:  <class 'type'>
Type of test_obj:  <class '__main__.Test'>
This is inherited method!
This is Test class method!
atul

Now let’s create a metaclass without using type() directly. In the following example, we will be creating a metaclass MultiBases which will check if the class being created has inherited from more than one base class. If so, it will raise an error.

# our metaclass 
class MultiBases(type): 
    # overriding __new__ method 
    def __new__(cls, clsname, bases, clsdict): 
        # if no of base classes is greator than 1 
        # raise error 
        if len(bases)>1: 
            raise TypeError("Inherited multiple base classes!!!") 
          
        # else execute __new__ method of super class, ie. 
        # call __init__ of type class 
        return super().__new__(cls, clsname, bases, clsdict) 
  
# metaclass can be specified by 'metaclass' keyword argument 
# now MultiBase class is used for creating classes 
# this will be propagated to all subclasses of Base 
class Base(metaclass=MultiBases): 
    pass
  
# no error is raised 
class A(Base): 
    pass
  
# no error is raised 
class B(Base): 
    pass
  
# This will raise an error! 
class C(A, B): 
    pass

Output:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 8, in __new__
TypeError: Inherited multiple base classes!!!

Solving problems with metaclass

There are some problems which can be solved by decorators (easily) as well as by metaclasses. But there are a few problems whose result can only be achieved by metaclasses. For example, consider a very simple problem of code repetition.

We want to debug class methods, what we want is that whenever the class method executes, it should print its fully qualified name before executing its body.

The very first solution that comes to our mind is using method decorators, following is the sample code:

from functools import wraps 
  
def debug(func): 
    '''decorator for debugging passed function'''
      
    @wraps(func) 
    def wrapper(*args, **kwargs): 
        print("Full name of this method:", func.__qualname__) 
        return func(*args, **kwargs) 
    return wrapper 
  
def debugmethods(cls): 
    '''class decorator make use of debug decorator 
       to debug class methods '''
      
    # check in class dictionary for any callable(method) 
    # if exist, replace it with debugged version 
    for key, val in vars(cls).items(): 
        if callable(val): 
            setattr(cls, key, debug(val)) 
    return cls
  
# sample class 
@debugmethods
class Calc: 
    def add(self, x, y): 
        return x+y 
    def mul(self, x, y): 
        return x*y 
    def div(self, x, y): 
        return x/y 
      
mycal = Calc() 
print(mycal.add(2, 3)) 
print(mycal.mul(5, 2))

Output:

Full name of this method: Calc.add
5
Full name of this method: Calc.mul
10

This solution works fine but there is one problem, what if we want to apply this method decorator to all subclasses which inherit this Calc class. In that case, we have to separately apply the method decorator to every subclass just like we did with the Calc class.

The problem is if we have many such subclasses, then in that case we won’t like adding a decorator to each one separately. If we know beforehand that every subclass must have this debug property, then we should look up a metaclass based solution.

Have a look at this metaclass based solution, the idea is that classes will be created normally and then immediately wrapped up by debug method decorator:

from functools import wraps 
  
def debug(func): 
    '''decorator for debugging passed function'''
      
    @wraps(func) 
    def wrapper(*args, **kwargs): 
        print("Full name of this method:", func.__qualname__) 
        return func(*args, **kwargs) 
    return wrapper 
  
def debugmethods(cls): 
    '''class decorator make use of debug decorator 
       to debug class methods '''
      
    for key, val in vars(cls).items(): 
        if callable(val): 
            setattr(cls, key, debug(val)) 
    return cls
  
class debugMeta(type): 
    '''meta class which feed created class object 
       to debugmethod to get debug functionality 
       enabled objects'''
      
    def __new__(cls, clsname, bases, clsdict): 
        obj = super().__new__(cls, clsname, bases, clsdict) 
        obj = debugmethods(obj) 
        return obj 
      
# base class with metaclass 'debugMeta' 
# now all the subclass of this  
# will have debugging applied 
class Base(metaclass=debugMeta):pass
  
# inheriting Base 
class Calc(Base): 
    def add(self, x, y): 
        return x+y 
      
# inheriting Calc 
class Calc_adv(Calc): 
    def mul(self, x, y): 
        return x*y 
  
# Now Calc_adv object showing 
# debugging behaviour 
mycal = Calc_adv() 
print(mycal.mul(2, 3)) 

Output:

Full name of this method: Calc_adv.mul
6

When to use Metaclasses

Most of the time we are not using metaclasses, they are like black magic and usually for something complicated, but a few cases where we use metaclasses are:

• As we have seen in the above example, metaclasses propagate down the inheritance hierarchies. It will affect all the subclasses as well. If we have such a situation, then we should use metaclasses.
• If we want to change class automatically when it is created
• If you are an API developer, you might use metaclasses

Classes as objects

Before understanding metaclasses, you need to master classes in Python. And Python has a very peculiar idea of what classes are, borrowed from the Smalltalk language.

In most languages, classes are just pieces of code that describe how to produce an object. That’s kinda true in Python too:

class ObjectCreator(object):
...       pass
...

my_object = ObjectCreator()
print(my_object)
<__main__.ObjectCreator object at 0x8974f2c>

But classes are more than that in Python. Classes are objects too.

Yes, objects.

As soon as you use the keyword class, Python executes it and creates an OBJECT. The instruction

class ObjectCreator(object):
...       pass
...

creates in memory an object with the name “ObjectCreator”.

This object (the class) is itself capable of creating objects (the instances), and this is why it’s a class.

But still, it’s an object, and therefore:

• you can assign it to a variable
• you can copy it
• you can add attributes to it
• you can pass it as a function parameter

e.g.:

print(ObjectCreator) # you can print a class because it's an object
<class '__main__.ObjectCreator'>
>>> def echo(o):
...       print(o)
...
echo(ObjectCreator) # you can pass a class as a parameter
<class '__main__.ObjectCreator'>
print(hasattr(ObjectCreator, 'new_attribute'))
False
ObjectCreator.new_attribute = 'foo' # you can add attributes to a class
print(hasattr(ObjectCreator, 'new_attribute'))
True
print(ObjectCreator.new_attribute)
foo
ObjectCreatorMirror = ObjectCreator # you can assign a class to a variable
print(ObjectCreatorMirror.new_attribute)
foo
print(ObjectCreatorMirror())
<__main__.ObjectCreator object at 0x8997b4c>

Creating classes dynamically

Since classes are objects, you can create them on the fly, like any object.

First, you can create a class in a function using class:

def choose_class(name):
...     if name == 'foo':
...         class Foo(object):
...             pass
...         return Foo # return the class, not an instance
...     else:
...         class Bar(object):
...             pass
...         return Bar
...
MyClass = choose_class('foo')
print(MyClass) # the function returns a class, not an instance
<class '__main__.Foo'>
print(MyClass()) # you can create an object from this class
<__main__.Foo object at 0x89c6d4c>

But it’s not so dynamic, since you still have to write the whole class yourself.

Since classes are objects, they must be generated by something.

When you use the class keyword, Python creates this object automatically. But as with most things in Python, it gives you a way to do it manually.

Remember the function type? The good old function that lets you know what type an object is:

print(type(1))
<type 'int'>
print(type("1"))
<type 'str'>
print(type(ObjectCreator))
<type 'type'>
print(type(ObjectCreator()))
<class '__main__.ObjectCreator'>

Well, type has a completely different ability, it can also create classes on the fly. type can take the description of a class as parameters, and return a class.

(I know, it’s silly that the same function can have two completely different uses according to the parameters you pass to it. It’s an issue due to backward compatibility in Python)

type works this way:

type(name of the class,
     tuple of the parent class (for inheritance, can be empty),
     dictionary containing attributes names and values)

e.g.:

>>> class MyShinyClass(object):
...       pass

can be created manually this way:

>>> MyShinyClass = type('MyShinyClass', (), {}) # returns a class object
>>> print(MyShinyClass)
<class '__main__.MyShinyClass'>
>>> print(MyShinyClass()) # create an instance with the class
<__main__.MyShinyClass object at 0x8997cec>

You’ll notice that we use “MyShinyClass” as the name of the class and as the variable to hold the class reference. They can be different, but there is no reason to complicate things.

type accepts a dictionary to define the attributes of the class. So:

>>> class Foo(object):
...       bar = True

Can be translated to:

>>> Foo = type('Foo', (), {'bar':True})

And used like a normal class:

>>> print(Foo)
<class '__main__.Foo'>
>>> print(Foo.bar)
True
>>> f = Foo()
>>> print(f)
<__main__.Foo object at 0x8a9b84c>
>>> print(f.bar)
True

And of course, you can inherit from it, so:

>>>   class FooChild(Foo):
...         pass

would be:

>>> FooChild = type('FooChild', (Foo,), {})
>>> print(FooChild)
<class '__main__.FooChild'>
>>> print(FooChild.bar) # bar is inherited from Foo
True

Eventually, you’ll want to add methods to your class. Just define a function with the proper signature and assign it as an attribute.

>>> def echo_bar(self):
...       print(self.bar)
...
>>> FooChild = type('FooChild', (Foo,), {'echo_bar': echo_bar})
>>> hasattr(Foo, 'echo_bar')
False
>>> hasattr(FooChild, 'echo_bar')
True
>>> my_foo = FooChild()
>>> my_foo.echo_bar()
True

And you can add even more methods after you dynamically create the class, just like adding methods to a normally created class object.

>>> def echo_bar_more(self):
...       print('yet another method')
...
>>> FooChild.echo_bar_more = echo_bar_more
>>> hasattr(FooChild, 'echo_bar_more')
True

You see where we are going: in Python, classes are objects, and you can create a class on the fly, dynamically.

This is what Python does when you use the keyword class, and it does so by using a metaclass.

What are metaclasses (finally)

Metaclasses are the ‘stuff’ that creates classes.

You define classes in order to create objects, right?

But we learned that Python classes are objects.

Well, metaclasses are what create these objects. They are the classes’ classes, you can picture them this way:

MyClass = MetaClass()
my_object = MyClass()

You’ve seen that type lets you do something like this:

MyClass = type('MyClass', (), {})

It’s because the function type is in fact a metaclass. type is the metaclass Python uses to create all classes behind the scenes.

Now you wonder why the heck is it written in lowercase, and not Type?

Well, I guess it’s a matter of consistency with str, the class that creates strings objects, and int the class that creates integer objects. type is just the class that creates class objects.

You see that by checking the __class__ attribute.

Everything and I mean everything, is an object in Python. That includes ints, strings, functions, and classes. All of them are objects. And all of them have been created from a class:

>>> age = 35
>>> age.__class__
<type 'int'>
>>> name = 'bob'
>>> name.__class__
<type 'str'>
>>> def foo(): pass
>>> foo.__class__
<type 'function'>
>>> class Bar(object): pass
>>> b = Bar()
>>> b.__class__
<class '__main__.Bar'>

Now, what is the __class__ of any __class__ ?

>>> age.__class__.__class__
<type 'type'>
>>> name.__class__.__class__
<type 'type'>
>>> foo.__class__.__class__
<type 'type'>
>>> b.__class__.__class__
<type 'type'>

So, a metaclass is just the stuff that creates class objects. You can call it a ‘class factory’ if you wish. type is the built-in metaclass Python uses, but of course, you can create your own metaclass.

The __metaclass__ attribute

In Python 2, you can add a __metaclass__ attribute when you write a class (see next section for the Python 3 syntax):

class Foo(object):
    __metaclass__ = something...
    [...]

If you do so, Python will use the metaclass to create the class Foo. Careful, it’s tricky. You write class Foo(object) first, but the class object Foo is not created in memory yet. Python will look for __metaclass__ in the class definition. If it finds it, it will use it to create the object class Foo. If it doesn’t, it will use type to create the class. Read that several times.

When you do:

class Foo(Bar):
    pass

Python does the following:

Is there a __metaclass__ attribute in Foo?

If yes, create in-memory a class object (I said a class object, stay with me here), with the name Foo by using what is in __metaclass__.

If Python can’t find __metaclass__, it will look for a __metaclass__ at the MODULE level, and try to do the same (but only for classes that don’t inherit anything, basically old-style classes). Then if it can’t find any __metaclass__ at all, it will use the Bar‘s (the first parent) own metaclass (which might be the default type) to create the class object. Be careful here that the __metaclass__ attribute will not be inherited, the metaclass of the parent (Bar.__class__) will be. If Bar used a __metaclass__ attribute that created Bar with type() (and not type.__new__()), the subclasses will not inherit that behavior.

Now the big question is, what can you put in __metaclass__ ?

The answer is: something that can create a class.

And what can create a class? type, or anything that subclasses or uses it.

Metaclasses in Python 3

The syntax to set the metaclass has been changed in Python 3:

class Foo(object, metaclass=something):
    ...

i.e. the __metaclass__ attribute is no longer used, in favor of a keyword argument in the list of base classes.

The behavior of metaclasses however stays largely the same.

One thing added to metaclasses in Python 3 is that you can also pass attributes as keyword arguments into a metaclass, like so:

class Foo(object, metaclass=something, kwarg1=value1, kwarg2=value2):
    ...

Read the section below for how python handles this.

Custom metaclasses

The main purpose of a metaclass is to change the class automatically when it’s created. You usually do this for APIs, where you want to create classes matching the current context. Imagine a stupid example, where you decide that all classes in your module should have their attributes written in uppercase. There are several ways to do this, but one way is to set __metaclass__ at the module level. This way, all classes of this module will be created using this metaclass, and we just have to tell the metaclass to turn all attributes to uppercase. Luckily, __metaclass__ can actually be any callable, it doesn’t need to be a formal class (I know, something with ‘class’ in its name doesn’t need to be a class, go figure… but it’s helpful). So we will start with a simple example, by using a function.

# the metaclass will automatically get passed the same argument
# that you usually pass to `type`
def upper_attr(future_class_name, future_class_parents, future_class_attr):
    """
      Return a class object, with the list of its attribute turned
      into uppercase.
    """

    # pick up any attribute that doesn't start with '__' and uppercase it
    uppercase_attr = {}
    for name, val in future_class_attr.items():
        if not name.startswith('__'):
            uppercase_attr[name.upper()] = val
        else:
            uppercase_attr[name] = val

    # let `type` do the class creation
    return type(future_class_name, future_class_parents, uppercase_attr)

__metaclass__ = upper_attr # this will affect all classes in the module

class Foo(): # global __metaclass__ won't work with "object" though
    # but we can define __metaclass__ here instead to affect only this class
    # and this will work with "object" children
    bar = 'bip'

print(hasattr(Foo, 'bar'))
# Out: False
print(hasattr(Foo, 'BAR'))
# Out: True

f = Foo()
print(f.BAR)
# Out: 'bip'

Now, let’s do exactly the same, but using a real class for a metaclass:

# remember that `type` is actually a class like `str` and `int`
# so you can inherit from it
class UpperAttrMetaclass(type):
    # __new__ is the method called before __init__
    # it's the method that creates the object and returns it
    # while __init__ just initializes the object passed as parameter
    # you rarely use __new__, except when you want to control how the object
    # is created.
    # here the created object is the class, and we want to customize it
    # so we override __new__
    # you can do some stuff in __init__ too if you wish
    # some advanced use involves overriding __call__ as well, but we won't
    # see this
    def __new__(upperattr_metaclass, future_class_name,
                future_class_parents, future_class_attr):

        uppercase_attr = {}
        for name, val in future_class_attr.items():
            if not name.startswith('__'):
                uppercase_attr[name.upper()] = val
            else:
                uppercase_attr[name] = val

        return type(future_class_name, future_class_parents, uppercase_attr)

But this is not really OOP. We call type directly and we don’t override or call the parent __new__. Let’s do it:

class UpperAttrMetaclass(type):

    def __new__(upperattr_metaclass, future_class_name,
                future_class_parents, future_class_attr):

        uppercase_attr = {}
        for name, val in future_class_attr.items():
            if not name.startswith('__'):
                uppercase_attr[name.upper()] = val
            else:
                uppercase_attr[name] = val

        # reuse the type.__new__ method
        # this is basic OOP, nothing magic in there
        return type.__new__(upperattr_metaclass, future_class_name,
                            future_class_parents, uppercase_attr)

You may have noticed the extra argument upperattr_metaclass. There is nothing special about it: __new__ always receives the class it’s defined in, as the first parameter. Just like you have self for ordinary methods which receive the instance as the first parameter, or the defining class for class methods.

Of course, the names I used here are long for the sake of clarity, but like for self, all the arguments have conventional names. So a real production metaclass would look like this:

class UpperAttrMetaclass(type):

    def __new__(cls, clsname, bases, dct):

        uppercase_attr = {}
        for name, val in dct.items():
            if not name.startswith('__'):
                uppercase_attr[name.upper()] = val
            else:
                uppercase_attr[name] = val

        return type.__new__(cls, clsname, bases, uppercase_attr)

We can make it even cleaner by using super, which will ease inheritance (because yes, you can have metaclasses, inheriting from metaclasses, inheriting from type):

class UpperAttrMetaclass(type):

    def __new__(cls, clsname, bases, dct):

        uppercase_attr = {}
        for name, val in dct.items():
            if not name.startswith('__'):
                uppercase_attr[name.upper()] = val
            else:
                uppercase_attr[name] = val

        return super(UpperAttrMetaclass, cls).__new__(cls, clsname, bases, uppercase_attr)

Oh, and in python 3 if you do this call with keyword arguments, like this:

class Foo(object, metaclass=Thing, kwarg1=value1):
    ...

It translates to this in the metaclass to use:

class Thing(type):
    def __new__(cls, clsname, bases, dct, kwargs1=default):
        ...

That’s it. There is really nothing more about metaclasses.

The reason behind the complexity of the code using metaclasses is not because of metaclasses, it’s because you usually use metaclasses to do twisted stuff relying on introspection, manipulating inheritance, vars such as __dict__, etc.

Indeed, metaclasses are especially useful to do black magic, and therefore complicated stuff. But by themselves, they are simple:

• intercept a class creation
• modify the class
• return the modified class

Why would you use metaclasses classes instead of functions?

Since __metaclass__ can accept any callable, why would you use a class since it’s obviously more complicated?

There are several reasons to do so:

• The intention is clear. When you read UpperAttrMetaclass(type), you know what’s going to follow
• You can use OOP. Metaclass can inherit from metaclass, and override parent methods. Metaclasses can even use metaclasses.
• Subclasses of a class will be instances of its metaclass if you specified a metaclass-class, but not with a metaclass function.
• You can structure your code better. You never use metaclasses for something as trivial as the above example. It’s usually for something complicated. Having the ability to make several methods and group them in one class is very useful to make the code easier to read.
• You can hook on __new__, __init__ and __call__. Which will allow you to do different stuff. Even if usually you can do it all in __new__, some people are just more comfortable using __init__.
• These are called metaclasses, damn it! It must mean something!

Why would you use metaclasses?

Now the big question. Why would you use some obscure error-prone feature? Well, usually you don’t.

The main use case for a metaclass is creating an API. A typical example of this is the Django ORM.

It allows you to define something like this:

class Person(models.Model):
    name = models.CharField(max_length=30)
    age = models.IntegerField()

But if you do this:

person = Person(name='bob', age='35')
print(person.age)

It won’t return an IntegerField object. It will return an int, and can even take it directly from the database.

This is possible because models.Model defines __metaclass__ and it uses some magic that will turn the Person you just defined with simple statements into a complex hook to a database field. Django makes something complex look simple by exposing a simple API and using metaclasses, recreating code from this API to do the real job behind the scenes.

The last word

First, you know that classes are objects that can create instances. Well, in fact, classes are themselves instances. Of metaclasses.

>>> class Foo(object): pass
>>> id(Foo)
142630324

Everything is an object in Python, and they are all either instances of classes or instances of metaclasses. Except for type. type is actually its own metaclass. This is not something you could reproduce in pure Python and is done by cheating a little bit at the implementation level.

Secondly, metaclasses are complicated. You may not want to use them for very simple class alterations. You can change classes by using two different techniques:

• monkey patching
• class decorators

99% of the time you need class alteration, you are better off using these. But 98% of the time, you don’t need class alteration at all.

The term metaprogramming refers to the potential for a program to have knowledge of or manipulate itself. Python supports a form of metaprogramming for classes called metaclasses. Metaclasses are an esoteric OOP concept, lurking behind virtually all Python code. You are using them whether you are aware of it or not. For the most part, you don’t need to be aware of it. Most Python programmers rarely, if ever, have to think about metaclasses.

When the need arises, however, Python provides a capability that not all object-oriented languages support: you can get under the hood and define custom metaclasses. The use of custom metaclasses is somewhat controversial, as suggested by the following quote from Tim Peters, the Python guru who authored the Zen of Python:

“Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).”

— Tim Peters

There are Pythonistas (as Python aficionados are known) who believe that you should never use custom metaclasses. That might be going a bit far, but it is probably true that custom metaclasses mostly aren’t necessary. If it isn’t pretty obvious that a problem calls for them, then it will probably be cleaner and more readable if solved in a simpler way.

Still, understanding Python metaclasses is worthwhile, because it leads to a better understanding of the internals of Python classes in general. You never know: you may one day find yourself in one of those situations where you just know that a custom metaclass is what you want.

Old-Style vs. New-Style Classes

In the Python realm, a class can be one of two varieties. No official terminology has been decided on, so they are informally referred to as old-style and new-style classes.

Old-Style Classes

In old-style classes, class and type are not quite the same things. An instance of an old-style class is always implemented from a single built-in type called instance. If obj is an instance of an old-style class, obj.__class__ designates the class, but type(obj) is always instance. The following example is taken from Python 2.7:

>>> class Foo:
...     pass
...
>>> x = Foo()
>>> x.__class__
<class __main__.Foo at 0x000000000535CC48>
>>> type(x)
<type 'instance'>


New-Style Classes

New-style classes unify the concepts of class and type. If obj is an instance of a new-style class, type(obj) is the same as obj.__class__:

>>> class Foo:
...     pass
>>> obj = Foo()
>>> obj.__class__
<class '__main__.Foo'>
>>> type(obj)
<class '__main__.Foo'>
>>> obj.__class__ is type(obj)
True
>>> n = 5
>>> d = { 'x' : 1, 'y' : 2 }

>>> class Foo:
...     pass
...
>>> x = Foo()

>>> for obj in (n, d, x):
...     print(type(obj) is obj.__class__)
...
True
True
True

Type and Class

In Python 3, all classes are new-style classes. Thus, in Python 3 it is reasonable to refer to an object’s type and its class interchangeably.

Note: In Python 2, classes are old-style by default. Before Python 2.2, new-style classes weren’t supported at all. From Python 2.2 onward, they can be created but must be explicitly declared as new-style. Remember that, in Python, everything is an object. Classes are objects as well. As a result, a class must have a type. What is the type of a class?

Consider the following:

>>> class Foo:
...     pass
...
>>> x = Foo()

>>> type(x)
<class '__main__.Foo'>

>>> type(Foo)
<class 'type'>

The type of x is class Foo, as you would expect. But the type of Foo, the class itself, is type. In general, the type of any new-style class is type.

The type of the built-in classes you are familiar with is also type:

>>> for t in int, float, dict, list, tuple:
...     print(type(t))
...
<class 'type'>
<class 'type'>
<class 'type'>
<class 'type'>
<class 'type'>

For that matter, the type of type is type as well (yes, really):

>>> type(type)
<class 'type'>

type is a metaclass, of which classes are instances. Just as an ordinary object is an instance of a class, any new-style class in Python, and thus any class in Python 3, is an instance of the type metaclass.

In the above case:

• x is an instance of class Foo.
• Foo is an instance of the type metaclass.
• type is also an instance of the type metaclass, so it is an instance of itself.

Defining a Class Dynamically

The built-in type() function, when passed one argument, returns the type of an object. For new-style classes, that is generally the same as the object’s __class__ attribute:

>>> type(3)
<class 'int'>

>>> type(['foo', 'bar', 'baz'])
<class 'list'>

>>> t = (1, 2, 3, 4, 5)
>>> type(t)
<class 'tuple'>

>>> class Foo:
...     pass
...
>>> type(Foo())
<class '__main__.Foo'>

You can also call type() with three arguments—type(<name>, <bases>, <dct>):

• <name> specifies the class name. This becomes the __name__ attribute of the class.
• <bases> specifies a tuple of the base classes from which the class inherits. This becomes the __bases__ attribute of the class.
• <dct> specifies a namespace dictionary containing definitions for the class body. This becomes the __dict__ attribute of the class.

Calling type() in this manner creates a new instance of the type metaclass. In other words, it dynamically creates a new class.

In each of the following examples, the top snippet defines a class dynamically with type(), while the snippet below it defines the class the usual way, with the class statement. In each case, the two snippets are functionally equivalent.

Example 1

In this first example, the <bases> and <dct> arguments passed to type() are both empty. No inheritance from any parent class is specified, and nothing is initially placed in the namespace dictionary. This is the simplest class definition possible:

>>> Foo = type('Foo', (), {})

>>> x = Foo()
>>> x
<__main__.Foo object at 0x04CFAD50>

>>> class Foo:
...     pass
...
>>> x = Foo()
>>> x
<__main__.Foo object at 0x0370AD50>

Example 2

Here, <bases> is a tuple with a single element Foo, specifying the parent class that Bar inherits from. An attribute, attr, is initially placed into the namespace dictionary:

>>> Bar = type('Bar', (Foo,), dict(attr=100))

>>> x = Bar()
>>> x.attr
100
>>> x.__class__
<class '__main__.Bar'>
>>> x.__class__.__bases__
(<class '__main__.Foo'>,)

>>> class Bar(Foo):
...     attr = 100
...

>>> x = Bar()
>>> x.attr
100
>>> x.__class__
<class '__main__.Bar'>
>>> x.__class__.__bases__
(<class '__main__.Foo'>,)

Example 3

This time, <bases> is again empty. Two objects are placed into the namespace dictionary via the <dct> argument. The first is an attribute named attr and the second a function named attr_val, which becomes a method of the defined class:

>>> Foo = type(
...     'Foo',
...     (),
...     {
...         'attr': 100,
...         'attr_val': lambda x : x.attr
...     }
... )

>>> x = Foo()
>>> x.attr
100
>>> x.attr_val()
100

>>> class Foo:
...     attr = 100
...     def attr_val(self):
...         return self.attr
...

>>> x = Foo()
>>> x.attr
100
>>> x.attr_val()
100

Example 4

Only very simple functions can be defined with lambda in Python. In the following example, a slightly more complex function is defined externally and then assigned to attr_val in the namespace dictionary via the name f:

>>> def f(obj):
...     print('attr =', obj.attr)
...
>>> Foo = type(
...     'Foo',
...     (),
...     {
...         'attr': 100,
...         'attr_val': f
...     }
... )

>>> x = Foo()
>>> x.attr
100
>>> x.attr_val()
attr = 100

>>> def f(obj):
...     print('attr =', obj.attr)
...
>>> class Foo:
...     attr = 100
...     attr_val = f
...

>>> x = Foo()
>>> x.attr
100
>>> x.attr_val()
attr = 100

Custom Metaclasses

Consider again this well-worn example:

>>> class Foo:
...     pass
...
>>> f = Foo()

The expression Foo() creates a new instance of class Foo. When the interpreter encounters Foo(), the following occurs:

• The __call__() method of Foo’s parent class is called. Since Foo is a standard new-style class, its parent class is the type metaclass, so type’s __call__() method is invoked.
• That __call__() method in turn invokes the following:
  • __new__()
  • __init__()

If Foo does not define __new__() and __init__(), default methods are inherited from Foo’s ancestry. But if Foo does define these methods, they override those from the ancestry, which allows for customized behavior when instantiating Foo. In the following, a custom method called new() is defined and assigned as the __new__() method for Foo:

>>> def new(cls):
...     x = object.__new__(cls)
...     x.attr = 100
...     return x
...
>>> Foo.__new__ = new

>>> f = Foo()
>>> f.attr
100

>>> g = Foo()
>>> g.attr
100

This modifies the instantiation behavior of class Foo: each time an instance of Foo is created, by default it is initialized with an attribute called attr, which has a value of 100. (Code like this would more usually appear in the __init__() method and not typically in __new__(). This example is contrived for demonstration purposes.)

Now, as has already been reiterated, classes are objects too. Suppose you wanted to similarly customize instantiation behavior when creating a class like Foo. If you were to follow the pattern above, you’d again define a custom method and assign it as the __new__() method for the class of which Foo is an instance. Foo is an instance of the type metaclass, so the code looks something like this:

# Spoiler alert:  This doesn't work!
>>> def new(cls):
...     x = type.__new__(cls)
...     x.attr = 100
...     return x
...
>>> type.__new__ = new
Traceback (most recent call last):
  File "<pyshell#77>", line 1, in <module>
    type.__new__ = new
TypeError: can't set attributes of built-in/extension type 'type'

Except, as you can see, you can’t reassign the __new__() method of the type metaclass. Python doesn’t allow it.

This is probably just as well. type is the metaclass from which all new-style classes are derived. You really shouldn’t be mucking around with it anyway. But then what recourse is there if you want to customize the instantiation of a class?

One possible solution is a custom metaclass. Essentially, instead of mucking around with the type metaclass, you can define your own metaclass, which derives from type, and then you can muck around with that instead.

The first step is to define a metaclass that derives from type, as follows:

>>> class Meta(type):
...     def __new__(cls, name, bases, dct):
...         x = super().__new__(cls, name, bases, dct)
...         x.attr = 100
...         return x
...

The definition header class Meta(type): specifies that Meta derives from type. Since type is a metaclass, that makes Meta a metaclass as well.

Note that a custom __new__() method has been defined for Meta. It wasn’t possible to do that to the type metaclass directly. The __new__() method does the following:

• Delegates via super() to the __new__() method of the parent metaclass (type) to actually create a new class
• Assigns the custom attribute attr to the class, with a value of 100
• Returns the newly created class

Now the other half of the voodoo: Define a new class Foo and specify that its metaclass is the custom metaclass Meta, rather than the standard metaclass type. This is done using the metaclass keyword in the class definition as follows:

>>> class Foo(metaclass=Meta):
...     pass
...
>>> Foo.attr
100

Voila! Foo has picked up the attr attribute automatically from the Meta metaclass. Of course, any other classes you define similarly will do likewise:

>>> class Bar(metaclass=Meta):
...     pass
...
>>> class Qux(metaclass=Meta):
...     pass
...
>>> Bar.attr, Qux.attr
(100, 100)

In the same way that a class functions as a template for the creation of objects, a metaclass functions as a template for the creation of classes. Metaclasses are sometimes referred to as class factories.

Compare the following two examples:

Object Factory:

>>> class Foo:
...     def __init__(self):
...         self.attr = 100
...

>>> x = Foo()
>>> x.attr
100

>>> y = Foo()
>>> y.attr
100

>>> z = Foo()
>>> z.attr
100

Class Factory

>>> class Meta(type):
...     def __init__(
...         cls, name, bases, dct
...     ):
...         cls.attr = 100
...
>>> class X(metaclass=Meta):
...     pass
...
>>> X.attr
100

>>> class Y(metaclass=Meta):
...     pass
...
>>> Y.attr
100

>>> class Z(metaclass=Meta):
...     pass
...
>>> Z.attr
100

Is This Really Necessary?

As simple as the above class factory example is, it is the essence of how metaclasses work. They allow customization of class instantiation.

Still, this is a lot of fuss just to bestow the custom attribute attr on each newly created class. Do you really need a metaclass just for that? In Python, there are at least a couple of other ways in which effectively the same thing can be accomplished:

Simple Inheritance:

>>> class Base:
...     attr = 100
...

>>> class X(Base):
...     pass
...

>>> class Y(Base):
...     pass
...

>>> class Z(Base):
...     pass
...

>>> X.attr
100
>>> Y.attr
100
>>> Z.attr
100

Class Decorator:

>>> def decorator(cls):
...     class NewClass(cls):
...         attr = 100
...     return NewClass
...
>>> @decorator
... class X:
...     pass
...
>>> @decorator
... class Y:
...     pass
...
>>> @decorator
... class Z:
...     pass
...

>>> X.attr
100
>>> Y.attr
100
>>> Z.attr
100

Conclusion

As Tim Peters suggests, metaclasses can easily veer into the realm of being a “solution in search of a problem.” It isn’t typically necessary to create custom metaclasses. If the problem at hand can be solved in a simpler way, it probably should be. Still, it is beneficial to understand metaclasses so that you understand Python classes in general and can recognize when a metaclass really is the appropriate tool to use.

References

• http://www.dabeaz.com/py3meta/Py3Meta.pdf
• https://stackoverflow.com/questions/100003/what-is-a-metaclass-in-python
• https://blog.ionelmc.ro/2015/02/09/understanding-python-metaclasses/
• https://eli.thegreenplace.net/2011/08/14/python-metaclasses-by-example
• https://realpython.com/python-metaclasses/