Python __new__ magic method explained

Python is an Object-oriented language, everything is an object in python. Python is having a special type of method called magic methods named with preceded and trailing double underscores.

When we talk about the magic method __new__ we also need to talk about __init__. These methods will be called when you instantiate (The process of creating an instance from class is called instantiation). That is when you create an instance. The magic method __new__ will be called when the instance is being created. Using this method you can customize the instance creation. This is only the method that will be called first then __init__ will be called to initialize the instance when you are creating an instance.

Method __new__ will take the class reference as the first argument followed by arguments that are passed to the constructor (Arguments passed to call of class to create an instance). Method __new__ is responsible to create instances, so you can use this method to customize object creation. Typically method __new__ will return the created instance object reference. Method __init__ will be called once __new__ method completed execution.

You can create a new instance of the class by invoking the superclass’s __new__ method using super. Something like super(currentclass, cls).__new__(cls, [,….])

Usual class declaration and instantiation:

class Foo(object):
    def __init__(self, a, b):
        self.a = a
        self.b = b

    def bar(self):
        pass
    
i = Foo(2, 3)

A class implementation with __new__ method overridden:

class Foo(object):
    def __new__(cls, *args, **kwargs):
        print ("Creating Instance")
        instance = super(Foo, cls).__new__(cls, *args, **kwargs)
        return instance

    def __init__(self, a, b):
        self.a = a
        self.b = b

    def bar(self):
        pass

Output:

i = Foo(2, 3)
# Creating Instance

Note:

You can create instance inside __new__  method either by using super function or by directly calling __new__ method over object  Where if parent class is object. That is,

instance = super(MyClass, cls).__new__(cls, *args, **kwargs)

or

instance = object.__new__(cls, *args, **kwargs)

Things to remember

If __new__ returns an instance of its own class, then the __init__ method of the newly created instance will be invoked with the instance as first (like __init__(self, [, ….]) argument following by arguments passed to __new__ or call of class.  So, __init__ will be called implicitly.

If __new__ method returns something else other than the instance of the class, then instances __init__ method will not be invoked. In this case, you have to call __init__ method yourself.

Applications

Usually, it’s uncommon to override __new__ method, but sometimes it is required if you are writing APIs or customizing class or instance creation, or abstracting something using classes.

SINGLETON USING __NEW__

You can implement the singleton design pattern using __new__ method. Where singleton class is a class that can only have one object. That is an instance of the class.

Here is how you can restrict creating more than one instance by overriding __new__Singleton implementation using __new__:

class Singleton(object):
    _instance = None  # Keep instance reference 
    
    def __new__(cls, *args, **kwargs):
        if not cls._instance:
            cls._instance = object.__new__(cls, *args, **kwargs)
        return cls._instance

It is not limited to the singleton. You can also impose a limit on the total number of created instances limiting the total number of instances that can be created using __new__:

class LimitedInstances(object):
    _instances = []  # Keep track of instance reference
    limit = 5 

    def __new__(cls, *args, **kwargs):
        if not len(cls._instances) <= cls.limit:
            raise RuntimeError, "Count not create instance. Limit %s reached" % cls.limit    
        instance = object.__new__(cls, *args, **kwargs)
        cls._instances.append(instance)
        return instance
    
    def __del__(self):
        # Remove instance from _instances 
        self._instance.remove(self)

Customize Instance Object

You can customize the instance created and make some operations over it before initializer __init__ is called. Also, you can impose restrictions on instance creation based on some constraints:

def createInstance():
    # Do what ever you want to determie if instance can be created
    return True 

class CustomizeInstance(object):

    def __new__(cls, a, b):
        if not createInstance():
            raise RuntimeError, "Count not create instance"
        instance = super(CustomizeInstance, cls).__new__(cls, a, b)
        instance.a = a
        return instance

    def __init__(self, a, b):
        pass

Customize Returned Object

Usually, when you instantiate a class it will return the instance of that class. You can customize this behavior and you can return some random object you want.

Following  one is a simple example to demonstrate that returning a random object other than the class instance:

class AbstractClass(object):

    def __new__(cls, a, b):
        instance = super(AbstractClass, cls).__new__(cls)
        instance.__init__(a, b)
        return 3

    def __init__(self, a, b):
        print ("Initializing Instance", a, b)

Output:

a = AbstractClass(2, 3)
# Initializing Instance 2 3
print(a) 
# 3

Here you can see when we instantiate class it returns  3 instead of instance reference. Because we are returning 3 instead of created instance from __new__ method. We are calling __init__ explicitly.  As I mentioned above, we have to call __init__ explicitly if we are not returning the instance object from __new__ method.

The __new__ method is also used in conjunction with metaclasses to customize class creation

Constructors

With the proliferation of class-based languages, constructors are likely the most popular method for instantiating objects.

Java

class StandardClass {
    private int x;
    public StandardClass() {
        this.x = 5;
    }

    public int getX() {
        return this.x;
    }
}

Python

class StandardClass(object):
    def __init__(self, x):
        self.x = x

Even JavaScript, a prototypical language, has object constructors via the new keyword.

function StandardClass(x) {
    this.x = x;
}

var standard = new StandardClass(5);
alert(standard.x == 5);

Newer is Better

In Python, as well as many other languages, there are two steps to object instantiation:

The New Step

Before you can access an object, it must first be created. This is not the constructor. In the above examples, we use this or self to reference an object in the constructor; the object had already been created by then. The New Step creates the object before it is passed to the constructor. This generally involves allocating space in memory and/or whatever language-specific actions newing up an object requires.

The Constructor Step

Here, the newed-up object is passed to the constructor. In Python, this is when __init__ is called.

Python Object Creation

This is the normal way to instantiate an StandardClass object:

standard = StandardClass(5)
standard.x == 5

StandardClass(5) is the normal instance creation syntax for Python. It performs the New Step followed by the Constructor Step for us. Python also allows us to deconstruct this process:

# New Step
newed_up_standard = object.__new__(StandardClass)
type(newed_up_standard) is StandardClass
hasattr(newed_up_standard,'x') is False

# Constructor Step
StandardClass.__init__(newed_up_standard, 5)
newed_up_standard.x == 5

object.__new__ is the default New Step for object instantiation. It’s what creates an instance from a class. This happens implicitly as the first part of StandardClass(5).

Notice x is not set until after newed_up_standard is run through __init__. This is because object.__new__ doesn’t call __init__. They are disparate functions. If we wanted to perform checks on newed_up_standard or manipulate it before the constructor is run, we could. However, explicitly calling the New Step followed by Constructor Step is neither clean nor scalable. Fortunately, there is an easy way.

Controlling New with __new__

Python allows us to override the New Step of any object via the __new__ magic method.

class NewedBaseCheck(object):
    def __new__(cls):
        obj = super(NewedBaseCheck,cls).__new__(cls)
        obj._from_base_class = type(obj) == NewedBaseCheck
        return obj
    def __init__(self):
        self.x = 5

newed = NewedBaseCheck()
newed.x == 5
newed._from_base_class is True

__new__ takes a class instead of an instance as the first argument. Since it creates an instance, that makes sense. super(NewedClass, cls).__new__(cls) is very important. We don’t want to call object.__new__ directly; again, you’ll see why later.

Why is from_base_class defined in __new__ instead of __init__? It’s metadata about object creation, which makes more semantic sense in __new__. However, if you really wanted to, you could place define _from_base_class:

class StandardBaseCheck(object):
    def __init__(self):
        self.x = 5
        self._from_base_class == type(self) == StandardBaseCheck

standard_base_check = StandardBaseCheck()
standard_base_check.x == 5
standard_base_check._from_base_class is True

There is a major behavioral difference between NewBaseCheck and StandardBaseCheck in how they handle inheritance:

class SubNewedBaseCheck(NewedBaseCheck):
    def __init__(self):
        self.x = 9

subnewed = SubNewedBaseCheck()
subnewed.x == 9
subnewed._from_base_class is False

class SubStandardBaseCheck(StandardBaseCheck):
    def __init__(self):
        self.x = 9

substandard_base_check = SubStandardBaseCheck()
substandard_base_check.x == 9
hasattr(substandard_base_check,"_from_base_class") is False

Because we failed to call super(...).__init__ in the constructors, _from_base_class is never set.

__new__ and __init__

Up until now, classes defining both __init__ and __new__ had no-argument constructors. Adding arguments has a few pitfalls to watch out for. We’ll modify NewBaseCheck:

class NewedBaseCheck(object):
    def __new__(cls):
        obj = super(NewedBaseCheck,cls).__new__(cls)
        obj._from_base_class = type(obj) == NewedBaseCheck
        return obj

    def __init__(self, x):
        self.x = x

try:
    NewedBaseCheck(5)
except TypeError:
    print True

Instantiating a new NewedBaseCheck throw a TypeError. NewedBaseCheck(5) first calls NewBaseCheck.__new__(NewBaseCheck, 5). Since __new__ takes only one argument, Python complains. Let’s fix this:

class NewedBaseCheck(object):
    def __new__(cls, x):
        obj = super(NewedBaseCheck,cls).__new__(cls)
        obj._from_base_class = type(obj) == NewedBaseCheck
        return obj

    def __init__(self, x):
        self.x = x

newed = NewedBaseCheck(5)
newed.x == 5

There are still problems with subclassing:

class SubNewedBaseCheck(NewedBaseCheck):
    def __init__(self, x, y):
        self.x = x
        self.y = y

try:
    SubNewedBaseCheck(5,6)
except TypeError:
    print True

We get the same TypeError as above; __new__ takes cls and x, and we’re trying to pass in cls, x, and y. The generic fix is fairly simple:

class NewedBaseCheck(object):
    def __new__(cls, *args, **kwargs):
        obj = super(NewedBaseCheck,cls).__new__(cls)
        obj._from_base_class = type(obj) == NewedBaseCheck
        return obj

    def __init__(self, x):
        self.x = x

newed = NewedBaseCheck(5)
newed.x == 5

subnewed = SubNewedBaseCheck(5,6)
subnewed.x == 5
subnewed.y == 6

Unless you have a good reason otherwise, always define __new__ with *args and **kwargs.

The Real Power of __new__

__new__ is incredibly powerful (and dangerous) because you manually return an object. There are no limitations to the type of object you return.

class GimmeFive(object):
    def __new__(cls, *args, **kwargs)):
        return 5

GimmeFive() == 5

If __new__ doesn’t return an instance of the class it’s bound to (e.g. GimmeFive), it skips the Constructor Step entirely:

class GimmeFive(object):
    def __new__(cls, *args, **kwargs):
        return 5

    def __init__(self,x):
        self.x = x

five = GimmeFive()
five == 5
isinstance(five,int) is True
hasattr(five, "x") is False

That makes sense: __init__ will throw an error if passed anything but an instance of GimmeFive, or a subclass, for self. Knowing all this, we can easily define Python’s object creation process:

def instantiate(cls, *args, **kwargs):
    obj = cls.__new__(cls, *args, **kwargs)
    if isinstance(obj,cls):
        cls.__init__(obj, *args, **kwargs)
    return obj

instantiate(GimmeFive) == 5
newed = instantiate(NewedBaseCheck, 5)
type(newed) == NewedBaseCheck
newed.x == 5

Don’t Do This. Ever.

While experimenting for this post I created a monster that, like Dr. Frankenstein, I will share with the world. It is a great example of how horrifically __new__ can be abused. (Seriously, don’t ever do this.)

class A(object):
    def __new__(cls):
        return super(A,cls).__new__(B)
    def __init__(self):
        self.name = "A"

class B(object):
    def __new__(cls):
        return super(B,cls).__new__(A)
    def __init__(self):
        self.name = "B"

a = A()
b = B()
type(a) == B
type(b) == A
hasattr(a,"name") == False
hasattr(b,"name") == False

The point of the above code snippet: please use __new__ responsibly; everyone you code with will thank you.

__new__ and the new step, in the right hands and for the right task, are powerful tools. Conceptually, they neatly tie together object creation. Practically, they are a blessing when you need them. They also have a dark side. Use them wisely.

Conclusion

There are many possibilities on how you can use this feature.  Mostly it is not always required to override __new__ method unless you are doing something regarding instance creation.

Simplicity is better than complexity. Try to make life easier use this method only if it is necessary to use.

__new__ is one of the most easily abused features in Python. It’s obscure, riddled with pitfalls, and almost every use case I’ve found for it has been better served by another of Python’s many tools. However, when you do need __new__, it’s incredibly powerful and invaluable to understand.

The predominant use case for __new__ is in metaclasses. Metaclasses are complex enough to merit their own article, so I don’t touch on them here. If you already understand metaclasses, great. If not, don’t worry; understanding how Python creates objects is valuable regardless.

References:

https://www.concentricsky.com/articles/detail/pythons-hidden-new

https://howto.lintel.in/python-new-magic-method-explained/