Python __new__ magic method explained

Python is an object-oriented programming language, meaning that everything in Python is an object. One of Python’s unique features is the use of “magic methods,” which are special methods that have a double underscore prefix and suffix.

When discussing the magic method __new__, it’s important to also mention __init__. Both methods are called when an object is instantiated, or created from a class. The __new__ method is called first during instantiation and is responsible for creating the object instance. This method allows for customization of the instance creation process. Once the instance is created, the __init__ method is called to initialize the instance.

The __new__ method takes the class reference as its first argument, followed by any arguments passed to the class constructor. This method is responsible for creating the instance, so it can be used to customize the creation process. Typically, the __new__ method returns a reference to the created instance object. Once the __new__ method has completed execution, the __init__ method is called to perform any additional initialization of the instance.

You can create a new instance of the class by invoking the superclass’s __new__ method using super. Something like super(currentclass, cls).__new__(cls, [,….])

Usual class declaration and instantiation:

class Foo(object):
    def __init__(self, a, b):
        self.a = a
        self.b = b

    def bar(self):
        pass
    
i = Foo(2, 3)

A class implementation with __new__ method overridden:

class Foo(object):
    def __new__(cls, *args, **kwargs):
        print ("Creating Instance")
        instance = super(Foo, cls).__new__(cls, *args, **kwargs)
        return instance

    def __init__(self, a, b):
        self.a = a
        self.b = b

    def bar(self):
        pass

Output:

i = Foo(2, 3)
# Creating Instance

Note:

You can create instance inside __new__  method either by using super function or by directly calling __new__ method over object  Where if parent class is object. That is,

instance = super(MyClass, cls).__new__(cls, *args, **kwargs)

or

instance = object.__new__(cls, *args, **kwargs)

Things to remember

If __new__ returns an instance of its own class, then the __init__ method of the newly created instance will be invoked with the instance as first (like __init__(self, [, ….]) argument following by arguments passed to __new__ or call of class.  So, __init__ will be called implicitly.

If __new__ method returns something else other than the instance of the class, then instances __init__ method will not be invoked. In this case, you have to call __init__ method yourself.

Applications

Usually, it’s uncommon to override __new__ method, but sometimes it is required if you are writing APIs or customizing class or instance creation, or abstracting something using classes.

SINGLETON USING __NEW__

You can implement the singleton design pattern using __new__ method. Where singleton class is a class that can only have one object. That is an instance of the class.

Here is how you can restrict creating more than one instance by overriding __new__Singleton implementation using __new__:

class Singleton(object):
    _instance = None  # Keep instance reference 
    
    def __new__(cls, *args, **kwargs):
        if not cls._instance:
            cls._instance = object.__new__(cls, *args, **kwargs)
        return cls._instance

It is not limited to the singleton. You can also impose a limit on the total number of created instances limiting the total number of instances that can be created using __new__:

class LimitedInstances(object):
    _instances = []  # Keep track of instance reference
    limit = 5 

    def __new__(cls, *args, **kwargs):
        if not len(cls._instances) <= cls.limit:
            raise RuntimeError, "Count not create instance. Limit %s reached" % cls.limit    
        instance = object.__new__(cls, *args, **kwargs)
        cls._instances.append(instance)
        return instance
    
    def __del__(self):
        # Remove instance from _instances 
        self._instance.remove(self)

Customize Instance Object

You can customize the instance created and make some operations over it before initializer __init__ is called. Also, you can impose restrictions on instance creation based on some constraints:

def createInstance():
    # Do what ever you want to determie if instance can be created
    return True 

class CustomizeInstance(object):

    def __new__(cls, a, b):
        if not createInstance():
            raise RuntimeError, "Count not create instance"
        instance = super(CustomizeInstance, cls).__new__(cls, a, b)
        instance.a = a
        return instance

    def __init__(self, a, b):
        pass

Customize Returned Object

Usually, when you instantiate a class it will return the instance of that class. You can customize this behavior and you can return some random object you want.

Following  one is a simple example to demonstrate that returning a random object other than the class instance:

class AbstractClass(object):

    def __new__(cls, a, b):
        instance = super(AbstractClass, cls).__new__(cls)
        instance.__init__(a, b)
        return 3

    def __init__(self, a, b):
        print ("Initializing Instance", a, b)

Output:

a = AbstractClass(2, 3)
# Initializing Instance 2 3
print(a) 
# 3

Here you can see when we instantiate class it returns  3 instead of instance reference. Because we are returning 3 instead of created instance from __new__ method. We are calling __init__ explicitly.  As I mentioned above, we have to call __init__ explicitly if we are not returning the instance object from __new__ method.

The __new__ method is also used in conjunction with metaclasses to customize class creation

Constructors

With the proliferation of class-based languages, constructors are likely the most popular method for instantiating objects.

Java

class StandardClass {
    private int x;
    public StandardClass() {
        this.x = 5;
    }

    public int getX() {
        return this.x;
    }
}

Python

class StandardClass(object):
    def __init__(self, x):
        self.x = x

Even JavaScript, a prototypical language, has object constructors via the new keyword.

function StandardClass(x) {
    this.x = x;
}

var standard = new StandardClass(5);
alert(standard.x == 5);

Newer is Better

In Python, as well as many other languages, there are two steps to object instantiation:

The New Step

Before you can access an object, it must first be created. This is not the constructor. In the above examples, we use this or self to reference an object in the constructor; the object had already been created by then. The New Step creates the object before it is passed to the constructor. This generally involves allocating space in memory and/or whatever language-specific actions newing up an object requires.

The Constructor Step

Here, the newed-up object is passed to the constructor. In Python, this is when __init__ is called.

Python Object Creation

This is the normal way to instantiate an StandardClass object:

standard = StandardClass(5)
standard.x == 5

StandardClass(5) is the normal instance creation syntax for Python. It performs the New Step followed by the Constructor Step for us. Python also allows us to deconstruct this process:

# New Step
newed_up_standard = object.__new__(StandardClass)
type(newed_up_standard) is StandardClass
hasattr(newed_up_standard,'x') is False

# Constructor Step
StandardClass.__init__(newed_up_standard, 5)
newed_up_standard.x == 5

object.__new__ is the default New Step for object instantiation. It’s what creates an instance from a class. This happens implicitly as the first part of StandardClass(5).

Notice x is not set until after newed_up_standard is run through __init__. This is because object.__new__ doesn’t call __init__. They are disparate functions. If we wanted to perform checks on newed_up_standard or manipulate it before the constructor is run, we could. However, explicitly calling the New Step followed by Constructor Step is neither clean nor scalable. Fortunately, there is an easy way.

Controlling New with __new__

Python allows us to override the New Step of any object via the __new__ magic method.

class NewedBaseCheck(object):
    def __new__(cls):
        obj = super(NewedBaseCheck,cls).__new__(cls)
        obj._from_base_class = type(obj) == NewedBaseCheck
        return obj
    def __init__(self):
        self.x = 5

newed = NewedBaseCheck()
newed.x == 5
newed._from_base_class is True

__new__ takes a class instead of an instance as the first argument. Since it creates an instance, that makes sense. super(NewedClass, cls).__new__(cls) is very important. We don’t want to call object.__new__ directly; again, you’ll see why later.

Why is from_base_class defined in __new__ instead of __init__? It’s metadata about object creation, which makes more semantic sense in __new__. However, if you really wanted to, you could place define _from_base_class:

class StandardBaseCheck(object):
    def __init__(self):
        self.x = 5
        self._from_base_class == type(self) == StandardBaseCheck

standard_base_check = StandardBaseCheck()
standard_base_check.x == 5
standard_base_check._from_base_class is True

There is a major behavioral difference between NewBaseCheck and StandardBaseCheck in how they handle inheritance:

class SubNewedBaseCheck(NewedBaseCheck):
    def __init__(self):
        self.x = 9

subnewed = SubNewedBaseCheck()
subnewed.x == 9
subnewed._from_base_class is False

class SubStandardBaseCheck(StandardBaseCheck):
    def __init__(self):
        self.x = 9

substandard_base_check = SubStandardBaseCheck()
substandard_base_check.x == 9
hasattr(substandard_base_check,"_from_base_class") is False

Because we failed to call super(...).__init__ in the constructors, _from_base_class is never set.

__new__ and __init__

Up until now, classes defining both __init__ and __new__ had no-argument constructors. Adding arguments has a few pitfalls to watch out for. We’ll modify NewBaseCheck:

class NewedBaseCheck(object):
    def __new__(cls):
        obj = super(NewedBaseCheck,cls).__new__(cls)
        obj._from_base_class = type(obj) == NewedBaseCheck
        return obj

    def __init__(self, x):
        self.x = x

try:
    NewedBaseCheck(5)
except TypeError:
    print True

Instantiating a new NewedBaseCheck throw a TypeError. NewedBaseCheck(5) first calls NewBaseCheck.__new__(NewBaseCheck, 5). Since __new__ takes only one argument, Python complains. Let’s fix this:

class NewedBaseCheck(object):
    def __new__(cls, x):
        obj = super(NewedBaseCheck,cls).__new__(cls)
        obj._from_base_class = type(obj) == NewedBaseCheck
        return obj

    def __init__(self, x):
        self.x = x

newed = NewedBaseCheck(5)
newed.x == 5

There are still problems with subclassing:

class SubNewedBaseCheck(NewedBaseCheck):
    def __init__(self, x, y):
        self.x = x
        self.y = y

try:
    SubNewedBaseCheck(5,6)
except TypeError:
    print True

We get the same TypeError as above; __new__ takes cls and x, and we’re trying to pass in cls, x, and y. The generic fix is fairly simple:

class NewedBaseCheck(object):
    def __new__(cls, *args, **kwargs):
        obj = super(NewedBaseCheck,cls).__new__(cls)
        obj._from_base_class = type(obj) == NewedBaseCheck
        return obj

    def __init__(self, x):
        self.x = x

newed = NewedBaseCheck(5)
newed.x == 5

subnewed = SubNewedBaseCheck(5,6)
subnewed.x == 5
subnewed.y == 6

Unless you have a good reason otherwise, always define __new__ with *args and **kwargs.

The Real Power of __new__

__new__ is incredibly powerful (and dangerous) because you manually return an object. There are no limitations to the type of object you return.

class GimmeFive(object):
    def __new__(cls, *args, **kwargs)):
        return 5

GimmeFive() == 5

If __new__ doesn’t return an instance of the class it’s bound to (e.g. GimmeFive), it skips the Constructor Step entirely:

class GimmeFive(object):
    def __new__(cls, *args, **kwargs):
        return 5

    def __init__(self,x):
        self.x = x

five = GimmeFive()
five == 5
isinstance(five,int) is True
hasattr(five, "x") is False

That makes sense: __init__ will throw an error if passed anything but an instance of GimmeFive, or a subclass, for self. Knowing all this, we can easily define Python’s object creation process:

def instantiate(cls, *args, **kwargs):
    obj = cls.__new__(cls, *args, **kwargs)
    if isinstance(obj,cls):
        cls.__init__(obj, *args, **kwargs)
    return obj

instantiate(GimmeFive) == 5
newed = instantiate(NewedBaseCheck, 5)
type(newed) == NewedBaseCheck
newed.x == 5

Don’t Do This. Ever.

While experimenting for this post I created a monster that, like Dr. Frankenstein, I will share with the world. It is a great example of how horrifically __new__ can be abused. (Seriously, don’t ever do this.)

class A(object):
    def __new__(cls):
        return super(A,cls).__new__(B)
    def __init__(self):
        self.name = "A"

class B(object):
    def __new__(cls):
        return super(B,cls).__new__(A)
    def __init__(self):
        self.name = "B"

a = A()
b = B()
type(a) == B
type(b) == A
hasattr(a,"name") == False
hasattr(b,"name") == False

The point of the above code snippet: please use __new__ responsibly; everyone you code with will thank you.

__new__ and the new step, in the right hands and for the right task, are powerful tools. Conceptually, they neatly tie together object creation. Practically, they are a blessing when you need them. They also have a dark side. Use them wisely.

Conclusion

There are many possibilities on how you can use this feature.  Mostly it is not always required to override __new__ method unless you are doing something regarding instance creation.

Simplicity is better than complexity. Try to make life easier use this method only if it is necessary to use.

__new__ is one of the most easily abused features in Python. It’s obscure, riddled with pitfalls, and almost every use case I’ve found for it has been better served by another of Python’s many tools. However, when you do need __new__, it’s incredibly powerful and invaluable to understand.

The predominant use case for __new__ is in metaclasses. Metaclasses are complex enough to merit their own article, so I don’t touch on them here. If you already understand metaclasses, great. If not, don’t worry; understanding how Python creates objects is valuable regardless.

References:

https://www.concentricsky.com/articles/detail/pythons-hidden-new

https://howto.lintel.in/python-new-magic-method-explained/