Avoiding Circular Imports in Python
If you’ve written enough Python code, you’ve almost certainly found yourself staring down a stack trace that that ends with:
In my experience, running into such an error usually prompts a long and tedious inspection of the codebase. The end result is almost always some sort of refactor–ranging from small adjustments of imports all the way to large changes in a project’s directory structures.
The reason why circular imports in Python result in errors has been explained by others much more eloquently than I could manage — instead of focusing on the reason behind the problem, in this post we will provide some practical advice that might help you avoid circular imports in large Python projects (or at least make them easier to debug!).
Not all Circular Imports are Created Equal
Some circular imports in Python are “real”, and cannot be resolved without moving where your objects are defined. An example of this can be seen in this repository. If you clone the repository locally, check out the circular-imports-require-refactor branch and try to run main.py , you’ll see the import error show up:
So, what is going on here? Well, this case is (deliberately) pretty simple:
- The Child class in the child file depends on the Person class in the parent module
- The Parent class in the parent file depends on the Child class in the child module
In this contrived example, the solution is really quite simple: move the Person class definition into a new file, and have child and parent pull from that. The notable thing here is that in order to resolve this circular import, we actually had to move a class definition into another file (the solution can be seen in another branch in the same repository).
Importing from Parent Modules
If you asked me a couple of weeks ago, I would have said that pretty much all circular imports would require such refactors (ie: changing where objects are defined) — of course, had I been correct, this would be a much less interesting blog post. In fact, as our Python libraries at Brex grow and become more complex, we started to notice that many circular imports had a very different root cause: they were happening because we were importing objects from the wrong place, not because we were defining objects in the wrong place. Rather than me fumbling for a description, let’s consider an example. Take the following directory structure and files (this is replicated in another branch of our repo):
This is exactly the same directory structure we had in the “fixed” branch for our imports. However, we have added some slight changes:
- We added imports for all of our classes in the objects/__init__.py file
- The imports in person.py and child.py were changed to import from objects , instead of importing from each submodule
In other words, we have:
The classes above are still defined in the same place as in the “fixed” branch, but the imports have been slightly modified (we now import from the module root, instead of importing from each specific file). So, what happens when we try to run main.py here?
Wait, what? Well, the problem here is that loading the __init__.py file requires loading the child.py file, which in turn requires loading the __init__.py file in order to import the Person class. This is pretty obvious when you think about it for a second, but the key insight here is that the “circular” import exists only because of how objects are imported! We can trivially fix this particular circular import by changing our imports to import from the module where the object is actually defined. You can see this fix in the following diff:
Does this actually happen?
Again, this example might seem a bit contrived… it would be pretty natural to ask at this point “does this actually happen in practice?” — to which I would answer: yes, most definitely!
We noticed this starting to become a pretty big problem at Brex with one of our larger Python libraries called “fractal” — this is a library built and maintained by our Data Platform team, and used mainly by our Data Science team. In order to allow our clients to easily import all the library’s objects, we adopted the convention of importing the public API into the top-level level __init__.py file of the project. In addition to being user-friendly, this practice ensures that our client’s imports won’t break if we refactor the internals of our library. So, why does this matter? Well, as more and more developers contributed to this library, we found that in many instances folks were importing from the root of the library from within the library itself. This is the exact same practice as shown in the example above, but in a real-world scenario where the less-than-ideal import strategy came about naturally, out of the ever-growing complexity of a large Python library.
The pattern we observed internally was that developers would import from the “root” module while writing their code (due to convenience — after all, everything is right there!), and that would work just fine for a while. However, when someone else came along and made subsequent code changes, they would end up hitting a circular import that could be resolved if we simply changed all the other imports to import directly from where classes are defined. This kind of issue significantly slowed down developers, as they had to unwind a very complicated dependency chain in order to fix circular imports. It would be much better for developer efficiency (and sanity) if we could simply make sure to import objects from the files they are defined in in the first place!
What can we do about this?
At this point, we have already pointed out a common cause of circular imports and how to fix them — we could very well leave it at that and simply establish the following principle:
“Always import objects from the files where they are defined.”
– Brex Engineering, 2022
While that is wise advice, at Brex we try harder than that — so we wanted to provide you with something more practical. Specifically, we implemented a function which can help you enforce the principle above in your Python code. We accomplish this by searching the code and detecting cases where classes are not being imported from the module they are defined in. In the interest of being practical, I won’t spend time walking through the implementation here (if you’d like to learn more about how it works, I recommend reading through it — it should be quite straight-forward to follow along). However, we can apply this to the example shown above. After checking out the test-imports-from-source branch of our repository, we can run:
You should see the following error in the pytest output:
And there we go! Switching out the problematic import to import directly from objects.child instead will make this error go away.
How can I use this?
The code linked to above provides a function that will automatically scan your project for imports that violate the “import from definition” principle. If you’d like to try this out in your own project, feel free to add that bit of code to it and see how many instances of “indirect imports” you have to deal with. In our case, we had quite a few imports that failed this test, so fixing them up involved a fairly significant refactor. The good news, however, is that the refactor was quite safe, since we really were limited to only changing the import locations.
In our case over at Brex, once we actually did this refactor, not only did we avoid circular dependencies that are due to “indirect imports”, but the “real” circular dependencies also became quite a bit easier to debug. This was due to the significantly simplified import chain, now that we were forcing ourselves to use only direct imports. As a way to visualize this simplification, I used pydeps to generate a before/after graph of how our Python files depend on each other. Here is the after graph (ie: after we simplified our imports):
Yes, this is the after graph! I attempted to generate a before graph to show that this is actually a lot simpler than our previous situation… but about 2h of waiting, pydeps crashed and refused to produce anything usable. I guess that alone should tell you something about the previous state, and how using direct imports improved our codebase’s sanity!
Python Circular Import
In this article, we will learn how circular imports commonly arise and the main ways to fix them. We will also see how to fix a circular import error caused by a type hint in python.
Reproduce Circular Import Error in Python
Have you ever come across an error when trying to import a module? In this case, we get an error that is shown below.
It tells us that the most likely cause is a circular import. Here is the simplest and most common way that this kind of import cycle can happen at runtime main tried to import something from My_Module_A .
So, it starts initializing My_Module_A and running the code from My_Module_A , but the first line in My_Module_A is to import something from module MY_Module_B .
So, it stops initializing module My_Module_A because MY_Module_B has to be initialized first. And then, it hops over to MY_Module_B and starts running that code instead.
But, the first line in module MY_Module_B is needed something from My_Module_A , so it stops running MY_Module_B and goes over to My_Module_A .
And it would like to start initializing My_Module_A , but it realizes that My_Module_A is already in the initialization process. So that’s where the error occurs.
We can see this chain of events unfold in the traceback. Here we see in main.py ; it tries to import function My_FUNC_A from module My_Module_A , which triggered the import of function MY_Func_B2 from module MY_Module_B .
And, MY_Module_B.py triggered the import of function My_FUNC_A again from module My_Module_A , where the final error occurred.
We have an import time cycle between the modules My_Module_A and MY_Module_B , but if you look at the names we imported, we do not use them at import time.
Please enable JavaScript
We use them when this function My_FUNC_A() or when this function MY_Func_B2() is called. Since we are not calling these functions at import time, there is no true cyclic dependency, which means we can eliminate this import cycle.
Solve Circular Import Error in Python
There are three common ways to resolve this issue; the first is to take your import and put it inside the function.
In python, imports can happen at any time, including when a function is called; if you need this function MY_Func_B1() , then it could be the import inside the My_FUNC_A() function.
It could be even more efficient because if you never call the function My_FUNC_A() , then it may be that you never even have to import MY_Func_B1 at all. For example, let’s say you have a bunch of different functions in My_Module_A , and many different ones use this function MY_Func_B1() .
In that case, the better solution would be to import the module MY_Module_B directly and in the functions where you use it, use the longer name like MY_Module_B.MY_Func_B1() . So go ahead and do this conversion for all modules involved in the cycle.
Let’s see how this resolves the import cycle at runtime error. First, our My_main_Func() function tries to import something from My_Module_A , causing My_Module_A to start running. Then, in My_Module_A.py , we get to import module MY_Module_B , which triggers MY_Module_B to start running.
In MY_Module_B.py , when we get to the import module My_Module_A , the module has already started initializing; this module object technically exists. Since it exists, it does not begin rerunning this.
It just says, okay, that exists, so the MY_Module_B module will continue, finish its import, and then after that import is done, the My_Module_A module will finish importing.
The third and final common way to eliminate this import cycle is to merge the two modules. So, again, no imports, no import cycles; however, if you had two or maybe even more modules in the first place, you probably had them for a reason.
We are not just going to recommend that you take all of your modules and merge them into one; that would be silly. So, you should probably prefer using one of the first two methods.
Source code of My_Module_A.py file.
Source code of MY_Module_B.py file.
Source code of main.py file.
Solve Circular Import Error Caused by Type Hint in Python
Let’s look at another example, the following most common kind of import cycle you will run due to using type hints. This example is unlike the previous one because type annotations are defined on a function or class definition.
When the function is defined, or the class is defined but not defined when we run it, the most common use case of type hinting is not even at runtime. If all we care about is static analysis, then we do not even need to do the import at runtime.
The typing module provides this variable, TYPE_CHECKING is a false constant at runtime. So, if we put all the imports we need for type checking in one of these, then none will happen at runtime.
Avoiding the import loop altogether, if you import this class Mod_A_class , you will get a name error because this name Mod_A_class has not been imported.
To fix this, add this from __future__ import annotations . What does this?
It changes the way annotations work; instead of evaluating Mod_b_class as a name, all annotations are converted to strings.
So now you do not get an error at runtime here because even though the name Mod_b_class is not imported, the string Mod_b_class certainly exists. However, you should be aware that whenever you do one of these from __future__ imports, it is not necessarily guaranteed behavior forever.
Now, this causes your annotations to be handled as strings, but there is a competing proposal to make them just be evaluated lazily. Hence, they are not evaluated unless you try to look at them and inspect your type hints at runtime.
This probably will not make any difference for your code base, but you should still be aware. An alternative solution is to again use import modules instead of from module imports , and you will still need the annotations from __future__ .
Complete source code of module My_Module_A.py .
Complete source code of module MY_Module_B.py .
Complete source code of main.py file.
Hello! I am Salman Bin Mehmood(Baum), a software developer and I help organizations, address complex problems. My expertise lies within back-end, data science and machine learning. I am a lifelong learner, currently working on metaverse, and enrolled in a course building an AI application with python. I love solving problems and developing bug-free software for people. I write content related to python and hot Technologies.
Циклический импорт в Python
Циклическая зависимость в Python возникает, когда два или более модуля зависят друг от друга. Это связано с тем, что каждый модуль определяется в терминах другого.
Приведенный выше код демонстрирует довольно очевидную циклическую зависимость. functionA() вызывает functionB(), следовательно, в зависимости от него, а functionB() вызывает functionA(). У этого типа циклической зависимости есть некоторые очевидные проблемы, которые мы опишем немного дальше в следующем разделе.
Проблемы с круговыми зависимостями
Циклические зависимости могут вызвать множество проблем в вашем коде. Например, это может привести к тесной связи между модулями и, как следствие, к снижению возможности повторного использования кода. Этот факт также усложняет сопровождение кода в долгосрочной перспективе.
Кроме того, циклические зависимости могут быть источником потенциальных сбоев, таких как бесконечные рекурсии, утечки памяти и каскадные эффекты. Если вы не будете осторожны и у вас есть циклическая зависимость в вашем коде, может быть очень сложно отладить множество потенциальных проблем, которые он вызывает.
Что такое циклический импорт в Python?
Циклический импорт в Python – это форма циклической зависимости, которая создается с помощью оператора импорта в Python.
Например, давайте проанализируем следующий код:
Когда Python импортирует модуль, он проверяет реестр модулей, чтобы убедиться, что он уже импортирован. Если модуль уже был зарегистрирован, Python использует этот существующий объект из кеша. Реестр модулей – это таблица модулей, которые были инициализированы и проиндексированы по имени модуля. К этой таблице можно получить доступ через sys.modules.
Если он не был зарегистрирован, Python находит модуль, при необходимости инициализирует его и выполняет в пространстве имен нового модуля.
В нашем примере, когда Python достигает import module2, он загружает и выполняет его. Однако module2 также вызывает module1, который, в свою очередь, определяет function1().
Проблема возникает, когда function2() пытается вызвать функцию module1 function3(). Поскольку модуль1 был загружен первым и, в свою очередь, загружен модуль2 до того, как он смог достичь function3(), эта функция еще не определена и выдает ошибку при вызове:
Как исправить?
Как правило, циклический импорт – это результат плохого дизайна. Более глубокий анализ программы мог бы сделать вывод, что зависимость на самом деле не требуется или что зависимые функции могут быть перемещены в другие модули, которые не будут содержать циклическую ссылку.
Простое решение состоит в том, что иногда оба модуля можно просто объединить в один более крупный. Результирующий код из нашего примера выше будет выглядеть примерно так:
Однако объединенный модуль может иметь некоторые несвязанные функции (тесная связь) и может стать очень большим, если в двух модулях уже есть много кода.
Так что, если это не сработает, можно было бы отложить импорт module2, чтобы импортировать его только тогда, когда это необходимо. Это можно сделать, поместив импорт module2 в определение function1():
В этом случае Python сможет загрузить все функции в module1, а затем загрузить module2 только при необходимости.
Этот подход не противоречит синтаксису, поскольку в документации сказано: «Обычно, но не обязательно, помещать все операторы импорта в начало модуля».
В документации также говорится, что рекомендуется использовать import X вместо других операторов, таких как from module import * или from module import a, b, c.
Вы также можете увидеть много кодовых баз, использующих отложенный импорт, даже если нет циклической зависимости, которая ускоряет время запуска, поэтому это вообще не считается плохой практикой (хотя это может быть плохой дизайн, в зависимости от вашего проекта) .
Заключение
Циклический импорт – это особый случай циклических ссылок. Как правило, их можно решить с помощью улучшенного дизайна кода. Однако иногда результирующий дизайн может содержать большой объем кода или смешивать несвязанные функции (жесткая связь).
Python Error: “most likely due to a circular import” (Solution)
In this article we will discuss the Circular Import Error that can occur when importing modules in Python, and how to solve it.
What is a Circular Import?
Circular imports in Python can be a tricky and confusing issue to deal with. A circular import occurs when two or more modules import each other, creating a looping cycle of imports that can lead to import errors. The below diagram illustrates what this looks like.
In this scenario, the Python interpreter will first encounter module A, which imports module B. However, since module B also imports module A, the interpreter will then try to import module A again. This creates a recursive loop where the interpreter keeps importing module A and module B indefinitely, without being able to complete the import process.
In this article, we will explore the various causes of circular imports and how to resolve them.
Circular Import Error
There are three major reasons for why this error can occur. The root cause for all these are the same, but the way in which they occur can vary a little. We will teach you how to identify these patterns, so that you can quickly resolve any circular import errors in your code.
Problem #1
We will start with the easiest problem to identify and solve. If you have given your Python file, the exact same name as the library you are importing, the Python interpreter will end up getting confused between the library which you actually mean to import, and your Python file (since they both have the sane name).
To fix this, all you need to do is change the name of your Python file to something else.
Here is an example you try out with the random library, where we also named our Python file “random.py”.
This code will throw the following error.
Changing the file name to something like random_code.py will make the code work.
Direct Imports
Direct imports is the problem that we described right in the start. Let’s take a look at some actual code to better understand how this problem can occur.
Here we have code snippets, one for each Python file we have created. The first one (called moduleA.py) defines a function called funcA . The second one (called moduleB.py) defines a function called funcB .
Running either of the above Python files will give us the error we got in the previous section.
The solution would be to put both functions in the same file, because their definitions depend on each other.
So something like this.
No need for imports here.
It may sound a bit odd, but there really is no other way. You need to be smart about how you write your code. Pre-planning and drawing of an import graph (like the one in the start of this article) can help you identify whether you have any “cycles”.
Indirect Imports
Indirect imports are actually the same thing as direct imports. The only difference is that the chain of imports is longer. Instead of just two modules importing each other “directly” we have a chain of modules where one imports another, which imports another, and so on, until it eventually leads back to the first module. In other words, more than two modules are involved.
This can make it difficult to identify the source of the circular import, as it is not immediately obvious which modules are involved in the cycle.
Regardless, the solution here is the as the one we discussed for direct import, which would be to refactor your code.
This marks the end of the tutorial on “most likely due to a circular import” error and how to solve it. Any suggestions or contributions for CodersLegacy are more than welcome. Questions regarding the tutorial content can be asked in the comments section below.