Currently, Python is the most used programming language for different projects around the world. According to statistics, 44.1% of programmers choose Python coding language for application/web development. However, that does not mean that Python developers are exempt from creating messy and inefficient code that can cost you and your clients time and money This is where Python code optimization comes in.
What is Optimization?
Let’s start by defining code optimization, so that you get the basic idea and understand why it’s needed. Sometimes it’s not enough to create code that just executes the task. Large and inefficient code can slow down the application, lead to financial losses for the customer, or require more time for further improvements and debugging.
Python code optimization is a way to make your program perform any task more efficiently and quickly with fewer lines of code, less memory, or other resources involved, while producing the right results.
It’s crucial when it comes to processing a large number of operations or data while performing a task. Thus, replacing and optimizing some inefficient code blocks and features can work wonders:
- Speed up the performance of the application;
- Make code clean and readable;
- Simplify error tracking and debugging;
- Save a lot of computational power, and so on.
Top 6 Python Code Optimization Tips
Python developers need to be able to use code optimization techniques instead of basic programming to ensure applications run smoothly and quickly. Below we have listed 6 tips on how to optimize Python code to make it clean and efficient.
1. Apply the Peephole Optimization Technique
To better understand the Peephole optimization technique, let’s start with how the Python code is executed. Initially the code is written to a standard file, then you can run the command “python -m compileall <filename>”and get the same file in *.pyc format which is the result of the optimization.
<Peephole> is a code optimization technique in Python that is done at compile time to improve your code performance. With the Peephole optimization technique, code is optimized behind the scenes and is done either by pre-calculating constant expressions or by using membership tests. For example, you can write something like the number of seconds in a day as a = 60*60*24 to make the code more readable, but the language interpreter will immediately perform the calculation and use it instead of repeating the above statement over and over again and thereby reducing the software performance.
The result of the Peephole optimization technique is that Python pre-calculates constant expressions 60*60*24, replacing them with 86400. So even if you write 60*60*24 all the time, it won’t decrease performance.
Using the Peephole Optimization Technique
Using this technique, you can replace a section of the program or a segment of instruction without significant changes in output.
Applying the optimization technique, you can:
- Turn mutable constructs into immutable ones. It can be done by using one of the 3 tuples:
- <__code__.co_varnames> that stores local variables with parameters included.
- <__code__.co_names> that holds global literals.
- <__code__.co_consts> that references all the constants.
- Verify the membership of the element by treating the instructions as a constant cost operation irrespective of the size of the set.
- Turn both the set and list into Constants.
Pay attention to the fact that this transformation can only be performed by Python for literals. Thus, the optimization will not happen if any of the sets or lists used are not literals.
Let’s take a look at some examples:
- “Hello, world!” * 5 is a constant expression less than 4096 long. So it gets evaluated by the compiler as “Hello, world!” 5 consecutive times
- [1, 2] * 7 is a list (mutable object) so it’s not evaluated.
- (10, 20, 30) * 3 is a sequence of length 9 which is less than 256 (for tuples), so it’s stored as (10, 20, 30, 10, 20, 30, 10, 20, 30).
2. Intern Strings for Efficiency
String objects in Python are sequences of Unicode characters, so they are called “text” sequences in the documentation. When characters of different sizes are added to a string, its total size and weight increase, but not only by the size of the added character. Python also allocates extra information to store strings, which causes them to take up too much space. To increase efficiency, an optimization method called string interning is used.
The idea behind string interning is to cache certain strings in memory as they are created. This means that only one instance of a particular string is active at any given time, and no new memory is needed to refer to it.
String interning has a lot in common with shared objects. When a string is interned, it is treated as a shared object because an instance of that string object is globally shared by all programs running in a given Python session.
As the most common implementation of the Python programming language, CPython loads shared objects into memory every time a Python interactive session is initialized.
This is why string interning allows Python to run efficiently, both in terms of saving time and memory.
Python tends to store only those strings that are most likely to be reused, namely identifier strings:
- Attribute names;
- Variable names;
- Argument names;
- Function and class names;
- Dictionary keys.
Principles according to which a string should be interned:
- Only a string loaded at compile time as a constant string will be interned, and conversely, a string constructed at runtime will not be interned.
- A string will not be interned if it is the product of constant folding and is more than 20 characters long, because it is hardly an identifier.
- Python will only intern a string and create a hash for it if it declares a string with a name that includes only combinations of letters/numbers/black characters and begins with either a letter or an underscore character.
Thus, all strings that are read from a file or received through the network are not part of the about-interning. However, just offload such a task to the intern() function to start interning such strings and processing them.
3. Profile Your Code
By profiling your code, you can identify areas of improvement in your code for further optimization. There are two main ways to do this.
1. Use <timeit>.
Use stop-watch profiling with this module. <timeit> records the time needed for task execution by a certain code segment and measures the time elapsed in milliseconds.
Here’s how it’s calculated:
2. Use <cProfile>.
This is advanced profiling, which is part of the Python package since Python 2.5. There are several ways to connect it to the Python code:
- Wrap a function inside its run method and thus measure the performance;
- Run the whole script from the command line while activating cProfile as an argument, using Python’s “-m” option.
Knowing the key elements of the cProfile report, you can find bottlenecks in your code.
Here are the elements to consider:
- <ncalls> — the number of calls made;
- <tottime> — the aggregate time spent in the given function and which has the greatest value;
- <percall> — the quotient of <tottime> divided by <ncalls>;
- <cumtime> — another parameter of high importance for all the projects that represents cumulative time in executing functions, its subfunctions;
- <percall> — the quotient of <cumtime> divided by primitive calls;
<filename_lineno(function)> — a point of action in a program.
4. Use Generators and Keys For Sorting
This is a way to optimize memory by using such a great tool as generators. Their peculiarity is that they don’t return all items (iterators) at once, but can return only one at a time. It’s better to use keys and the default <sort()> method while sorting items in a list. Thus, for instance, you can sort the list and strings according to the index selected as part of the key argument.
What this might look like:
5. Don’t Forget About Built-in Operators and External Libraries
There are thousands of built-in operators and libraries available in Python. It’s better to use the built-ins wherever possible to make your code more efficient. It’s possible due to the fact that all the built-ins are pre-compiled and, thus, pretty fast.
The “C” equivalent of some Python libraries gives you the same features as the original library but with faster performance. So, try to use cPickle instead of Pickle, for example, to see the difference. The PyPy package and <Cython> are a way to optimize a static compiler to make the process even faster.
6. Avoid Using Globals
Globals can have non-obvious and hidden side effects resulting in Spaghetti code. What’s more, Python is slow at accessing external variables. Herewith, it’s better to avoid using them, or at least limit their usage. If they are a necessity, here are a few recommendations:
- Use the global keyword to declare an external variable;
- Make a local copy before using them inside loops.
It’s critical to create a robust and scalable application that performs tasks rapidly and smoothly. However, it’s impossible to develop such an application by using only basic coding techniques. That’s why you need to optimize the Python code. At the same time, using the optimization techniques described in the article, you can not only create a clean code, but also improve the app performance and save a lot of time and money.
Bogdan Ivanov is a CTO at DDI development. He is a professional with an advanced degree in Cybersecurity, and 7 years of experience in building a cybersecurity strategy for all the company’s projects. He has a deep understanding of network security, compliance, and operational security.