05 February 2024

Python sucks (sometimes)

In summary

• Python can do a lot of things well, but sometimes, it really is the wrong tool for the job
• When using Python to build a CLI, Louder encountered issues with installation and dependency management
• During the migration to a compiled language, Rust alleviated a lot of issues when creating our CLI

A bit about Python

“Heresy!”, I hear the Python fanatics cry upon reading the title of this article. But clickbait aside, we at Louder love Python, and use it for just about every software product we create. It’s an amazing tool to quickly prototype and iterate on designs with, and you’ll get something functional and mostly production ready at the end. However, as useful as Python often is, it can be equally frustrating.

The monorepo

For the uninitiated, a monorepo is a way of organising code so that it all remains in one place. This comes with it’s own pros and cons, but we’ll discuss this another time. The main reason for doing so was our version controlling was getting out of hand with so many projects. Additionally, we had started developing libraries that were getting shared across different projects. At the time, the solution to accessing these different libraries was to copy them into each project that used them. This created different versions and projects that had access to different features of different libraries. It was a mess.

However, migrating to a monorepo comes with its difficulties, and in our case it was the language of choice - Python. The primary issue we had was with importing libraries within our monorepo. Python doesn’t let you import relative files above the scope of the file being executed. Any files within the same folder or lower in the file hierarchy are free game though. We decided to mitigate this using Poetry.

Poetry

Poetry allows for better dependency management than Python’s default package manager, Pip, verifying version conflicts, as well as letting you specify whether a package is local, in a private repository, or publicly available in Pip, per dependency. While this tackled a few of the issues we were facing, it didn’t quite tackle everything.

Poetry required a lot of configuration out of the box to get it where we needed it, and even with that configuration, it didn’t resolve all of our issues. The majority of our projects end up deployed in the Google Cloud Platform (GCP), and our libraries get hosted in a private Pypi repository in Artifact Registry. Configuring Poetry per project became a lot of repetitive shell commands, on top of all of the commands required to deploy to GCP. Our interim solution here was to use GNU Make to build out rules that run all of the commands. Unfortunately, this means copying a LOT of Make code to each project.

A CLI?

With the above problems defined, it seemed like a no-brainer to write an internal Command Line Interface (CLI) to manage our projects and libraries. To do this, we used a Python library - Typer. Louder had previously adopted FastAPI by the same author, tiangolo, due to its tidy integration with Pydantic, and the wonderful developer experience. I proposed Typer due to similar benefits over Click. Having written the CLI, all seemed well. We had a nice and easy set of commands to run to publish libraries, deploy projects, and maintain the monorepo and project dependencies. Well, that was until one of the other developers tried to install it…

Welcome to dependency hell

As it turns out, Python has different behaviours with imports depending on a few different factors, namely platform, and runtime. We found code that would run as expected on MacOS, but would throw import errors on Linux. Moving from relative imports to full path imports seemed to fix some of these issues, but there were instances of the CLI failing to look up it’s own source code.

So, what are the issues?

• Installing the CLI in a virtualenv means you have to reinstall it for each project you work on
• Installing into a virtualenv only makes a reference to the source code with a relative path. This can fail when the code is looked up at runtime
• Installing the CLI globally can interfere with other global installs, such as gcloud, aws, or other Python written CLI tools
  • This can leave you with different versions of the same dependency, and break other tools. This could be avoided using a virtualenv, but see above

What now?

With all the issues of runtime imports and other shenanigans, perhaps it was time we changed our tooling up. We needed something that works cross platform, is compiled (to get rid of those runtime import issues), and has an equivalent or better developer experience.

Rust was our primary contender for an alternate language, so let’s weigh it up against Python!

Python vs Rust comparison

Having discussed the above pros and cons among the team, we decided to rewrite our CLI in Rust!

Outcomes

Our install script is now just a single simple command, cargo install --path tools/cli. It is also significantly more consistent to install, as we have not experienced any issues with this. It has also enabled a better error handling flow using enums, pattern matching, and the Result type.

Should I just replace Python with Rust?

Yes, and no. Just as Python can do pretty much anything, Rust can too, but it’s not always going to be the best tool for the job. I love using Rust in my personal projects, and will use it just about anywhere I can. That doesn’t mean that in a production environment I would always choose it. As discussed above, both languages have their pros and cons, and other languages may have a monopoly over a particular environment, cough cough, JavaScript… So weigh it up for each project and decide for yourself.

Next steps

If you need help with your tech stack or software and data projects, get in touch to discuss how we can help. You can also sign up to our newsletter to receive the latest industry updates in your inbox.

About Sam Kenney

---

Sam is a data engineer. In his spare time he plays guitar for the UK-based alternative band, Worst Case Scenario.