Weeknotes: Squish all the bugs

Client Work

In looking back over the PRs I submitted this week for my main client, who has a microservice architecture, I see that I was fixing a lot of small, annoying bugs.

These bugs were introduced as part of a rewrite I’m working on to improve the performance of a few API endpoints in one of the microservices. These endpoints depend on getting a lot of data from another microservice and we made some other changes recently that had the unexpected side effect of slowing these endpoints down quite a bit. I wound up making some pretty significant changes to how these endpoints get their data that involved also making some changes to the endpoints in the microservice it was calling, so it’s not surprising that there were some bugs to work out.

What was surprising was how much I appreciated having a user in the dev environment that had lots of weird data in it.

I have a useful habit: Every time a bug pops up in production or staging that’s due to some unexpected or weird data in someone’s account, I create a record attached to my dev user that shares those qualities. You never know when something like a field being null and not 0 will trigger a bug in the system. It’s hard to account for everything, and it’s easy to make assumptions about how you think a service you’re calling will return null data to you. What this meant was that I caught a lot of bugs before they even hit the staging environment! It also meant I spent the week playing whack-a-mole with bugs. Here are some of the bugs I fixed.

Arrays in query parameters

These were some bugs that I have encountered before, but I always forget.

When you’re calling an API and passing an array as one of the query parameters, you need to stringify it.

$ ids = [123, 456, 789]
$ ids_query_param = “,”.join([str(id) for id in ids])
$ print(ids_query_param)
“123,456,789”

This means you don’t wind up accidentally constructing an API call that looks like this:

localhost:8000/api/my-stuff/?ids=[123, 456, 789]

Which won’t work. Instead, your call will look like this:

localhost:8000/api/my-stuff/?ids=123,456,789

# or 

localhost:8000/api/my-stuff/?ids=123%2C456%2C789

Depending on whether you’re endcoding the commas in your URLs.

I also remembered that retrieving an array query parameter in your view isn’t as simple as request.query_params.get(). That will get you only the first item in your array. To retrieve an array query parameter, you must use .getlist.

ids = request.query_params.getlist(“ids”)

Null values where you expect zeros

The microservice I was working in was making a call to another microservice that returned a numeric total. It did this in this format:

{“total”: {“amount”: 100}}

There were some other fields in the total dictionary, but the one I cared about was amount. I knew that the service I was calling was getting this total by actually performing an aggregate query using Sum, so I assumed that if the fields it was totaling were null, it would return {“total”: {“amount”: 0}}.

This was a bad assumption.

I found this out when this code resulted in an AttributeError:

return response.json()[“total”].get(“amount”, 0)

I thought I was being so clever, including a default value for amount! Alas, the AttributeError was due to the response actually looking like this:

{“total”: null}

So I changed my code to this instead:

return response.json()[“total”].get(“amount”, 0) if response.json()[“total”] else 0

It’s maybe not the most elegant solution, but there you go.

Why is my custom DRF serializer context not in my serializer?

I am giving a talk tomorrow about Django REST Framework that includes a section on customizing your serializer context and I still introduced this bug this week. (This code is paraphrased.)

# views.py 
context = self.get_serializer_context()
context[“my_field”] = value 

serializer = self.get_serializer(data, context=context)
return serializer.data

# serializers.py 
my_field = self.context[“my_field”] # Error!!

This is because get_serializer() already calls get_serializer_context() and passes it into the serializer. Me passing another kwarg called context didn’t override this behavior.

The better practice is to call get_serializer_class instead.

# views.py 
context = self.get_serializer_context()
context[“my_field”] = value 

serializer = self.get_serializer_class()(data, context=context)
return serializer.data

# serializers.py 
my_field = self.context[“my_field”] # No error!

PyCascades

I’m presenting tomorrow at PyCascades! My talk, What You Should Know About Django REST Framework, is tomorrow at 11:05 AM. Next week, I’ll be posting a couple blog posts that summarize the talk. I’ll also make my slides available once the talk is over. I recorded the talk a couple of weeks ago and it’s my first virtual conference talk (and first conference talk in some time, and first talk on DRF), so I’m excited!

TIL

I posted a new TIL to that repo, Using Coalesce to provide a default value for aggregate queries. It doesn’t have a ton of data in it that isn’t in the Django docs, but the Coalesce function was entirely new to me and I wanted to make sure I remember it!

This website

I redesigned my website this week. Simplified it, made the home page more of a “resume” that includes my articles and talks, and removed those as individual pages. It just felt like it made more sense.

I still like Squarespace for my main site, but I do think I would like to move the blog to something like Jekyll or Datasette with GitHub actions. I am finding it so much easier to push a markdown file to my TIL repo than to create a post on Squarespace. In fact, Squarespace froze in a way that made me lose all my work on this blog post, so I’m actually writing this in Apple Notes so I can copy and paste it.

GitHub Profile

I finally added a README to my GitHub profile! I feel like I am the last one to do this, but it was time.