A Performance Story

When I’m in between pieces of work I love poking around in the performance stats of the microservices I’m familiar with.

I was looking at one of the API requests in New Relic and noticed that it was making multiple calls to an external service every time it was requested. This seemed a bit inefficient so I begun investigating.

We call the external service an average of 4.77 times. I wanted to see if we could make this better.

This API call generates a timeline that shows multiple users communicating with each other. Think something like a Facebook Messenger group chat. The problem was that for every “message” in this timeline we were making a request to the external service to get information about the user who created the message. This isn’t optimal for a few reasons:

• The number of calls to the external service will scale linearly with the number of messages. Meaning that users who have a high number of messages, let’s say 100, will have to wait for 100 external calls for their timeline to load.
• This doesn’t take into account multiple messages by one user. This means we do multiple API calls for the same person.

At this point, I started looking at the external service and found out that you can pass it multiple user IDs at the same time. This was really exciting because doing this would solve both of my problems.

I hypothesised that by giving the external service an array of deduplicated user IDs, I’d be able to bring the average calls to the external service down to 1 and decrease the latency of this API call.

To solve this, I grabbed all of the user IDs from the timeline items, deduped them and then made a single call to the external service. I tested this, made sure it worked, and then went back to New Relic to see if my hypothesis was correct.

It was!

I saw that the average request time had gone down from ~75ms to ~25ms. This is great because the time had dropped to about a third of what it was when I started.

My main takeaways from doing this are:

• Have some little things to investigate in between big chunks of work
• Be curious about why things are the way they are in your systems.
• Don’t be afraid to rely on third party tools like New Relic to identify improvements you can make.
• A small win is still a win!