The Platformer #24: Regressions
Tags: performance, testing
Published: 2022-10-13 18:40:05

It’s the first week of the month, and not just any month. It’s October! And we all know what that means: the start of a new quarter — Q4 no less. Time to look back and forward.

A common theme for the next quarter for many platform teams is going to be performance monitoring and regression detection.

To explain why this matters, let’s look back to the biggest performance regression we witnessed in the last year: the introduction of collapsed-reply threads (CRT). [CRT](https://handbook.mattermost.com/company/about-mattermost/list-of-terms#collapsed-reply-threads-crt) is a shockingly complicated and large project, with many complicated aspects. Sadly, one unanticipated effect was a significant regression in performance. We should have caught this before shipping it to customers. And we could have — if only we had the right infrastructure and focus on this topic in place. While we had the infrastructure (our load testing tool), we didn’t properly leverage it.

[We try to learn from our mistakes](https://mattermost.com/blog/scaling-collapsed-reply-threads/), so after some heroic efforts to fix the performance issues, we now have renewed focus on this topic. On the server platform team, we have Alejandro García dedicated to this topic. And guess what? It works. Even yesterday we detected a performance regression using our tool and [reverted the change](https://github.com/mattermost/mattermost-server/pull/21233) before it impacted anybody. Not a CRT-scale regression, but a regression nonetheless.

Performance testing is one of the highest impact topics we can work on in platform teams, so we want to “spread the love” beyond just server, specifically to mobile, web, and ideally, desktop as well, leveraging the pipeline and tooling from QA. Therefore, this will be a shared topic among all four platform teams. Stay tuned.

##OKRy picks
So, what did Q3 bring for the platform teams, and what will Q4 look like? Let’s dive in.

On the mobile platform end, both in Q3 and Q4, the big effort remains to support the planned release of v2 by the end of the year. In addition, we managed to spend some time on platform topics as well. Specifically, this quarter we: 

- Built a logging framework;
- Are now using GraphQL for the initial load (on servers that have GraphQL enabled);
- Made good progress on integrating Sentry;
- Vastly expanded our end-to-end test coverage. 

In Q4, we will ship this Sentry integration and keep expanding our test coverage. In addition, we’ll be laying the groundwork to have the first Boards and Playbook functionality be integrated into the mobile app next year. As part of this, we’ll invest heavily in further smoothening the developer experience. The goal is to have a one-line command to set up your development environment end-to-end. This should remove a big barrier to entry to get started on mobile.

On the web platform end, we made solid progress on the multi-product architecture for front-end in Q3 and will continue this effort in Q4. In addition, we are now using GraphQL (for servers that have it enabled) in more places. A few big refactor and cleanup efforts are still in progress and will be continued in Q4, including the menu component and post components. In Q4, we will spec out our new front-end Suite API, implement the data manager, and avoid DoS-ing ourselves (avoid [the thundering herd problem](https://en.wikipedia.org/wiki/Thundering_herd_problem) in more places). In terms of focus on performance, we intend to improve channel and team switching performance by 20% and contribute to the platform-wide effort to detect performance regressions.

On the desktop platform end, we internationalized the desktop app! We also implemented the new download list UI. Both of these major features will ship as part of desktop 5.2, which will ship later this month. In addition, we now have support for native node modules, which will allow us to do more operating-specific cool stuff like native notifications, which we hope to leverage in Q4. In Q4, we intend to further invest in reliability by making it easier to run our E2E test pipelines and reduce the number of flaky tests. We also want to make technical strides by attempting to compile-in some parts of the webapp into the desktop as a proof of concept, starting with the top bar. Last but not least, we want to invest in supportability, another big platform topic, by giving customers a simple screen to “sanity check” their installation to make sure everything is set up right.

On the server platform side, we did a few huge lifts with big diffs in Q3. In addition to supporting the mobile and web teams with GraphQL work, we landed a few huge refactors reorganizing how code is structured as part of the multi-product architecture. [The biggest one](https://www.google.com/url?q=https://github.com/mattermost/mattermost-server/pull/20899&sa=D&source=docs&ust=1665685558213360&usg=AOvVaw1futp1bM2gUGkuhfiq97fJ) was merged yesterday, which resulted in some issues that [we’re still figuring out as we speak](https://community.mattermost.com/core/pl/yeswy9zco7npuqd8admdrwm9pe). 

However, refactoring is not all we did. We also significantly improved our error handling in many places (resulting in more massive PRs). This means we’re ready for our next big lift: supporting Playbooks in their move to the multi-product architecture as well in Q4. Other things on the docket for Q4: automatic continuous profiling, rollback support for database migrations, mmctl import/export improvements, tons of improvements to our load testing tool, and scope of the load tests. In addition, we want to look at effective ways we can use Go’s generics to improve our code base, and will hopefully introduce the Event Bus.

On the QA platform side, we made major strides on our testing pipelines. Not only supporting the current ones, but also creating new ones for our desktop and mobile apps. They are now operational but can still be optimized significantly in terms of run time, which we’ll work on in Q4. Further in Q3, we kept iterating on [our Rainforest tests](https://mattermost.com/blog/mattermost-rainforest-qa/), supported weekly releases, and worked the infrastructure to allow us to engage the QA community through writing test cases in the future. In Q4, our work on pipelines and infrastructure will continue. We also hope to continue growing the QA community again with five new people, publish best practices on how to write test cases, and continue to expand the scope and robustness of our end-to-end tests in Cypress and Rainforest.

And that is all for this week. Somehow, I’m not super how high on the dad joke humor today. So, to compensate, I’ll end this with a dad joke [I pulled from the Twitters](https://twitter.com/dadsaysjokes/status/1577392348417269760):

I was walking past a farm and a sign said: Duck, eggs!
I thought: That's an unnecessary comma — and then it hit me.

_Want to take a closer look at the inner workings of the Platform team? Join the [Community server](https://community.mattermost.com/login/) to be the first to read The Platformer every Friday._