Why should you avoid using an arbiter node when upgrading to Mongodb 5.0 or later?

Introduction

Since I started working with MongoDB Databases, we have faced a lot of projects in different clients, and we have seeing some cases where the client had chosen an arbiter node as member of the Replica Set to ensure an odd number of votes in case of election. The benefit of having and arbiter node is economical, basically. An arbiter instance requests a minimal amount of RAM, low usage of CPU, does not contain data (minimal amount of storage) and is not considered for licensing purposes.

Until the version 4.4 of MongoDB having an arbiter node in a Replica Set was a common practice, for example, the architecture PSA (Primary, Secondary and Arbiter) could be acceptable for non production environment or also PSSSA (Primary, Secondary, Secondary, Secondary and Arbiter) could be acceptable even in production environment, I saw both of them in these years.

The change

At MongoDB version 5.0 there was an important change: if the majority commit point get stuck the Replica Set will face a sensible performance impact, even if it is specified the write concert to one (w:1) and the number of voting members if enough to elect a primary. With the lastCommittedOpTime stuck, the performance will be affected because the storage engine will keep all writes that happen since the last majority commit point in disk, increasing the I/O and cache pressure. Another documented behavior is at the oplog which will be allowed to grow over the configured capped size.

At the OPS Manager metrics is visible the number of opcounters decreasing, the tickets available keep stable at 128, and queues rising. The primary is elected, the secondary is health but the application can face a great performance degradation.

To mitigate this performance issue, the procedure is reconfigure the Replica Set setting votes and priority to zero at the lagging (or down) node.

Conclusion

So if you are planning an upgrade of MongoDB Replica Set to version 5.0 (or later) keep in mind this behavior to avoid be caught by performance issues.

It can bypassed by reconfiguring the Replica Set in scheduled activities, such as patching, resync and so on or even may be an opportunity to rethinking the architecture avoiding using the arbiter node.

Reference: https://www.mongodb.com/docs/v5.0/tutorial/mitigate-psa-performance-issues/