Complex auth - micro-services & test isolation

I wrote about how we reworked our internal authorization code in types & tests and the biggest issue: cache invalidation.

And the story continues because these two topics are closely related to each other. For integration tests, it's necessary to have them isolated. Why? Because your tests may:

• fail due to an uncertain state from the previous test
• succeed even if they should not - and this is a way bigger problem

Let's imagine the situation. You want to test your micro-services in a production-like environment with all its parts - PostgreSQL, Memcached, Minio (S3-compatible storage), ActiveMQ (what Amazon MQ is built on top), etc.

You want to be sure that the API request stores data, queues messages, and that the messages are correctly handled. You want to be sure that the whole process, all its parts, is working as desired.

For tests, you can generate randomized data to prevent collisions. However, it becomes complicated to validate the results. But it's great as you can run tests in parallel; they do not affect each other.

They can still, to some extent, interfere by overflowing the message broker, etc. And not only that, imagine that you reuse the testing set for just a slightly different case and forget to randomize data (or there is an internal state you can't affect). This can easily happen for very complex situations with many relations where randomizing data makes the test extremely complicated. The first test populates the cache (remember, we are doing an integration test on a production-like setup). The second test may not even run the critical code because the required data is taken from the cache. Oops.

So, for the integration tests for Localazy, I have a helper that is run before the test to:

• truncate all tables in the database
• wait for all unprocessed messages in the message broker
• invalidate all caches
• removes all files from the storage

With this, I can be sure that each of the tests runs on a completely clean setup and is not affected by other tests.

And even better, I can maintain the whole stack independently and don't need it to be recreated for tests. Actually, I have the stack launched on a different computer. It's always ready, so I can, without any consequences, run one or all tests anytime I want without waiting for the testing environment to be created. And those saved seconds can quickly sum up.

Btw, I also have a helper for ActiveMQ that stops the processing and waits for the queue to be empty. This helps me synchronize integration tests across several independent micro-services to be sure that the action has the full and complete expected reaction.

There is an issue about this: You can't run tests in parallel. But I can live with it as I don't need to run all tests during the development. I run test and go rest.