Sidekiq and Web limits

Both Sidekiq and Web limits only limit the concurrency, meaning how much “work” they can do simultaneously.

As an example, a limit of 2 Sidekiq threads will only allow at any given time a maximum of 2 Sidekiq jobs to be running.

The same goes for the Web process but the Web process allows for multiple workers. In theory a Web process with 1 worker, 4 threads (1×4) will be equivalent to 2 workers, 2 threads (2×2).

To try and keep this explanation as simple as possible I will only give a brief overlook to what Sidekiq and the Web processes are but if you need more details, please contact me or check the Mastodon documentation.

Sidekiq deals with all the work that doesn’t have to be done immediately. For example when you favourite or follow or post something, that will add jobs to the Sidekiq queue and will only process them when the Sidekiq has available threads to deal with it.

So, if you are followed by 500 users in 50 instances. When you post something publicly, 50 new jobs are added to the queue so your instance will “notify” the remote instances about the new post. This may sound like something that will take a lot of time for Sidekiq to process but a 2 threaded Sidekiq can do this in well under a minute.

The Web process is what responds to realtime requests for short-lived http requests, like loading the web interface. The logic to the Web process threads is similar but there is no queue because every request needs to be replied to, if the reply takes too long it just timeouts and you get an error.

The Web process is very fast, the time it takes to respond is measured in microseconds, so when you request a page load, as soon as the page finishes loading the thread is free again, while you are reading and browsing the page the Web process is not requested again.

In both cases it is really hard to make an estimate on how many threads are required for each instance because it depends on factors from the following/follower counts to the frequency of posts made and received and the general usage of the instance.

Each instance is unique and the usage depends greatly on the community that forms around it. Only with time one can start to get a sense of the resources needed. Still, you can upgrade and downgrade your plan at any time and create your own instance rules to try and adjust the usage to the plan you want to use.