Sudden Spikes in Remote Media Storage

For the past few months, some Mastodon servers have had sudden spikes in media storage. In some cases, a server that was well within its plan limit ends up over it in just a few days, without anything changing on the server itself.

What is happening

Bots browse the public pages of Mastodon servers, like the Explore page, public timelines and remote profiles. When they access older remote posts, Mastodon downloads the media of those posts again if it was already removed from the cache. This happens even for posts that no one on your server has looked at in months.

It's a known issue in Mastodon, and you can follow it on GitHub.

Typically, a server suddenly pulls in 10 to 20 times the usual amount of remote media for a few days, then goes back to normal.

Does it go away?

In most cases, this resolves itself. All servers on Masto.host automatically delete the remote media cache after 7 days (see RE: Mastodon media storage), so once the spike is over, media storage goes back to normal. The problem is that the bots may come back, so the spikes can repeat.

How to reduce it

Under Administration -> Server Settings -> Discovery, set all the Public Timelines options to "Authenticated users only". This removes a big part of what the bots can crawl, but trending posts stay public. If you don't use trends, you can also uncheck "Enable trends" on the same page. It won't stop the bots completely, as other public pages are still accessible, but it should help a lot.

If your media storage keeps going up after changing the Discovery settings and waiting a week, please email info@masto.host and I'll take a look.