Limiting ROS Realm file size growth


#1

Hi Folks,

Getting near to deploying our custom Node app with ROS into production and we’re seeing enormous Realm file size growth server side. We’re monitoring/tracking deployed IoT devices that updates its data frequently: About every 30 seconds.

For just one IoT endpoint (corresponds to one Realm file), there are 12 schemas with a total of about 500 entries for the endpoint. Mix of short strings and ints. Flat file size is well under 100KB, but we’re seeing growth of 50MB per day per Realm Database file (one IoT endpoint). If we look at all our endpoints we want to track, that’s nearly 50 Terabytes per year and isn’t realistic.

The only thing connected to ROS is our servers via GraphQL and the node app that is reacting to realm changes. We’ve turned off the Realm Studio to see if that helped…it didn’t seem to have any impact.

With frequent updating data, I’m guessing that the file is full of transaction logs…but with no clients connected…why does Realm need to store all this history?

In reading other posts, it’s possible that this space is a lot of empty/unused space. What is the best practice here to keep these on-disk file sizes compacted/compressed? Or at the very least, what options do we have to limit the file size here?

Wes

ROS v3.1.0 RC1


#2

I’m having similar “growth rate” for a non-IOT solution. I think the server saves the whole history and als leaves free space. Especially with regards to the new cloud pricing limit of 2.5GB this throws up questions about future plans regarding file sizes…


#3

Likewise. Yesterday I uploaded a test data set to Realm Cloud. This data is normally ~500mb whether it be json, SQLite, or MS Access. The local realm grew to a tad over 1gb before our instance disappeared completely.

Even if our server does come back, the file size is a surprise. Some time ago (can’t find the post) I read that the realm file size would be smaller than json/SQLite. The unexpectedly large file size means we have to revisit the cost of realm and how we pass that onto our users.


#4

Hey everyone, we appreciate you bringing this to light. We are aware of this and are investigating potential solutions. We will share more information as we have it.


#5

Any update on this? I am having the same issue with my realm file growing, I am already up to 2.5gb and yesterday it was just under 1.9gb? I am self hosting btw…


#6

I am facing same issue with a node.js app for storing latest state of iot devices.
At present I have 3 schemas (10 columns each - all numbers and booleans ) each with 1 row / device. We have around 1600 devices live and around they send data every second. At max 200 devices are online at any given time.

I always keep realm in transaction mode by calling beginTransaction at start of the app and then every second I call commitTransaction followed by beginTransaction. This ensures that realm is always in write mode,

In all there are close to 4000 rows which should be around 1MB of data. But the realm file has grown to 290 MB in 2 days.
As others a have pointed out, this is due transaction logs. Is there a way to disable it as I do not need any historical data?

I am using realm.js 2.3.3.