How to improve performance when streaming json in Realm on Android



My current Android application has to download and store a Json array.

This array can have upwards of 40 Million objects each of roughly 80 Bytes in length
(In string form).

I have implemented a Streaming server JAXRS endpoint to serve the Json data

and I am using retrofit and RxJava witihin my Android Application to consume it.

When I my profile my application memory usage it shows java.lang.String as having a shallow size of 178MB

my logcat is full of messages that resemble

zygote64: Background concurrent copying GC freed 10(32KB) AllocSpace objects, 3(64MB) LOS objects, 15% free, 129MB/153MB, paused 157us total 510.928ms

I am using this realm method to persist my json

memoryRealm.executeTransaction(realm -> realm.createOrUpdateAllFromJson(Code.class, inputStream));

the Realm is an inMemory() realm.

IS there anything I can configure to improve the performance of Realm streaming this massive amount of Json?


Unfortunately not easily.

The problem with createOrUpdateAllFromJson is that it returns the object being created. This means that you are effectively creating 40 million objects and throwing them away, which will cause GC.

This was the whole reason we created the insertOrUpdate method which is heavily optimized in terms of speed and allocations and for that reason also doesn’t return the object being inserted, but the JSON methods still use the slower copyToRealm method.

In your case what you really need is a streaming variant that uses the same optimizations as done in insertOrUpdate.

Something like that isn’t on our immediate roadmap, so your only choice right now would probably be to re-use the logic from our annotation processor to create your own that could generate it (probably easier said than done though).