Massive memory spike when CommitWrite()


#1

We see a massive spike in memory usage while inserting a large amount of data (more precisely, when committing a transaction). Of course we would like to avoid this memory pressure. Here a quick screenshot, and more detailed below

  • Realm swift version used : 3.11.0
  • iOS: 12
  • Amount of data to be inserted : 200 Mo, Over 270k Objects

Plus, we have the following requirements :

  • We download data from distant SQL Server
  • We need to be sure that all data has been correctly downloaded, as we do not want to end up with a error in the intergrity of our database (missing data)

So with this, we decided the following steps:

  • Storing all the downloaded data into temporary seperate files (on disk)
  • Once all data downloaded is complete, we go into a specific operation queue
  • We open a transaction
  • We open file by file, saving objects, then closing the file
  • If no error during the process, commit the transaction
  • And this is at this point that we see the memory spike, when commiting the transaction

This translate roughly like this in term of code (simplified)

//We have a small helper, that is use also for other classes
static func getRealm() -> Realm {
        do {
            let realm = try Realm()
            return realm
        } catch let error as NSError {
				//Log + error
        }
}

// The code where the spike occurs
DispatchQueue.global(qos: .background).async {
	do {
		// opening transaction
		RealmHelper.getRealm().beginWrite()
		
		let filesURL = StorageHelper.retrieveFilesURL()
		for fileURL in filesURL {
			autoreleasepool { // to clear decoded data
				//Retrieving data + Decoding objects
				let objects = StorageHelper.retrieve(forURL: fileURL)
				realm.add(objects.persons, update: true)
				realm.add(objects.buildings, update: true)
				// and so on (15 differents objects types)
			}
		}
		
		RealmHelper.getRealm().commitWrite()
	} catch let error as NSError {
		RealmHelper.getRealm().cancelWrite()
	}
}

While it works, do you think this approach is the best one ? If there any way to reduce the memory spike? We initialy thought that by saving objects bits by bits, and batching the save, it would avoid this kind of behavior, but it doesn’t. And as I’ve stated earlier, we cannot split into several transactions (plus according to the documentation it is not recommanded for large amount of data).