Update/Write really slow when db grows


#1

I’m using a query based database in my app because I need the fine-grained permission system. I still need a full-sync experience so when I open the realm I do something like that to subscribe to all the data on cloud:


var realm = new Realm(config)

var obClass1 = realm.objects('obClass1')
obClass1.subscribe()
obClass1.addListener( // update UI )

I have some collections of x00k objects with relations between them and I have listeners to changes. I can read the data very fast.

The write operation, however, is really slow and so my app (the UI updates when the transaction is ended and a listener get the notification of changes).

I tried to delete some data from the realm and there is an improvement but I thought I could manage subscription to collections of hundreds of thousands of objects (very small ones).

I tried to compact the database like so and it seems better (is it even possible?)

Some write operations are writing more than one object at the same time (2-3, but since there are relations maybe more, at max 7-8).

I run the app on a Huawei M5 10’’, not the faster but not even a slow device.

I expect a write transition like the ones I do will take at max half a second.

The actual result is 4-5 seconds to complete.


#2

@ianward, can you please help me with this? It’s a crucial part for me to understand


#3

@au.petrone Not much I can say considering there is only high-level information. Performance issues are in the details unfortunately.

I can say that at a high-level if your performance is slow you can reduce the amount of data that you sync down to the device and simplify the schema of your data model. You could also look to do less logic in a write transaction or do it in smaller chunks. Part of this depends on what you are doing and your data model - and how ‘shardable’ it is


#4
export const Cliente = {
  name: "Cliente",
  primaryKey: "id",
  properties: {
    id: "string",
    tipologia: "string?",
    owner: "string",
    brand: 'string',
    agenzia: "Agenzia[]",
    ragionesociale: "string?",
    piva: "string?",
    nazione: "string?",
    indirizzo: "string?",
    citta: "string?",
    provincia: "string?",
    cap: "string?",
    email: "string?",
    email2: "string?",
    email3: "string?",
    telefono: "string?",
    telefono2: "string?",
    telefono3: "string?",
    telefono3: "string?",
    fax: "string?",
    web: "string?",
    referente: "string?",
    pictures: "Pic[]",
    schede: "Scheda[]",
    books: "Book[]",
    permissions: '__Permission[]'
  }
};

export const Scheda = {
  name: "Scheda",
  primaryKey: "id",
  properties: {
    id: "string",
    owner: "string",
    cliente: {type: 'linkingObjects', objectType: 'Cliente', property: 'schede'},
    preferenze : 'PreferenzaProdotto[]',
    timestamp: "date",
    tipologia: "string",
    note: 'string',
    permissions: '__Permission[]'
  }
};

export const Agente = {
  name: "Agente",
  primaryKey: "id",
  properties: {
    id: "string",
    user: "__User",
    nome: "string",
    agenzia: {type: 'linkingObjects', objectType: 'Agenzia', property: 'agenti'},
    progressivo: 'int',
    sigla: 'string'
  }
};

export const Agenzia = {
  name: "Agenzia",
  primaryKey: "id",
  properties: {
    id: "string",
    nome: "string",
    clienti: {type: 'linkingObjects', objectType: 'Cliente', property: 'agenzia'},
    agenti: "Agente[]",
  }
};


export const PreferenzaProdotto = {
  name: "PreferenzaProdotto",
  primaryKey: "id",
  properties: {
    id: "string",
    timestamp: "date",
    prodotto: "Prodotto?",
    variantipreferenzaprodotto: "VariantePreferenzaProdotto[]",
    prezzo_valore: "string?",
    prezzo_unita: "string?",
    prezzo_standard: "bool?",
    permissions: '__Permission[]',
    scheda: {type: 'linkingObjects', objectType: 'Scheda', property: 'preferenze'}
  }
};

export const VariantePreferenzaProdotto = {
  name: "VariantePreferenzaProdotto",
  primaryKey: "id",
  properties: {
    id: "string",
    colore: "string",
    quantita_valore: "string?",
    quantita_unita: "string?",    
    timestamp: "date",
    permissions: '__Permission[]',
    preferenzaprodotto: {type: 'linkingObjects', objectType: 'PreferenzaProdotto', property: 'variantipreferenzaprodotto'},
  }
};

export const Prodotto = {
  name: "Prodotto",
  primaryKey: "id",
  properties: {
    id: "string",
    colori: "string[]",   
    nome: "string",
    tipologia: "string",
    prezzo1: "string",
    collezione: 'string',
    prezzo2: "string",
    picture: "Pic",
    pictures: "Pic[]",
    cartellariferimento: 'string',
  }
};

export const Comunicazione = {
  name: "Comunicazione",
  primaryKey: "id",
  properties: {
    id: "string",
    title: "string",
    subtitle: "string",
    timestamp: "date",
    foto: 'data',
    fotoAutore: 'data?',
    body: "string",
    autore: "string",
    permissions: '__Permission[]',
  }
};

export const Notifica = {
  name: "Notifica",
  primaryKey: "id",
  properties: {
    id: "string",
    azione: "string",
    oggetto: "string",
    info: "string",
    data: 'date',
    autore: "string",
  }
};

export const Book = {
  name: "Book",
  primaryKey: "id",
  properties: {
    id: "string",
    timestamp: "date",
    stato :"string",
    owner: "string",
    note: 'string',
    consegnatoda: "string?",
    cliente: {type: 'linkingObjects', objectType: 'Cliente', property: 'books'},
    timestampconsegna: "date?",
    permissions: '__Permission[]',
  }
};


export const Pic = {
  name: "Pic",
  primaryKey: "id",
  properties: {
    id: "string",
    timestamp: "date",
    mediaParent:  {type: 'linkingObjects', objectType: 'Cliente', property: 'pictures'},
    permissions: '__Permission[]',
    data: "data",
  }
};

export const __Class = {
  name: '__Class',
  primaryKey: 'name',
  properties: {
    name: 'string',
    permissions: '__Permission[]'
  }
}

export const __Permission = {
  name: '__Permission',
  properties: {
    role: '__Role',
    canRead: 'bool',
    canUpdate: 'bool',
    canDelete: 'bool',
    canSetPermissions: 'bool',
    canQuery: 'bool',
    canCreate: 'bool',
    canModifySchema: 'bool'
  }
}

export const __Realm = {
  name: '__Realm',
  primaryKey: 'id',
  properties: {
    id: 'int',
    permissions: '__Permission[]'
  }
}

export const __Role = {
  name: '__Role',
  primaryKey: 'name',
  properties: {
    name: 'string',
    members: '__User[]'
  }
}

export const __User = {
  name: '__User',
  primaryKey: 'id',
  properties: {
    id: 'string',
    role: '__Role',
    group: '__Role'
  }
}

This is the current datamodel. Is it too much complex to handle?

My goal is to manage about xk cliente, xxk scheda, xxxk preferenza, xm variantipreferenza, that are linked togheter in this way cliente has a collection of scheda that has a collection of preferenza that has a collection of variantipreferenza. There are also backlinks.


#5

@au.petrone No - that data model is relatively simple. How much data are you pulling down per device and how much data are you updating per write?


#6

There are different groups of users.

The users that belong to the main company download 100% of the database on their devices. Other users may download less.

I write very small data (like create a new user, create new Scheda, etc or update specific fields of a document ). I suppose something like 50kb per write.

I think the problem may be related to the listeners but for what I understood it shouldn’t be a problem to have a listener to big collections.

Thanks for supporting me on this. If I can resolve this Realm will be my first choice for offline-first app surely.