WaitForDownloadAsync Never Returns


#1

This is more of a .NET question than Xamarin. I have a Windows Service that runs periodically to update the data in the ROS from our CRM. On a number of occasions when debugging the service I have noticed that it hangs when running the following code;

var session = driverRealm.GetSession();
Thread.Sleep(250);
AsyncContext.Run(() => session.WaitForDownloadAsync());

I have checked the Nito and Realm documentation and can’t find any information on setting a timeout to circumvent this problem.

Do you have any suggestions?


(Nikola Irinchev) #2

The AsyncContext.Run(() => session.WaitForDownloadAsync()) will be executed on a different thread which is not supported. Instead, you should wrap all the Realm access logic inside a single AsyncContext.Run block:

AsyncContext.Run(async () =>
{
    using (var realm = await Realm.GetInstanceAsync(...))
    {
        // write some data
        await realm.GetSession().WaitForDownloadAsync();
    }
});

#3

I modified my code as follows;

foreach (DataRow driverRow in drivers)
{
	var userName = driver.Field<string>("UserName");
	var driverId = driverRow.Field<Guid>("DriverId").ToString();
	var driver = GetDbDriver(db, driverId);
	var user = GetRealmUser(db, driver);
	try
	{
		AsyncContext.Run(async () =>
		{
			using (var driverRealm = OpenRealm($"~/{RealmName}", user, userName))
			{
				//await SynchroniseRealm(driverRealm, false, true);
				await driverRealm.GetSession().WaitForDownloadAsync();
				...
			}
		});
	}
	catch (Exception ex)
	{
		Service.LogException(ex.Message));
	}
}

private static async Task SynchroniseRealm(Realm realm, bool upload, bool download)
{
	var session = realm.GetSession();
	//Thread.Sleep(250);
	if (upload) await session.WaitForUploadAsync();
	if (download) await session.WaitForDownloadAsync();
}

And regardless of whether I use SynchroniseRealm (with or without Thread.Sleep) or the syntax shown above, I get an unknown error when executing WaitForDownloadAsync.

The error occurs intermittently, that is; it can occur on the first iteration of the enclosing foreach or any subsequent iteration.

The output window shows;

Exception thrown: 'Realms.Exceptions.RealmException' in mscorlib.dll

And the Exception message contains;

A system error occurred while waiting for completion. See InnerException for more details

And the inner exception contains;

Unknown error


#4

Could this be related to GitHub issue WaitForDownloadAsync often does not terminate after switching from offline to online #1887?


(Nikola Irinchev) #5

Interesting, can you post the entire exception + stacktrace here? I believe just .ToString-ing it will produce a detailed enough output to investigate. I’m mostly interested in the ErrorCode if there’s one and the stacktrace.


#6

Hi Nikola,

Sorry I didn’t get back to you on this one but the issue was resolved after I made to some mods to the code in the way that I synchronised the data.

I have just raised a support ticket (https://support.realm.io/helpdesk/tickets/4783) regarding an issue deleting realms and now find that WaitForUploadAsync never returns on those realms that were not deleted correctly. I have had similar issues recently when there has been a problem with the ROS and was again looking at the issue of timing out the task in the event it appears to have hung.

I was contemplating using the following code;

		public async Task DoWithTimeout(Action action, Action timeoutAction, TimeSpan timeout)
		{
			using (var cancellationSource = new CancellationTokenSource())
			{
				var task = Task.Run(action, cancellationSource.Token);

				if (await Task.WhenAny(task, Task.Delay(timeout)) != task)
				{
					cancellationSource.Cancel();
					timeoutAction();
				}
			}
		}

My concern is; would this leave the realm in an inconsistent state?


#7

We also have a support ticket outstanding (ticket 4767) for WaitForDownloadAsync never returns. It stopped returning on Friday and our users have been are out of business since.


#8

Hi Nikola,

This is still an issue for us as we still have realms that can’t be processed. Should I be raising this issue in the forum or should I raise a support ticket?


(Nikola Irinchev) #9

What do you mean by “can’t be processed”? I can imagine that WaitForUploadAsync would hang for a Realm that is deleted as the session would be disconnected. You can subscribe to the Session.Error event to get notified of the deletion and cancel the wait, although I can imagine that would be a bit cumbersome. Alternatively, you can set a timeout for the WaitForUploadAsync operation.


#10

The realm is not being deleted during the operation of WaitForUploadAsync or WaitForDownloadAsync, the realm was deleted using Realm Studio however it was not deleted correctly and now exists without a schema - https://support.realm.io/support/tickets/4783. This ticket has been open for over 6 days now and I have not had a response despite repeated requests.

When our mobile app or any of our services attempts to access these defective realms using WaitForDownloadAsync or WaitForUploadAsync it waits indefinitely and never returns so I wanted to see if it was feasible to cancel the task after a given period without corrupting the realm.

I checked the documentation and there is no timeout mentioned for the upload or download operations so I presume you mean we can forcefully cancel the operation as I discuss above. This still leaves the question; if I forcefully cancel the operation will this leave the realm in an indeterminate state. Will I need to delete the realm and start again or will the realm be left in the same state as before the operation was commenced?


(Nikola Irinchev) #11

The Realm will never be corrupted regardless of whether you wait for the download to complete or not. The “defective” Realms issue sounds like you’re deleting a Realm while there are connected clients. A Realm is created on the server whenever a client tries to get an access token for it. That means that if a user that has rights to create a Realm in a particular location (e.g. an admin user or a regular user accessing a Realm in their own folder /~/), it’s possible that a token refresh request is in flight when the deletion goes through. This will cause ROS to recreate the server Realm file to make sure that the client has something to connect to with the newly issued access token. Such a Realm will have no schema because no client has uploaded the schema to it. If that’s the case, you can just delete the Realm again.

A client whose server Realm has been deleted will experience a client reset error so you’ll need to handle that to avoid waiting indefinitely.


#12

The Realms I was deleting are used for my testing and I can verify there were no clients attached when I was deleting them.

Is there any way I can get someone to respond to my ticket? It has now been over a week since I reported this issue and I still can’t correctly delete any of these realms.


(Nikola Irinchev) #13

It may not be obvious, but Studio is a client itself, so if you had them open in Studio while deleting them, that could also cause them to be recreated. Similarly, if you have a client that tries to open a Realm after it was deleted, it will cause the Realm to be recreated if the client has permissions to create it.

In any case, have you tried deleting the Realms again? It’d be very surprising if they are undeletable.


#14

Yes - tried again yesterday but got another error which I reported. Still have not had a response regarding this open issue. Not impressed.


#15

This is the error log if it helps;

Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)

(Nikola Irinchev) #16

Interesting… looks like the deletion goes through in one subsystem, but fails in the other. I expect the Realm disappears from the list of realms in Studio? Then when you try to connect to it, it exhibits weird behavior? Do you have a timeframe for when this error occurred? I’ll have to look at the server logs to try and understand what was special about these Realms and it’ll help to narrow down the search interval.


#17

The Realm eventually disappears from the list of Realms but not immediately.

The following are the dates and time I have attempted to delete a Realm. The times appear to be my local time which is Brisbane Australia (UTC +10:00)

Errors occurred between 9/17/2019 1:27:53 PM and 9/17/2019 2:08:53 PM

Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)

Errors occurred between 9/18/2019 10:03:54 AM and 9/18/2019 10:07:54 AM

Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)

Errors occurred between 9/20/2019 8:45:55 AM and 9/20/2019 8:49:55 AM

Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)

Errors occurred between 9/23/2019 9:07:58 AM and 9/23/2019 9:11:58 AM

Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)

Errors occurred between 9/25/2019 1:43:59 PM and 9/25/2019 1:47:59 PM

Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)
Could not delete realm from backend sync worker: Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19)

I have just tried deleting another test Realm and the problem is still apparent.


(Nikola Irinchev) #18

I can see the failures in the logs - seems like connectivity issue between the internal components. Need to investigate further to find out the root cause. In the meantime, you can work around it by adding a number suffix to the Realm path and incrementing it every time you need to “delete” the existing Realm. I know it’s less than ideal - we’ll try to resolve that quickly.