Notifications
Clear all

Big slowdown and allocations when querying a leaderboard that does not yet exist

(@punchey)
Eminent Member Registered

If I query for a leaderboard and it turns out not to exist yet, I get a huge lag. I profiled it and found this in the profiler (note the allocations shown in the attached screenshot). I don't seem to have any slowdowns if the leaderboard exists.  Any ideas what might be happening?

ReplyQuote
Topic starter Posted : 25/03/2020 1:32 am
(@punchey)
Eminent Member Registered

Okay, I've learned a bit more about this and made a HUGE optimization. Allocations went down from about 1.5GB to about 35.4kb!

The issue was repeated string concatenation in MiniJSON.parseString().  I simply replaced:

string s = "";

with:

StringBuilder s = new StringBuilder();

And then every occurrence of "s += <some character or variable>" I replaced with: "s.Append(<some character or variable>)".

The problem was that with each additional character that was appended, an entirely new copy of the string so far was created. So for a result string that's 42,000 characters long (as mine is), it's allocating roughly 42,000 new strings of increasing length.  Speaking of 42,000 characters, that brings me to the other issue I discovered:

If I request scores for a specific user on a leaderboard ID that doesn't exist yet, or for some reason has a malformed ID, instead of just returning some kind of error, it returns a list of records for every single leaderboard that exists, even if the user doesn't have any scores in them. Currently, I have 61 leaderboards, but as users create their own custom content, each with its own leaderboard, that number will grow drastically, probably eventually numbering in the thousands. Obviously, I don't want under any circumstances for a result string to be that huge.  It would cripple my server and saturate the user's bandwidth.

Is there a modification I can make to the server scripts to just have it return an error string of some kind instead?

ReplyQuote
Topic starter Posted : 26/03/2020 10:37 pm
(@skaredcreations)
Prominent Member Admin

Oh MiniJSON is an open-source project/script and honestly I never went through it for optimization, but I'll test your suggestion since StringBuilder is usually much preferred to string concatenation for the reason you already said.

I'll also check your issue upon retrieving specific user score in inexistent leaderboards and will let you know here.

FRANCESCO CROCETTI @ SKARED CREATIONS

ReplyQuote
Posted : 27/03/2020 12:59 am
(@skaredcreations)
Prominent Member Admin

Ok I checked out your issue and by reading the code now I remember why it's returning scores from all leaderboards, basically this was intended in order exactly to get the scores of all leaderboards.

If you don't need this feature then for now you can force to FALSE the variable $isMultiple at line 118 of /leaderboards.php, I will add an optional boolean parameter to LoadScoresByUser to decide on client side if it should retrieve the scores of all leaderboards (default will be true, for compatibility reason).

Also since you're now my favorite tester, please could you try the MiniJSON script attached here (rename to .cs) and eventually report any issue using it? I've implemented your suggested change and other small optimizations: removed use of "MiniJSON." to get local class members/constants and replaced value type usages (String, Char, Double, etc) with their respective primitive types.

FRANCESCO CROCETTI @ SKARED CREATIONS

ReplyQuote
Posted : 27/03/2020 1:38 am
(@punchey)
Eminent Member Registered

I see, I just Googled it, and it looks like someone has already integrated the same change as I made:
https://gist.github.com/darktable/1411710

Looking forward to hearing news about the other issue.  Thanks so much!

ReplyQuote
Topic starter Posted : 27/03/2020 1:59 am
(@punchey)
Eminent Member Registered

Haha, sorry, I posted that reply without seeing that you had already replied. I'll give that a test right away.  And thanks for the solution for leaderboards.php!  I'll give that a try as well.  Will report back here.

Also, I think I discovered another issue: if for some reason the userID is invalidated (either from a failure to authenticate, or failure to ping), then if you make a call to LoadScoresByUser(), the part where it parses the result will generate a null ref as the response will only have Rank, User, Scores, and Page fields, but no IdLeaderboard or CodeLeaderboard, which the Score() constructor tries to concatenate to a string, but will return null since those keys are not present. I realize that technically you're not supposed to call it under those circumstances, but it might be good to put a check in there just to avoid null refs just in case.  Something like:


score = new Score(
result.ContainsKey("IdLeaderboard") ? result["IdLeaderboard"] + "" : "",
result.ContainsKey("CodeLeaderboard") ? result["CodeLeaderboard"] + "" : "",
...

 

ReplyQuote
Topic starter Posted : 27/03/2020 8:30 pm
(@punchey)
Eminent Member Registered

Everything seems to be working fine!  Thanks so much!

ReplyQuote
Topic starter Posted : 28/03/2020 3:18 am
Share: