Server stalls from a growing history: before and after

Obeya's server runs on one thread. Several of the reads it repeats on every change, every Koordinator command and every pass over the shared pages went through whole tables, so they took longer with every day of use. While one ran, every page and the phone froze.

99server stalls of 0.5–6.9 s, 30 Oct – 8 Nov
92 sof frozen server in total
6 / 6listed rounds now within budget over a 10× history

What each round costs over a history ten times today's

A synthetic database with ten times the history of the owner's Obeya: 1.6 million events over 400 days, 9,000 archived cards and 4,500 shared pages on one canvas, and 10,000 Koordinator exchanges. The unit is one rebuild of the canvas view on a canvas with the same cards and no history (about 1.5 ms on this laptop), measured right beside each round, so the budgets hold on slower machines too. The scale is logarithmic.

BeforeAfterBudget in the load test
Table view

On a copy of today's database (55 MB)

The same reads on a copy of the real database, warm cache. A busy machine that has to read pages back from disk or swap stretches these into the stalls the log shows.

ReadBeforeAfter
Koordinator brief: what happened in the last 14 days28 ms0.14 ms
Learner: what the owner said in the last days24 ms0.10 ms
Steps the Koordinator waits to take (every change)0.06 ms0.002 ms

Where the 99 stalls went

Cause (top of the sampled stack)StallsSecondsNow
Synchronous process calls (git, ssh)2017.1Most fixed before this card; the rest moved off the thread, a test counts what remains
Synchronous file calls (demo site, demo sweep, stats)1913.3Moved to async reads; what stays is listed with a reason
Notice of outdated pages (every shared-card change)1419.2Shared cards kept up to date from a log of their writes
Rebuilding the canvas view96.0Project costs and workstreams read by index
Steps the Koordinator waits on97.9Partial index
Checking the sites' shared pages911.2No archived cards built; one target per site
Koordinator brief (history of the last days)65.1Read by time through an index
Learner (the owner's recent words)54.4Read by time through an index
Other87.8Not addressed

Guard against new synchronous calls

A test counts every synchronous file and process call (every function ending in Sync) in the server's code against a list: per file, how many of each it may make and why. A new call fails the test until it is made asynchronous or listed with a reason; a call that goes must leave the list, so the list only shrinks. 48 files are listed today, most with startup, script or single-stat reasons.

Moved off the server thread in this change: a canvas's repository settings (no fallback read with git), the first canvas's folder, a demo site page's captions, posters and sound, the demo sweep's settings file, and a repository's theme for a page about to go out.