Stall log check, 30 October to 5 November

Obeya's stalls.log and app.log on the owner's Mac, from 29 October around 20:00 to 5 November 08:00 (CET). Follow-up to the fix for slow look-ups of archived cards.

Fourth check, 5 November: more stalls, five new groups

From 2 November 08:00 to 5 November 08:00 (CET), with nothing lost (stalls.log did not start afresh). Obeya logged 72 server stalls, together 71 s; the 43 hours before had one. Most fell on 2 and 3 November, during long working days with many landings, the 0.11 release and a large number of shared pages.

shorterlooking up archived cards for sharing: the prototypes no longer show up, the look-up itself still stalls
72server stalls of 0.5–6.9 s; 7 of them over 1.5 s
106page stalls on this machine, up to 36 s, cause not logged

Looking up archived cards for sharing

The fix Looking up an archived card reads the canvas's prototypes once per change was running from 2 November. Reading the prototypes (withPrototypes) shows up in one stall only, on 2 November at 13:20, for 13% of it, and in none after. But sharing still stalls on the same path: 11 stalls, up to 4.8 s (3 November, 16:30, when Obeya checked its demo sites). The time now goes into building each of the about 460 shared cards one by one: reading its row several times and checking its demo files on disk.

board snapshotsharing look-upsgit callsdatabase, Mac busydemo sitesingle cases 2 Nov 08:003 Nov 08:004 Nov 08:005 Nov 08:00

Server stalls from 2 November 08:00 to 5 November 08:00 (CET), one row per group; the larger the dot, the longer the stall.

New causes, grouped

GroupStallsLengthWhere the time wentProposed card
1. Board snapshot
on changes, 22 s in all
180.5–6.9 s Each rebuild works out anew which shared pages are outdated: it reads all ~460 shared cards, parses them and checks the demo files of each outdated page on disk. Since pages take their repository's look (3 November), a theme change makes nearly every page outdated. Reading plan folders and placing new cards adds to it. Hinweis auf veraltete geteilte Seiten verteuert jeden Board-Snapshot
2. Synchronous git calls
mostly at server starts
190.5–4.7 s Reading every repository of every canvas again after a start, one after the other (the 4.7 s stall on 2 November, 13:31); asking each repository for its origin at every settings check; asking git for the GitHub login (git credential fill) for commands on a host. The earlier fix took git off agent starts and landings; these places remained. Restliche synchrone git-Aufrufe im Server
3. Sharing look-ups
checking the demo sites, publishing
110.5–4.8 s See above: every shared card is looked up and built on its own Teilen baut jede geteilte Karte einzeln auf
4. Database reads while the Mac was busy
mostly 3 November, 15:17–16:42
130.5–1.6 s Reads of the Koordinator's history and of the questions it waits on, and writing a conversation line. Measured alone just now, the same reads take 0.2–25 ms; in the log they took hundreds to thousands of times longer, four of them in a burst of seven stalls within 17 seconds. That points at the Mac being short of memory or disk at the time, not at the code. none
5. Demo site
at each share
90.5–1.1 s Rewriting the site's overview and feed from every page's files, checking every page's files against the live site, asking git for the author's name, all with synchronous file calls Demo-Site schreibt ihre Übersicht synchron beim Teilen
Single cases 20.5–0.7 s Listing an agent's skills; reading a session's transcript for the work retrospective none

Page stalls

106 stalls of a page on this machine (93 on canvas Obeya, 10 on Nemotron), mostly in bursts: 2 November 09:11–09:15 (up to 6.1 s), 3 November 10:48–10:53 (34 in four minutes), 11:55–12:03 (29, up to 36 s) and 16:32 on Nemotron (up to 9.6 s). None took a click or key, and the server had no stall at those times. The browser says nothing of where the time went, so it is WebKit (the Mac app or Safari). Whether the page really froze or the system held the window back, as with the false alarm fixed on 30 October, the log cannot tell.

Proposed card: Lange Seiten-Hänger ohne Ursache im Log (bis 36 s), to find out which it is and log it.

2 November

Third check: the three groups are fixed

All three groups below were fixed on 31 October at 12:47 (demo frames clean-up, board changes, git calls on agent starts and landings). In the 43 hours since then, Obeya logged:

WhatStallsLengthWhere the time wentProposed card
Looking up archived cards for sharing
31 Oct, 20:08, as a landing shared its demo
10.5 s The same path as the original problem, now far shorter: each look-up of one archived card still loads and prepares every prototype of the canvas, once for each of about 200 shared cards Nachschlagen archivierter Karten lädt alle Prototypen jedes Mal neu
A page on this machine
2 Nov, 02:25 and 05:01, canvas Obeya
20.7–0.8 s The browser does not say where the time went; both at night none

No stall of the demo frames clean-up, the board snapshot or the git calls has come back.

30 and 31 October

0stalls from looking up archived cards (the fixed cause)
gonefalse page stalls, about one per second, from the Mac app's window
12real server stalls of 0.5–1.1 s, in three new groups

The fixed cause stays away

Before the fix, sharing looked up every shared card in the archive and held the server up to 10 s. In both checks no stall pointed there, although many cards landed and were shared in that time. One stall on 30 October touched an archived-card look-up for 7% of its 0.7 s; most of its time went elsewhere (see group 2 below).

The flood of false page stalls, now fixed

From the app's start on 29 October 20:32 until 30 October 09:37, the Mac app's window (canvas Super 59) reported a stall of about 0.9 s nearly every second: over 40,000 entries, all night long. The page was not frozen (its process idled at under 1% CPU): WebKit ran the page's watch timer only once a second while still calling the page visible. The flood made stalls.log start afresh every two hours, so real server stalls of the afternoon of 29 October were lost.

Fixed on 30 October at 12:02 (A page held back by the browser no longer reports stalls; repeated ones are counted). Since then: one page stall in 21 hours (0.6 s, canvas Obeya).

30 Oct 09:0015:0021:0031 Oct 03:0009:00 ● git calls ● board snapshot ● demo frames clean-up

Server stalls from 30 October 09:00 to 31 October 09:00 (CET). Eight of them fell in the half hour around noon on 30 October, when many cards landed one after the other.

New causes, grouped

GroupStallsLengthWhere the time wentProposed card
1. Demo frames clean-up
at start, then every six hours
50.5–0.7 s Walking the demo folder (about 1,000 entries) with synchronous file calls (lstatSync, readdirSync in demo-space.ts) Aufräumen alter Demo-Frames ohne Server-Hänger
2. Board snapshot on every change
during bursts of activity
40.5–0.7 s The Koordinator rebuilds the whole board snapshot on every change only to find projects being arranged; the owner's language file is read from disk at every translation call Server-Hänger durch Board-Snapshot bei jeder Änderung
3. Synchronous git calls
during bursts of landings
30.7–1.1 s Synchronous git when an agent starts (is the repository's Claude configuration trusted?) and when a repository is read again after main moved, plus many git processes started at once Synchrone git-Aufrufe beim Agentenstart und Neulesen der Repositories

Group 1 counts one stall from 29 October, 20:32, before the window above. None of the stalls are long enough to notice as more than a brief hitch; the longest was 1.1 s.

What could not be checked