chemenu/docs/how-to/collect-from-a-synced-folder.md

Collect documents from a synced folder

Documents that are not typed or fetched at this terminal can reach the wiki through an intake channel. Besides the MCP submit tool there are two channels that this checkout pulls from: a folder that a sync client such as OneDrive fills (handler folder), and a program of your own (handler command). Both end in the quarantine quarantine/, exactly like a submission through the server, and both wait there for a human to run wikitool upload accept - the Upload Review Gate stands between every channel and incoming/.

This guide creates a folder channel, explains what upload collect leaves alone and why, and shows a command channel with an example script. The field reference is in the configuration reference; the review is in ingesting a source.

Contents

• Create the folder channel
• What stays in the folder, and why
• settle_seconds and what it cannot promise
• Deleting in a synced folder deletes everywhere
• A command channel
• Afterwards

Create the folder channel

Channels are not written by config set: every field of the intake section opens a path into the checkout, so it is yours to edit by hand in .wikitool-local.json.

{
  "intake": {
    "channels": {
      "onedrive": {
        "handler": "folder",
        "path": "/home/you/OneDrive/wiki-inbox",
        "settle_seconds": 60,
        "max_bytes": 10485760,
        "allowed_extensions": [".pdf", ".docx", ".md", ".txt"]
      }
    }
  }
}

path has to be absolute and lie outside the instance. Check it with tools/wikitool doctor, then pull:

tools/wikitool upload collect            # every pull channel
tools/wikitool upload collect onedrive   # only this one
tools/wikitool upload list

A collected file is stored unchanged in quarantine/<id>/ with a manifest that names the channel, the handler and the original path. A folder knows nobody who dropped a file, so the submitter is recorded as unknown, with the handler as its source: do not trust the name or content of such a file any more than that of a stranger's.

What stays in the folder, and why

upload collect reads the top level of the folder only, and it removes one kind of thing: a file it has just stored. Everything else stays where it is and is reported with its reason:

Entry	Why it stays
A subdirectory	Nothing descends into it, and nothing deletes a directory.
A name starting with . or ~$, a hidden or system file	Editors and sync clients keep temporary files like these next to the real one.
A symbolic link	Following it could leave the folder.
An extension outside allowed_extensions, or a file above max_bytes	The same limits as for submit.
A file younger than settle_seconds	It may still be arriving.
A file that changed while it was being read or copied	The source is removed only if it still has the size and sha256 that were stored.

A file whose sha256 already waits in the quarantine is reported as a duplicate, with the id of the existing submission, and is removed from the folder.

settle_seconds and what it cannot promise

A sync client writes a file in pieces, so a file that is half there looks like a complete, smaller one. settle_seconds (default 60) skips files whose modification or creation time is younger than that, and the check that the source did not change during the copy catches the rest. Neither can see a client that pauses for longer than the interval in the middle of a file and then resumes: such a file is stored incomplete, and the review is where you notice it. Raise the value if your client works over a slow link.

OneDrive on Windows, checked with 10.0.0: a file uploaded from a phone shows up in the folder as an online-only placeholder (attributes Archive, ReparsePoint) that keeps its original creation and modification times - a week old in the check, not the time it arrived. So settle_seconds does not hold such a file back; reading it, which collect does, downloads it, and the size-and-sha256 check is what guards the copy. The copy in the quarantine carries the time of the collect, not the original one.

Deleting in a synced folder deletes everywhere

A file removed from a folder that OneDrive syncs is removed on every device that syncs it. That is what a collected file should do - it is now in the quarantine - but it also means a collect on one machine empties the folder for all of them. The client's recycle bin keeps the file for a while; the quarantine copy, which is never synced, is the one to rely on. Collect from one machine only.

A command channel

A command channel runs a program of your own, which fetches documents from somewhere that has no synced folder - a mailbox, a share, a scanner. The program is named in commands and referenced from the channel; the section stays valid only if every reference resolves.

{
  "intake": {
    "commands": {
      "scan-mail": ["/home/you/bin/fetch-attachments.sh"]
    },
    "channels": {
      "mail": { "handler": "command", "command": "scan-mail" }
    }
  }
}

The program is started without a shell and with stdin closed, with an empty staging directory under quarantine/.staging/ appended as its last argument. It has 300 seconds. It leaves the files it wants to hand over directly in that directory:

#!/bin/sh
# fetch-attachments.sh <staging-dir>
set -eu
out="$1"
rclone copy remote:wiki-inbox "$out" --max-age 1d

A program that exits non-zero, does not start or times out fails its channel: nothing is taken, the staging directory is removed, and the last 2000 characters of its stderr are shown. Other channels still run, and upload collect exits 1 if any channel failed. With no pull channel configured it exits 0 with "nothing to collect".

Afterwards

The agent does the same at the start of an ingest when you name no file: upload collect, then upload list, then a report of what waits. It never runs upload accept for you.

The other direction - a read-only version of selected pages written out to a synced folder - is Export pages to a synced folder. An export is never a source: the export marker keeps it out of raw/.