chemenu/docs/how-to/collect-from-a-synced-folder.md

# Collect documents from a synced folder

Documents that are not typed or fetched at this terminal can reach the wiki through an intake
channel. Besides the MCP `submit` tool there are two channels that this checkout *pulls* from: a
folder that a sync client such as OneDrive fills (handler `folder`), and a program of your own
(handler `command`). Both end in the quarantine `quarantine/`, exactly like a submission through the
server, and both wait there for a human to run `wikitool upload accept` - the Upload Review Gate
stands between every channel and `incoming/`.

This guide creates a `folder` channel, explains what `upload collect` leaves alone and why, and shows
a `command` channel with an example script. The field reference is in the
[configuration reference](../reference/configuration.md#intake-channels); the review is in
[ingesting a source](ingest-a-source.md#a-document-submitted-from-outside).

<!-- wikitool:toc -->
## Contents

- [Create the folder channel](#create-the-folder-channel)
- [What stays in the folder, and why](#what-stays-in-the-folder-and-why)
- [`settle_seconds` and what it cannot promise](#settle_seconds-and-what-it-cannot-promise)
- [Deleting in a synced folder deletes everywhere](#deleting-in-a-synced-folder-deletes-everywhere)
- [A command channel](#a-command-channel)
- [Afterwards](#afterwards)
<!-- /wikitool:toc -->

## Create the folder channel

Channels are not written by `config set`: every field of the `intake` section opens a path into the
checkout, so it is yours to edit by hand in `.wikitool-local.json`.

```json
{
  "intake": {
    "channels": {
      "onedrive": {
        "handler": "folder",
        "path": "/home/you/OneDrive/wiki-inbox",
        "settle_seconds": 60,
        "max_bytes": 10485760,
        "allowed_extensions": [".pdf", ".docx", ".md", ".txt"]
      }
    }
  }
}
```

`path` has to be absolute and lie outside the instance. It may sit in the folder that also holds
the backup and an export, as `<OneDrive - company>\Chemenu\Inbox` beside `git`, the machine folder
and `Export` ([the layout](back-up-to-a-synced-folder.md#one-folder-for-the-backup-the-intake-and-the-export)),
but it must not be, or lie in, `<backup.target>/git` or a machine folder of the backup. `upload
collect` also refuses, for that channel alone and before it reads anything, a folder that holds or
lies in a machine folder of any machine, a bare repository or an export: it empties its folder, and
these are not submissions. Check it with `tools/wikitool doctor`, then pull:

```bash
tools/wikitool upload collect            # every pull channel
tools/wikitool upload collect onedrive   # only this one
tools/wikitool upload list
```

A collected file is stored unchanged in `quarantine/<id>/` with a manifest that names the channel,
the handler and the original path. A folder knows nobody who dropped a file, so the submitter is
recorded as `unknown`, with the handler as its source: do not trust the name or content of such a
file any more than that of a stranger's.

## What stays in the folder, and why

`upload collect` reads the top level of the folder only, and it removes one kind of thing: a file it
has just stored. Everything else stays where it is and is reported with its reason:

| Entry | Why it stays |
|---|---|
| A subdirectory | Nothing descends into it, and nothing deletes a directory. |
| A name starting with `.` or `~$`, a hidden or system file | Editors and sync clients keep temporary files like these next to the real one. |
| A symbolic link | Following it could leave the folder. |
| An extension outside `allowed_extensions`, or a file above `max_bytes` | The same limits as for `submit`. |
| A file younger than `settle_seconds` | It may still be arriving. |
| A file that changed while it was being read or copied | The source is removed only if it still has the size and sha256 that were stored. |

A file whose sha256 already waits in the quarantine is reported as a duplicate, with the id of the
existing submission, and is removed from the folder.

## `settle_seconds` and what it cannot promise

A sync client writes a file in pieces, so a file that is half there looks like a complete, smaller
one. `settle_seconds` (default 60) skips files whose modification or creation time is younger than
that, and the check that the source did not change during the copy catches the rest. Neither can see
a client that pauses for longer than the interval in the middle of a file and then resumes: such a
file is stored incomplete, and the review is where you notice it. Raise the value if your client
works over a slow link.

OneDrive on Windows, checked with 10.0.0: a file uploaded from a phone shows up in the folder as an
online-only placeholder (attributes `Archive, ReparsePoint`) that keeps its original creation and
modification times - a week old in the check, not the time it arrived. So `settle_seconds` does not
hold such a file back; reading it, which `collect` does, downloads it, and the size-and-sha256
check is what guards the copy. The copy in the quarantine carries the time of the collect, not the
original one.

## Deleting in a synced folder deletes everywhere

A file removed from a folder that OneDrive syncs is removed on every device that syncs it. That is
what a collected file should do - it is now in the quarantine - but it also means a collect on one
machine empties the folder for all of them. The client's recycle bin keeps the file for a while; the
quarantine copy, which is never synced, is the one to rely on. Collect from one machine only.

## A command channel

A `command` channel runs a program of your own, which fetches documents from somewhere that has no
synced folder - a mailbox, a share, a scanner. The program is named in `commands` and referenced from
the channel; the section stays valid only if every reference resolves.

```json
{
  "intake": {
    "commands": {
      "scan-mail": ["/home/you/bin/fetch-attachments.sh"]
    },
    "channels": {
      "mail": { "handler": "command", "command": "scan-mail" }
    }
  }
}
```

The program is started without a shell and with stdin closed, with an empty staging directory under
`quarantine/.staging/` appended as its last argument. It has 300 seconds. It leaves the files it
wants to hand over directly in that directory:

```bash
#!/bin/sh
# fetch-attachments.sh <staging-dir>
set -eu
out="$1"
rclone copy remote:wiki-inbox "$out" --max-age 1d
```

A program that exits non-zero, does not start or times out fails its channel: nothing is taken, the
staging directory is removed, and the last 2000 characters of its stderr are shown. Other channels
still run, and `upload collect` exits 1 if any channel failed. With no pull channel configured it
exits 0 with "nothing to collect".

## Afterwards

The agent does the same at the start of an ingest when you name no file: `upload collect`, then
`upload list`, then a report of what waits. It never runs `upload accept` for you.

The other direction - a read-only version of selected pages written out to a synced folder - is
[Export pages to a synced folder](export-to-a-synced-folder.md). An export is never a source: the
export marker keeps it out of `raw/`.