---
title: "Destination Send Workflow"
output: 
  rmarkdown::html_vignette:
    css: mermaid-vignette.css
vignette: >
  %\VignetteIndexEntry{Destination Send Workflow}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

<script type="module">
  import mermaid from 'https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.esm.min.mjs';
  mermaid.initialize({ startOnLoad: true, theme: 'neutral' });
</script>

This vignette explains the internal workflow for sending content to remote destinations during builds. It describes the three main stages that occur when projr uploads artifacts to GitHub releases, local directories, or OSF nodes.

## Overview

When you run a production build (`projr_build_patch()`, `projr_build_minor()`, or `projr_build_major()`), projr can automatically send artifacts to configured remote destinations. This process is controlled by the `.dest_send_label()` function in `R/dest-send-label.R`, which orchestrates three distinct stages:

1. **Getting the Remotes** - Resolve remote locations and metadata
2. **Building the Plan** - Determine which files need to be added, removed, or updated
3. **Implementing the Plan** - Execute the file operations and update remote metadata

Each stage builds upon the previous one to ensure efficient and accurate synchronization between your local project and remote destinations.

## Stage 1: Getting the Remotes

The first stage resolves remote locations and determines what versions exist on the remote destination.

```{mermaid}
flowchart TD
    Start([Start: .dsl_get_remotes]) --> GetPre[Get remote_pre<br/>Base remote location]
    GetPre --> GetDestFull[Get remote_dest_full<br/>Full version destination]
    GetDestFull --> GetDestEmpty[Get remote_dest_empty<br/>Empty version destination]
    GetDestEmpty --> GetVersionComp[Determine version_comp<br/>Version to compare against]
    
    GetVersionComp --> CheckInspect{inspect == 'none'?}
    CheckInspect -->|Yes| ReturnNull[version_comp = NULL]
    CheckInspect -->|No| CheckStructure{structure?}
    
    CheckStructure -->|latest| LatestLogic[Check if remote exists<br/>Trust based on manifest]
    CheckStructure -->|archive| ArchiveLogic[Find latest acceptable<br/>archive version]
    
    LatestLogic --> GetComp[Get remote_comp<br/>Remote for comparison]
    ArchiveLogic --> GetComp
    ReturnNull --> GetComp
    
    GetComp --> End([Return remote list])
    
    style Start fill:#e1f5e1
    style End fill:#e1f5e1
    style GetVersionComp fill:#fff4e1
```

**Key Outputs:**

- `remote_pre` - The base remote location (e.g., GitHub release tag, local directory path)
- `remote_dest_full` - Where files will be uploaded (full version, not empty)
- `remote_dest_empty` - Empty version variant (for GitHub empty placeholders)
- `version_comp` - Version to compare against (NULL if no trusted version available)
- `remote_comp` - The actual remote object used for comparison

**Version Comparison Logic:**

The `version_comp` determines which remote version can be trusted for comparisons:

- **NULL** - No trusted version available; treat remote as empty (upload everything)
- **Version string** - Trusted version exists; compare local files against this version

For `latest` structure:
- Uses the current remote contents if they exist
- Trusts manifest if available and valid

For `archive` structure:
- Finds the latest archive version that matches local state
- Must be recent enough (no local changes since that version)
- For `inspect = "file"`, always trusts the version by hashing files
- For `inspect = "manifest"`, only trusts if manifest is valid

## Stage 2: Building the Plan

The second stage compares local and remote state to determine which files need to be uploaded, removed, or updated.

```{mermaid}
flowchart TD
    Start([Start: .dsl_get_plan]) --> GetFilenames[Get file lists<br/>fn_source & fn_dest]
    
    GetFilenames --> Strategy{send_strategy?}
    
    Strategy -->|upload-all| UploadAll[fn_add = all local files<br/>fn_rm = none]
    Strategy -->|upload-missing| UploadMissing[fn_add = local files not on remote<br/>fn_rm = none]
    Strategy -->|sync-diff| SyncDiff[Compare hashes<br/>fn_add = new + changed<br/>fn_rm = deleted]
    Strategy -->|sync-purge| SyncPurge[Compare hashes<br/>fn_add = all local<br/>purge = true]
    
    UploadAll --> BuildActions[Translate to actions]
    UploadMissing --> BuildActions
    SyncDiff --> BuildActions
    SyncPurge --> BuildActions
    
    BuildActions --> CheckCue{send_cue?}
    CheckCue -->|never| SkipSend[Skip upload]
    CheckCue -->|always| ProceedSend[Proceed with upload]
    CheckCue -->|if-change| CheckChanges{Any changes?}
    
    CheckChanges -->|Yes| ProceedSend
    CheckChanges -->|No| SkipSend
    
    ProceedSend --> BuildMetadata[Build metadata updates<br/>- manifest.csv<br/>- VERSION file<br/>- changelog]
    SkipSend --> End
    BuildMetadata --> End([Return plan])
    
    style Start fill:#e1f5e1
    style End fill:#e1f5e1
    style BuildActions fill:#fff4e1
```

**Key Outputs:**

- `fn_add` - Files to upload to the remote
- `fn_rm` - Files to remove from the remote
- `version` - Updated VERSION file content
- `manifest` - Updated manifest.csv content
- `purge` - Whether to purge all remote files before upload
- `ensure_remote_dest_exists` - Whether to create the remote if it doesn't exist
- `is_remote_dest_empty` - Whether the remote will be empty after operations

**Send Strategies:**

Different strategies determine how local and remote files are compared:

- **upload-all** - Upload all local files, ignore remote state
- **upload-missing** - Only upload files that don't exist on remote
- **sync-diff** - Upload new/changed files, remove deleted files
- **sync-purge** - Remove all remote files, then upload all local files

**Inspection Modes:**

The `inspect` parameter controls how projr determines what's on the remote:

- **manifest** - Use remote manifest.csv (fast, requires valid manifest)
- **file** - Download and hash actual files (slower, always accurate)
- **none** - Treat remote as empty (always upload everything)

## Stage 3: Implementing the Plan

The third stage executes the file operations and updates remote metadata.

```{mermaid}
flowchart TD
    Start([Start: .dsl_implement_plan]) --> CheckPurge{purge?}
    
    CheckPurge -->|Yes| PurgeRemote[Remove all files<br/>from remote_dest_full]
    CheckPurge -->|No| AddFiles
    
    PurgeRemote --> AddFiles[Add files from fn_add]
    
    AddFiles --> CheckAdd{fn_add empty?}
    CheckAdd -->|No| CreateIfNeeded{remote exists?}
    CheckAdd -->|Yes| RemoveFiles
    
    CreateIfNeeded -->|No| CreateRemote[Create remote destination]
    CreateIfNeeded -->|Yes| UploadFiles[Upload files to remote]
    
    CreateRemote --> UploadFiles
    UploadFiles --> RemoveFiles[Remove files from fn_rm]
    
    RemoveFiles --> CheckRm{fn_rm empty?}
    CheckRm -->|No| DeleteFiles[Delete files from remote]
    CheckRm -->|Yes| FinalizeRemotes
    
    DeleteFiles --> FinalizeRemotes[Finalize remote states<br/>- Remove empty variant if full exists<br/>- Create empty variant if needed]
    
    FinalizeRemotes --> CheckBoth{Both full & empty exist?}
    CheckBoth -->|Yes| RemoveEmpty[Remove empty remote]
    CheckBoth -->|No| CheckCreate{Create empty needed?}
    
    RemoveEmpty --> UpdateMetadata
    CheckCreate -->|Yes| CreateEmpty[Create empty remote]
    CheckCreate -->|No| UpdateMetadata
    
    CreateEmpty --> UpdateMetadata[Upload metadata files<br/>- manifest.csv<br/>- VERSION file<br/>- CHANGELOG.md]
    
    UpdateMetadata --> End([Complete])
    
    style Start fill:#e1f5e1
    style End fill:#e1f5e1
    style FinalizeRemotes fill:#fff4e1
```

**Key Operations:**

1. **Purge** (if `purge = TRUE`) - Remove all existing files from remote
2. **Add files** - Upload files in `fn_add` to remote destination
3. **Remove files** - Delete files in `fn_rm` from remote destination
4. **Finalize remotes** - Handle empty/full remote variants:
   - For GitHub: manages `-empty.zip` placeholder files
   - For local: manages empty version directories
   - Ensures only one variant exists (full takes priority)
5. **Update metadata** - Upload manifest.csv, VERSION file, and optionally CHANGELOG.md

**Remote Variants:**

For archive structures, projr manages two variants of each version:

- **Full remote** (`remote_dest_full`) - Contains actual files (e.g., `output-v0.0.1.zip`)
- **Empty remote** (`remote_dest_empty`) - Placeholder when no files exist (e.g., `output-v0.0.1-empty.zip`)

This allows projr to distinguish between:
- "Version never uploaded" (neither variant exists)
- "Version uploaded but empty" (empty variant exists)
- "Version uploaded with files" (full variant exists)

Only one variant exists at a time. When files are added to an empty remote, the empty variant is deleted and the full variant is created.

## Complete Workflow

Here's how all three stages work together during a build:

```{mermaid}
flowchart TD
    Start([Production Build]) --> Loop{For each<br/>destination}
    
    Loop --> Stage1[Stage 1: Get Remotes<br/>.dsl_get_remotes]
    
    Stage1 --> Stage2[Stage 2: Build Plan<br/>.dsl_get_plan]
    
    Stage2 --> CheckCue{send_cue<br/>allows?}
    CheckCue -->|No| Loop
    CheckCue -->|Yes| Stage3[Stage 3: Implement Plan<br/>.dsl_implement_plan]
    
    Stage3 --> Loop
    Loop --> End([Build Complete])
    
    style Start fill:#e1f5e1
    style End fill:#e1f5e1
    style Stage1 fill:#e3f2fd
    style Stage2 fill:#fff9c4
    style Stage3 fill:#f3e5f5
```

**Build Integration:**

The destination send process is triggered during post-build operations:

1. **Pre-build** - Clear directories, hash input files
2. **Build** - Render documents and scripts
3. **Post-build** - Hash output files, commit to git
4. **Destination Send** - Upload to remotes (this workflow)
5. **Dev Version Bump** - Increment to next dev version

For each configured destination in `_projr.yml`:
- Check if `send_cue` condition is met (always, if-change, never)
- Get remotes and determine comparison version
- Build upload/removal plan based on `send_strategy` and `send_inspect`
- Execute plan if approved by cue logic

## Configuration Parameters

The workflow behavior is controlled by parameters in `_projr.yml`:

**structure** (where versions are stored):
- `archive` - Create separate versions (v0.0.1, v0.0.2, etc.)
- `latest` - Overwrite same location each time

**send_cue** (when to upload):
- `always` - Upload every build
- `if-change` - Only upload if content changed
- `never` - Skip upload

**send_strategy** (how to upload):
- `sync-diff` - Upload changed/new files, remove deleted files
- `sync-purge` - Remove all, then upload all
- `upload-all` - Upload all files
- `upload-missing` - Only upload missing files

**send_inspect** (how to check remote):
- `manifest` - Use manifest.csv (fast)
- `file` - Hash actual files (slower, accurate)
- `none` - Treat as empty (always upload)

## Examples

### Example 1: First Upload (No Remote Exists)

**Configuration:**
```yaml
structure: archive
send_cue: if-change
send_strategy: sync-diff
send_inspect: manifest
```

**Workflow:**
1. **Get Remotes**: `version_comp = NULL` (no remote exists)
2. **Build Plan**: `fn_add = all local files`, `fn_rm = []`
3. **Implement**: Create remote, upload all files, write manifest

### Example 2: Incremental Update (Files Changed)

**Configuration:**
```yaml
structure: archive
send_cue: if-change
send_strategy: sync-diff
send_inspect: manifest
```

**Workflow:**
1. **Get Remotes**: `version_comp = "0.0.1"` (last upload version)
2. **Build Plan**: Compare against v0.0.1 manifest
   - `fn_add = [new_file.txt, changed_file.txt]`
   - `fn_rm = [deleted_file.txt]`
3. **Implement**: Upload 2 files, remove 1 file, update manifest

### Example 3: No Changes (Skip Upload)

**Configuration:**
```yaml
structure: archive
send_cue: if-change
send_strategy: sync-diff
send_inspect: manifest
```

**Workflow:**
1. **Get Remotes**: `version_comp = "0.0.1"`
2. **Build Plan**: Compare against v0.0.1 manifest
   - `fn_add = []`, `fn_rm = []` (no changes)
3. **Implement**: Skip upload (cue not met)

### Example 4: Latest Structure (Always Overwrite)

**Configuration:**
```yaml
structure: latest
send_cue: always
send_strategy: sync-purge
send_inspect: none
```

**Workflow:**
1. **Get Remotes**: `version_comp = NULL` (inspect = none)
2. **Build Plan**: `purge = true`, `fn_add = all local files`
3. **Implement**: Remove all remote files, upload all local files

## Trust and Manifest Validation

The workflow maintains a trust system to ensure accurate synchronization:

**Trusted Versions** (can use manifest for comparison):
- Uploaded with `sync-diff` or `sync-purge` strategy
- Has valid manifest.csv on remote
- No local changes since that version

**Untrusted Versions** (must hash actual files):
- Uploaded with `upload-all` or `upload-missing` strategy
- No manifest.csv on remote
- Manifest doesn't match local manifest for that version

The VERSION file tracks trust status with an asterisk:
```
project: 0.0.2
output: 0.0.1*  # Asterisk means untrusted
raw-data: 0.0.1  # No asterisk means trusted
```

When a version is untrusted, projr falls back to hashing actual files (`inspect = "file"`) even when `inspect = "manifest"` is configured.

## Performance Considerations

**Fast Configuration** (recommended default):
```yaml
structure: archive
send_cue: if-change
send_strategy: sync-diff
send_inspect: manifest
```

This is fastest because:
- Archives preserve version history
- `if-change` skips unchanged content
- `sync-diff` uploads only differences
- `manifest` avoids downloading files

**Simple Configuration** (easier to understand):
```yaml
structure: latest
send_cue: always
send_strategy: upload-all
send_inspect: none
```

This is simpler but slower:
- Always uploads all files
- No comparison logic needed
- No manifest dependency

## Debugging

Enable debug output to see workflow details:

```{r eval=FALSE}
# Set debug output level
Sys.setenv(PROJR_OUTPUT_LEVEL = "debug")

# Run production build
projr_build_patch()

# View detailed log
projr_log_view()
```

Debug output shows:
- Remote configuration (id, structure, strategy, inspect)
- Remote existence checks
- Upload plan (files to add/remove)
- File operation details
- Metadata updates

## See Also

- `vignette("send-to-remotes")` - User guide for configuring remote destinations
- `vignette("build-process")` - Complete build process overview
- `?projr_yml_dest_add_github` - GitHub remote configuration
- `?projr_yml_dest_add_local` - Local remote configuration