Destination Send Workflow

Overview

When you run a production build (projr_build_patch(), projr_build_minor(), or projr_build_major()), projr can automatically send artifacts to configured remote destinations. This process is controlled by the .dest_send_label() function in R/dest-send-label.R, which orchestrates three distinct stages:

1. Getting the Remotes - Resolve remote locations and metadata
2. Building the Plan - Determine which files need to be added, removed, or updated
3. Implementing the Plan - Execute the file operations and update remote metadata

Each stage builds upon the previous one to ensure efficient and accurate synchronization between your local project and remote destinations.

Stage 1: Getting the Remotes

The first stage resolves remote locations and determines what versions exist on the remote destination.

flowchart TD
    Start([Start: .dsl_get_remotes]) --> GetPre[Get remote_pre<br/>Base remote location]
    GetPre --> GetDestFull[Get remote_dest_full<br/>Full version destination]
    GetDestFull --> GetDestEmpty[Get remote_dest_empty<br/>Empty version destination]
    GetDestEmpty --> GetVersionComp[Determine version_comp<br/>Version to compare against]
    
    GetVersionComp --> CheckInspect{inspect == 'none'?}
    CheckInspect -->|Yes| ReturnNull[version_comp = NULL]
    CheckInspect -->|No| CheckStructure{structure?}
    
    CheckStructure -->|latest| LatestLogic[Check if remote exists<br/>Trust based on manifest]
    CheckStructure -->|archive| ArchiveLogic[Find latest acceptable<br/>archive version]
    
    LatestLogic --> GetComp[Get remote_comp<br/>Remote for comparison]
    ArchiveLogic --> GetComp
    ReturnNull --> GetComp
    
    GetComp --> End([Return remote list])
    
    style Start fill:#e1f5e1
    style End fill:#e1f5e1
    style GetVersionComp fill:#fff4e1

Key Outputs:

• remote_pre - The base remote location (e.g., GitHub release tag, local directory path)
• remote_dest_full - Where files will be uploaded (full version, not empty)
• remote_dest_empty - Empty version variant (for GitHub empty placeholders)
• version_comp - Version to compare against (NULL if no trusted version available)
• remote_comp - The actual remote object used for comparison

Version Comparison Logic:

The version_comp determines which remote version can be trusted for comparisons:

• NULL - No trusted version available; treat remote as empty (upload everything)
• Version string - Trusted version exists; compare local files against this version

For latest structure: - Uses the current remote contents if they exist - Trusts manifest if available and valid

For archive structure: - Finds the latest archive version that matches local state - Must be recent enough (no local changes since that version) - For inspect = "file", always trusts the version by hashing files - For inspect = "manifest", only trusts if manifest is valid

Stage 2: Building the Plan

The second stage compares local and remote state to determine which files need to be uploaded, removed, or updated.

flowchart TD
    Start([Start: .dsl_get_plan]) --> GetFilenames[Get file lists<br/>fn_source & fn_dest]
    
    GetFilenames --> Strategy{send_strategy?}
    
    Strategy -->|upload-all| UploadAll[fn_add = all local files<br/>fn_rm = none]
    Strategy -->|upload-missing| UploadMissing[fn_add = local files not on remote<br/>fn_rm = none]
    Strategy -->|sync-diff| SyncDiff[Compare hashes<br/>fn_add = new + changed<br/>fn_rm = deleted]
    Strategy -->|sync-purge| SyncPurge[Compare hashes<br/>fn_add = all local<br/>purge = true]
    
    UploadAll --> BuildActions[Translate to actions]
    UploadMissing --> BuildActions
    SyncDiff --> BuildActions
    SyncPurge --> BuildActions
    
    BuildActions --> CheckCue{send_cue?}
    CheckCue -->|never| SkipSend[Skip upload]
    CheckCue -->|always| ProceedSend[Proceed with upload]
    CheckCue -->|if-change| CheckChanges{Any changes?}
    
    CheckChanges -->|Yes| ProceedSend
    CheckChanges -->|No| SkipSend
    
    ProceedSend --> BuildMetadata[Build metadata updates<br/>- manifest.csv<br/>- VERSION file<br/>- changelog]
    SkipSend --> End
    BuildMetadata --> End([Return plan])
    
    style Start fill:#e1f5e1
    style End fill:#e1f5e1
    style BuildActions fill:#fff4e1

Key Outputs:

• fn_add - Files to upload to the remote
• fn_rm - Files to remove from the remote
• version - Updated VERSION file content
• manifest - Updated manifest.csv content
• purge - Whether to purge all remote files before upload
• ensure_remote_dest_exists - Whether to create the remote if it doesn’t exist
• is_remote_dest_empty - Whether the remote will be empty after operations

Send Strategies:

Different strategies determine how local and remote files are compared:

• upload-all - Upload all local files, ignore remote state
• upload-missing - Only upload files that don’t exist on remote
• sync-diff - Upload new/changed files, remove deleted files
• sync-purge - Remove all remote files, then upload all local files

Inspection Modes:

The inspect parameter controls how projr determines what’s on the remote:

• manifest - Use remote manifest.csv (fast, requires valid manifest)
• file - Download and hash actual files (slower, always accurate)
• none - Treat remote as empty (always upload everything)

Stage 3: Implementing the Plan

The third stage executes the file operations and updates remote metadata.

flowchart TD
    Start([Start: .dsl_implement_plan]) --> CheckPurge{purge?}
    
    CheckPurge -->|Yes| PurgeRemote[Remove all files<br/>from remote_dest_full]
    CheckPurge -->|No| AddFiles
    
    PurgeRemote --> AddFiles[Add files from fn_add]
    
    AddFiles --> CheckAdd{fn_add empty?}
    CheckAdd -->|No| CreateIfNeeded{remote exists?}
    CheckAdd -->|Yes| RemoveFiles
    
    CreateIfNeeded -->|No| CreateRemote[Create remote destination]
    CreateIfNeeded -->|Yes| UploadFiles[Upload files to remote]
    
    CreateRemote --> UploadFiles
    UploadFiles --> RemoveFiles[Remove files from fn_rm]
    
    RemoveFiles --> CheckRm{fn_rm empty?}
    CheckRm -->|No| DeleteFiles[Delete files from remote]
    CheckRm -->|Yes| FinalizeRemotes
    
    DeleteFiles --> FinalizeRemotes[Finalize remote states<br/>- Remove empty variant if full exists<br/>- Create empty variant if needed]
    
    FinalizeRemotes --> CheckBoth{Both full & empty exist?}
    CheckBoth -->|Yes| RemoveEmpty[Remove empty remote]
    CheckBoth -->|No| CheckCreate{Create empty needed?}
    
    RemoveEmpty --> UpdateMetadata
    CheckCreate -->|Yes| CreateEmpty[Create empty remote]
    CheckCreate -->|No| UpdateMetadata
    
    CreateEmpty --> UpdateMetadata[Upload metadata files<br/>- manifest.csv<br/>- VERSION file<br/>- CHANGELOG.md]
    
    UpdateMetadata --> End([Complete])
    
    style Start fill:#e1f5e1
    style End fill:#e1f5e1
    style FinalizeRemotes fill:#fff4e1

Key Operations:

1. Purge (if purge = TRUE) - Remove all existing files from remote
2. Add files - Upload files in fn_add to remote destination
3. Remove files - Delete files in fn_rm from remote destination
4. Finalize remotes - Handle empty/full remote variants:
   • For GitHub: manages -empty.zip placeholder files
   • For local: manages empty version directories
   • Ensures only one variant exists (full takes priority)
5. Update metadata - Upload manifest.csv, VERSION file, and optionally CHANGELOG.md

Remote Variants:

For archive structures, projr manages two variants of each version:

• Full remote (remote_dest_full) - Contains actual files (e.g., output-v0.0.1.zip)
• Empty remote (remote_dest_empty) - Placeholder when no files exist (e.g., output-v0.0.1-empty.zip)

This allows projr to distinguish between: - “Version never uploaded” (neither variant exists) - “Version uploaded but empty” (empty variant exists) - “Version uploaded with files” (full variant exists)

Only one variant exists at a time. When files are added to an empty remote, the empty variant is deleted and the full variant is created.

Complete Workflow

Here’s how all three stages work together during a build:

flowchart TD
    Start([Production Build]) --> Loop{For each<br/>destination}
    
    Loop --> Stage1[Stage 1: Get Remotes<br/>.dsl_get_remotes]
    
    Stage1 --> Stage2[Stage 2: Build Plan<br/>.dsl_get_plan]
    
    Stage2 --> CheckCue{send_cue<br/>allows?}
    CheckCue -->|No| Loop
    CheckCue -->|Yes| Stage3[Stage 3: Implement Plan<br/>.dsl_implement_plan]
    
    Stage3 --> Loop
    Loop --> End([Build Complete])
    
    style Start fill:#e1f5e1
    style End fill:#e1f5e1
    style Stage1 fill:#e3f2fd
    style Stage2 fill:#fff9c4
    style Stage3 fill:#f3e5f5

Build Integration:

The destination send process is triggered during post-build operations:

1. Pre-build - Clear directories, hash input files
2. Build - Render documents and scripts
3. Post-build - Hash output files, commit to git
4. Destination Send - Upload to remotes (this workflow)
5. Dev Version Bump - Increment to next dev version

For each configured destination in _projr.yml: - Check if send_cue condition is met (always, if-change, never) - Get remotes and determine comparison version - Build upload/removal plan based on send_strategy and send_inspect - Execute plan if approved by cue logic

Configuration Parameters

The workflow behavior is controlled by parameters in _projr.yml:

structure (where versions are stored): - archive - Create separate versions (v0.0.1, v0.0.2, etc.) - latest - Overwrite same location each time

send_cue (when to upload): - always - Upload every build - if-change - Only upload if content changed - never - Skip upload

send_strategy (how to upload): - sync-diff - Upload changed/new files, remove deleted files - sync-purge - Remove all, then upload all - upload-all - Upload all files - upload-missing - Only upload missing files

send_inspect (how to check remote): - manifest - Use manifest.csv (fast) - file - Hash actual files (slower, accurate) - none - Treat as empty (always upload)

Examples

structure: archive
send_cue: if-change
send_strategy: sync-diff
send_inspect: manifest

structure: archive
send_cue: if-change
send_strategy: sync-diff
send_inspect: manifest

structure: archive
send_cue: if-change
send_strategy: sync-diff
send_inspect: manifest

structure: latest
send_cue: always
send_strategy: sync-purge
send_inspect: none

Trust and Manifest Validation

The workflow maintains a trust system to ensure accurate synchronization:

Trusted Versions (can use manifest for comparison): - Uploaded with sync-diff or sync-purge strategy - Has valid manifest.csv on remote - No local changes since that version

Untrusted Versions (must hash actual files): - Uploaded with upload-all or upload-missing strategy - No manifest.csv on remote - Manifest doesn’t match local manifest for that version

The VERSION file tracks trust status with an asterisk:

project: 0.0.2
output: 0.0.1*  # Asterisk means untrusted
raw-data: 0.0.1  # No asterisk means trusted

When a version is untrusted, projr falls back to hashing actual files (inspect = "file") even when inspect = "manifest" is configured.

Performance Considerations

Fast Configuration (recommended default):

structure: archive
send_cue: if-change
send_strategy: sync-diff
send_inspect: manifest

This is fastest because: - Archives preserve version history - if-change skips unchanged content - sync-diff uploads only differences - manifest avoids downloading files

Simple Configuration (easier to understand):

structure: latest
send_cue: always
send_strategy: upload-all
send_inspect: none

This is simpler but slower: - Always uploads all files - No comparison logic needed - No manifest dependency

Debugging

Enable debug output to see workflow details:

# Set debug output level
Sys.setenv(PROJR_OUTPUT_LEVEL = "debug")

# Run production build
projr_build_patch()

# View detailed log
projr_log_view()

Debug output shows: - Remote configuration (id, structure, strategy, inspect) - Remote existence checks - Upload plan (files to add/remove) - File operation details - Metadata updates

See Also

• vignette("send-to-remotes") - User guide for configuring remote destinations
• vignette("build-process") - Complete build process overview
• ?projr_yml_dest_add_github - GitHub remote configuration
• ?projr_yml_dest_add_local - Local remote configuration