# DIY Transcriptomics — Course Maintenance & Content Update Guide This guide provides instructors, teaching assistants, and developers with standard operating procedures for updating, adding, or modifying course content (**lectures, labs, R scripts, datasets, and AI models**) while preserving site stability, WebR browser compatibility, and AI alignment. --- ## 1. System Architecture & Interdependency Map The platform is built on four decoupled, cooperating layers. Understanding how they talk to each other prevents accidental regressions: ```mermaid flowchart TD subgraph ContentLayer ["1. Content & Static Pages"] LecMarkdown["_lectures/*.markdown
(Vimeo IDs, slides, notes)"] LabMarkdown["_labs/*.markdown
(Submission types, tasks)"] RScripts["Code/files/*.R
(Canonical Step scripts)"] Datasets["Data/files/*
(TSV, counts, myDGEList)"] end subgraph WebRLayer ["2. In-Browser WebR Workspace"] StepCat["js/src/webr/step-catalog.js
(Required files, packages, setup)"] LabCat["js/src/webr/lab-catalog.js
(Skeletons, tasks, related steps)"] WebREngine["js/src/webr/webr-engine.js
(Tier 1 WASM & Tier 2 Shims)"] Workspace["js/src/webr/webr-workbench.js
(Editor, VFS, RStudio Project Zip)"] end subgraph AILayer ["3. AI & Knowledge Base (Gemini)"] Transcripts["./my_transcripts/*.vtt
(Vimeo video transcripts)"] UploadScript["scripts/upload_transcripts.py"] FirestoreKB["Firestore: lecture_knowledge & global_knowledge"] VectorDB["Firestore: /course_knowledge_embeddings
(gemini-embedding-001)"] CloudFunctions["functions/index.js
(Ask DIY, Critics, Team Planner)"] end subgraph BuildLayer ["4. Verification & Deployment Gate"] VerifyScript["scripts/verify-webr-architecture.js"] Esbuild["esbuild (JS Bundles)"] Jekyll["Jekyll (Static HTML & search_index.json)"] FirebaseDeploy["Firebase Hosting & Cloud Functions"] end LecMarkdown -->|Jekyll Build| Jekyll LabMarkdown -->|Jekyll Build| Jekyll RScripts -->|Dynamic Fetch ?v=time| Workspace Datasets -->|VFS Mounting / home / web_user| WebREngine StepCat -->|Guides| Workspace LabCat -->|Guides| Workspace WebREngine -->|Execution| Workspace Transcripts -->|upload_transcripts.py| FirestoreKB RScripts -.->|Admin Sync| FirestoreKB FirestoreKB -->|Cloud Function: indexKnowledgeBase| VectorDB VectorDB -->|Semantic RAG| CloudFunctions StepCat -->|Validates| VerifyScript WebREngine -->|Validates| VerifyScript VerifyScript -->|npm run build| Esbuild Esbuild -->|npm run build| Jekyll Jekyll --> FirebaseDeploy ``` --- ## 2. Core Invariants (The "Do No Harm" Rules) Before making any changes, review these core invariants: > [!IMPORTANT] > 1. **Zero Path Assumptions in R Scripts:** Scripts in `/Code/files/` must continue to use relative paths (e.g. `read_tsv("studydesign.txt")`, `load("myDGEList")`). WebR mounts files directly into `/home/web_user/` (the working directory). Hardcoded local file paths like `~/Downloads/...` will break both the WebR workspace and the exported RStudio projects. > 2. **Package Tier Isolation:** Never declare WASM-native CRAN packages (`tidyverse`, `plotly`, `gt`, `DT`, `pheatmap`, etc.) in `desktopShimmed` or `webr_shims$desktop_pkgs`. Never assign their functions directly to `.GlobalEnv`. Always declare them in `requiredPackages` in [`step-catalog.js`](file:///Users/danielbeiting/github/DIYtranscriptomics.github.io/js/src/webr/step-catalog.js). > 3. **Verification Gate Must Pass:** Every commit involving code or catalogs must successfully pass `npm run build && bundle exec jekyll build` before deployment. > 4. **AI Sync After Video or Script Changes:** Whenever you update lecture videos or canonical analysis steps, you must refresh transcripts and trigger the one-click vector re-indexing so "Ask DIY" does not hallucinate outdated code or old study designs. --- ## 3. How to Update Existing Course Materials ### Scenario A: Modifying an Existing R Analysis Script * **Files:** [`Code/files/Step*.R`](file:///Users/danielbeiting/github/DIYtranscriptomics.github.io/Code/files/) 1. **Edit the Script:** Open and edit the canonical `.R` script (e.g., [`Step1_TxImport.R`](file:///Users/danielbeiting/github/DIYtranscriptomics.github.io/Code/files/Step1_TxImport.R)). 2. **Dynamic Cache Busting:** The WebR Workspace fetches scripts using `?v=${Date.now()}`. All students starting a fresh session will instantly receive your new code. 3. **Student Edits Notice:** Students who previously edited and saved that step in local storage will see their saved draft. They can click **"Revert to Original"** in the code editor toolbar to load your latest canonical script. 4. **Run Verification:** Run `npm test` to ensure your updated script passes R syntax validation. ### Scenario B: Updating an Existing Course Dataset * **Files:** [`Data/files/*`](file:///Users/danielbeiting/github/DIYtranscriptomics.github.io/Data/files/) 1. **Replace the File:** Place the updated file in `/Data/files/` with the exact same filename (e.g., `studydesign.txt`, `myDGEList`). 2. **Verify Column Names:** If you rename columns in `studydesign.txt` (e.g., `treatment` $\to$ `condition`): - Update all downstream R scripts in `Code/files/` that reference those columns. - Update `setupCode` in [`js/src/webr/step-catalog.js`](file:///Users/danielbeiting/github/DIYtranscriptomics.github.io/js/src/webr/step-catalog.js) for steps that initialize variables like `sampleID <- studyDesign$sampleID`. 3. **RStudio Project Exporter:** The exporter automatically reads the new file from `/home/web_user/` and packages it into the `.zip` bundle. ### Scenario C: Updating Lecture Videos & Transcripts * **Files:** [`_lectures/*.markdown`](file:///Users/danielbeiting/github/DIYtranscriptomics.github.io/_lectures/), [`scripts/upload_transcripts.py`](file:///Users/danielbeiting/github/DIYtranscriptomics.github.io/scripts/upload_transcripts.py) 1. **Update Markdown:** Edit the frontmatter `vimeo_ids` array in the corresponding `_lectures/YYYY-MM-DD-Lecture_NN.markdown` file: ```yaml vimeo_ids: - "987654321" # New Vimeo ID ``` 2. **Export Transcripts:** Download the `.vtt` or `.srt` subtitle files from Vimeo. Name the file with its Vimeo ID (e.g., `my_transcripts/987654321.vtt`). 3. **Upload Transcripts to Firestore:** ```bash python scripts/upload_transcripts.py ``` 4. **Re-index Embeddings:** Follow [Section 6](#6-syncing-ai-models--vector-embeddings) to re-index. --- ## 4. How to Add New Content ### Adding a New Lecture Step (e.g., Step 11) 1. **Add Canonical Script:** Place `Step11_NewTopic.R` into [`Code/files/`](file:///Users/danielbeiting/github/DIYtranscriptomics.github.io/Code/files/). 2. **Register in Step Catalog:** Open [`js/src/webr/step-catalog.js`](file:///Users/danielbeiting/github/DIYtranscriptomics.github.io/js/src/webr/step-catalog.js) and add the step entry: ```javascript 'step-11': { id: 'step-11', title: 'Step 11: Advanced Spatial Omics', shortTitle: 'Step 11: Spatial', category: 'Specialized Omics', icon: 'fas fa-map-marked-alt', scriptUrl: '/Code/files/Step11_NewTopic.R', description: 'Explore spatial transcriptomics workflows and spot deconvolution.', learningObjectives: [ 'Load spatial count matrices and coordinates', 'Perform spot deconvolution with reference scRNA-seq' ], requiredPackages: ['tidyverse', 'matrixStats'], requiredFiles: [ { name: 'spatial_coords.tsv', url: '/Data/files/spatial_coords.tsv' } ], setupCode: ` # Code to initialize pre-requisite variables ` } ``` 3. **Create Lecture Page:** Add `_lectures/YYYY-MM-DD-Lecture_11.markdown` with appropriate frontmatter (`layout: project`, `vimeo_ids`, `title`, etc.). 4. **Verify Build:** Run `npm run build && bundle exec jekyll build`. ### Adding a New Data Lab (e.g., Lab 9) 1. **Create Lab Markdown:** Add `_labs/YYYY-MM-DD-week_09.markdown` with frontmatter: ```yaml --- title: 'Spatial Transcriptomics Challenge' subtitle: 'Lab 9 • Spring 2027' date: 2027-04-15 00:00:00 submission_type: image # 'image' for Figure Critic or 'code' for Code Critic submission_placeholder: 'Paste your spatial analysis code here...' --- ``` 2. **Register in Lab Catalog:** Open [`js/src/webr/lab-catalog.js`](file:///Users/danielbeiting/github/DIYtranscriptomics.github.io/js/src/webr/lab-catalog.js) and add: ```javascript 'lab-9': { id: 'lab-9', pageId: 'week-09', // Must match the slug from _labs filename or permalink week: 'Week 9', title: 'Lab 9: Spatial Analysis Challenge', shortTitle: 'Lab 9: Spatial', category: 'Data Challenge', datasetName: '10x Visium Mouse Brain', icon: 'fas fa-brain', relatedStep: 'step-11', learningObjectives: [ ... ], requiredPackages: ['tidyverse'], requiredFiles: [ ... ], skeletonScript: `# Starter template code for students` } ``` 3. **Configure Figure Specs / Rubrics (If `submission_type: image`):** In the Admin Dashboard (`/admin/`), navigate to the **Knowledge Base** tab $\to$ **Gold Standards** and configure the expected figure criteria (axes, color maps, labels). --- ## 5. WebR Package Tiers & Shims Guidelines When introducing new R code or packages to the course: ```mermaid flowchart TD PkgCheck{"Is the package on CRAN & compiled for WASM?"} PkgCheck -- Yes (Tier 1) --> Tier1Action["1. Add to requiredPackages in step-catalog.js
2. DO NOT add to desktopShimmed
3. DO NOT mask functions in .GlobalEnv"] PkgCheck -- No (Tier 2 / Bioconductor) --> Tier2Check{"Is it already shimmed in webr-engine.js?"} Tier2Check -- Yes --> Tier2Action["Functions (DGEList, cpm, voom, topTable, Seurat, SingleR) work out-of-the-box."] Tier2Check -- No --> NeedsShim["Implement zero-dependency mathematical shim inside webr_shims environment."] ``` ### Tier 1: WebAssembly CRAN Packages (Native WASM) * **Supported Packages:** `tidyverse` (`readr`, `dplyr`, `ggplot2`, `tidyr`, `tibble`, `purrr`, `stringr`), `plotly`, `gt`, `DT`, `pheatmap`, `RColorBrewer`, `cowplot`, `matrixStats`, `jsonlite`, `scales`, `base64enc`, `gplots`. * **Rules:** 1. Simply add to `requiredPackages: ['packageName']` in `step-catalog.js` or `lab-catalog.js`. 2. WebR installs and attaches these directly in browser memory. 3. Interactive HTML output (`gt`, `DT`, `plotly`) automatically routes to the **Viewer** pane via S3 hooks. ### Tier 2: Desktop-Only Dependencies (Curated Shims) * **Supported Packages:** `edgeR`, `limma`, `rhdf5`, `tximport`, `ensembldb`, `EnsDb.Hsapiens.v86`, `biomaRt`, `DropletUtils`, `Seurat`, `SingleR`, `celldex`, `Azimuth`, `SingleCellExperiment`. * **Rules:** 1. Shimmed because they depend on multi-gigabyte Bioconductor annotation databases or uncompiled C++ libraries. 2. All shimmed functions live inside the `webr_shims` search path environment (`pos = 2`), so genuine packages always take precedence if installed. 3. If you write code using an unsupported Bioconductor function (e.g., `DESeq2::DESeq()`), it will throw an `object not found` error unless a shim is written. 4. Refer to [`docs/WEBR_SHIMS_ARCHITECTURE.md`](file:///Users/danielbeiting/github/DIYtranscriptomics.github.io/docs/WEBR_SHIMS_ARCHITECTURE.md) for shim mathematical implementations. --- ## 6. Syncing AI Models & Vector Embeddings "Ask DIY" and the multimodal critics ground their answers in course materials via **Retrieval-Augmented Generation (RAG)**. When content changes, refresh the knowledge base: ```mermaid sequenceDiagram autonumber actor Instructor participant AdminUI as Admin Dashboard (/admin/) participant CloudFn as Cloud Function: indexKnowledgeBase participant Gemini as Gemini API (gemini-embedding-001) participant Firestore as Firestore: /course_knowledge_embeddings Instructor->>AdminUI: Paste updated master scripts into Global Knowledge Instructor->>AdminUI: Click "Re-index All Course Materials" AdminUI->>CloudFn: Trigger indexKnowledgeBase() CloudFn->>CloudFn: Chunk transcripts, slide notes, & scripts (300 words, 50 overlap) CloudFn->>Gemini: Compute 768-D Vector Embeddings Gemini-->>CloudFn: Vectors returned CloudFn->>Firestore: Batch write FieldValue.vector() CloudFn-->>AdminUI: Complete (e.g. 142 chunks indexed across 12 modules) AdminUI-->>Instructor: Green Status Confirmation ``` ### Operational Steps: 1. **Global Master Scripts:** Go to `/admin/` $\to$ **Knowledge Base** tab $\to$ **Global Knowledge (Course Scripts)**. Paste the latest combined R scripts for the course and click **Save Global Knowledge**. 2. **Trigger Re-Indexing:** In the **Admin Knowledge Base** tab, click **"Re-index All Course Materials"**. 3. **Wait for Completion:** Indexing takes ~30–60 seconds. A status banner will report: ```text ✓ Complete! Indexed 148 chunks across 15 modules. ``` 4. **Verification:** Open any lecture or lab in an incognito window, open **Ask DIY**, and ask a question specifically about your new code or dataset (e.g. *"What is the formula we used in Step 4?"*). Verify that the model cites the new material. --- ## 7. Quality Assurance & Pre-Flight Verification Gate Always run the automated verification gate before pushing to GitHub or deploying to Firebase: ```bash # 1. Run WebR architectural checks and bundle ES6 modules npm run build # 2. Run Jekyll static site compiler bundle exec jekyll build ``` ### What `npm run build` Validates Automatically: * **R Syntax Validation:** Runs `parse()` over all embedded setup and template scripts to detect unclosed quotes, brackets, or syntax errors. * **Tier 1 Package Protection:** Confirms that no WASM-native packages were added to `desktopShimmed` or masked in `.GlobalEnv`. * **Catalog Declarations:** Validates that steps requiring `gt`, `DT`, and `plotly` have them in `requiredPackages`. * **ES6 Compilation:** Verifies `esbuild` can bundle `dashboards-controller.js` and `webr-workbench.js` with zero import errors. --- ## 8. Deployment Checklist When your updates are verified locally: ```bash # 1. Deploy static website & JavaScript bundles to Firebase Hosting firebase deploy --only hosting # 2. If you modified functions/index.js (Cloud Functions) firebase deploy --only functions # 3. If you modified firestore.rules (Security Rules) firebase deploy --only firestore:rules ``` ### Post-Deployment Smoke Test (5-Minute Checklist): - [ ] **Home Page & Lectures:** Navigate to `/project/lecture-01`. Confirm video player, slide link, and reading links load. - [ ] **WebR Workspace:** Navigate to `/webr/`. Select **Step 1**, click **Run All**, and confirm counts load without errors. - [ ] **Export Dropdown:** Click **Export $\to$ RStudio Project (.zip)**. Unzip the download, verify `DIY_Transcriptomics.Rproj` is present, and check that `install_packages.R` contains the correct package lists. - [ ] **Data Labs:** Navigate to `/lab/week-03` (or any lab). Switch to **Data Labs** track in WebR. Confirm skeleton script loads. - [ ] **Ask DIY:** Ask *"What packages are loaded in this step?"* and verify response streams smoothly. - [ ] **Admin Dashboard:** Open `/admin/` and verify that student rosters, vector stats, and progress metrics render properly. --- ## 9. Troubleshooting & Rollback Matrix | Issue | Likely Cause | Resolution | | :--- | :--- | :--- | | **Script loads empty or displays `# (Error loading script)` in WebR** | Typo in `scriptUrl` inside `step-catalog.js` or file not committed to `/Code/files/`. | Verify the exact URL path `/Code/files/StepX.R`. Ensure file exists and is accessible. | | **`Error in library(xyz): there is no package called 'xyz'` in WebR** | Package not included in `requiredPackages` in `step-catalog.js`. | Add the package name to `requiredPackages: ['xyz']`. If it's a Bioconductor package, verify it has a shim in `webr-engine.js`. | | **`package or namespace load failed for 'xyz'` (WASM error)** | Package was added to `desktopShimmed` or shadowed in `.GlobalEnv`. | Remove the package from `desktopShimmed` and `desktop_pkgs`. Native WASM packages must use standard `webr::install()`. | | **Student sees an outdated version of an edited script** | Local browser storage is caching an earlier edit under `diy_webr_script_{stepId}`. | Have the student click **"Revert to Original"** in the code editor toolbar, or run `localStorage.clear()` in the browser console. | | **Ask DIY references old course content or outdated variable names** | Vector embeddings in `/course_knowledge_embeddings` have not been re-indexed. | Open `/admin/` $\to$ **Knowledge Base** $\to$ Click **"Re-index All Course Materials"**. | | **Jekyll build fails with Liquid or Rouge syntax error** | Unclosed tag or invalid YAML frontmatter in `_lectures/` or `_labs/`. | Inspect the build log error line number. Check for unescaped colons or quotes in YAML frontmatter. | | **RStudio Project `.zip` throws path errors when student runs script locally** | Script in `/Code/files/` contains hardcoded paths (e.g. `~/Downloads/`) rather than relative filenames. | Change script to use relative paths: `read_tsv("filename.txt")`. The exporter places all mounted VFS files in the project root. | --- *Document Version:* `1.0.0` *Maintained by:* DIY Transcriptomics Course Engineering Team *Associated Architecture Docs:* [`docs/WEBR_SHIMS_ARCHITECTURE.md`](file:///Users/danielbeiting/github/DIYtranscriptomics.github.io/docs/WEBR_SHIMS_ARCHITECTURE.md), [`docs/WEBR_GRAPHICS_AND_VISUALIZATION_PIPELINE.md`](file:///Users/danielbeiting/github/DIYtranscriptomics.github.io/docs/WEBR_GRAPHICS_AND_VISUALIZATION_PIPELINE.md)