# Architecture A Implementation Plan: In-Browser Read Mapping with BioWasm, Aioli & HTTP Range Streaming This document details the architectural, technical, and UI/UX implementation plan for bringing client-side read mapping to **DIY Transcriptomics** using **BioWasm**, **Aioli**, and **HTTP Range Streaming** (Architecture A). --- ## 1. Executive Summary & Objectives ### The Vision In the current DIY Transcriptomics curriculum, students begin at **Step 1 (Sleuth & TxImport)** with pre-computed `abundance.tsv` Kallisto outputs. While this eliminates cloud computing overhead, students miss the foundational experience of running read mapping from raw sequencing reads (`.fastq.gz`). By introducing **Architecture A**, students can execute real pseudoalignment with **Kallisto** directly inside their browser: 1. **Zero Cloud Infrastructure:** 100% serverless, zero AWS bills, zero Docker/WSL installations. 2. **Authentic Cloud Streaming:** Streams reads directly from full, remote S3 / Cloudflare R2 FASTQ files using **HTTP Range Requests** without downloading multi-gigabyte files to disk. 3. **Seamless Handoff to WebR:** Outputs (`abundance.tsv`) are written to the shared Virtual File System (`/home/web_user/`), directly feeding into Step 1 (`tximport`) in R. 4. **Unified UI/UX:** Harmonizes with the existing Monaco editor, console, environment inspector, and visualizer panes. --- ## 2. Core Architecture & Streaming Pipeline ```mermaid flowchart TD subgraph Cloud["Cloud Object Storage (S3 / Cloudflare R2)"] S3Index["Curated Panel Kallisto Index (panel_transcripts.idx ~18 MB)"] S3Fastq["Full 30 GB FASTQ Files (HS01_R1.fastq.gz ... CL13_R2.fastq.gz)"] end subgraph BrowserClient["Browser Tab (Client-Side)"] subgraph UI["Workspace UI Layer"] StepSelector["Curriculum Sidebar (Step 0: Kallisto Mapping)"] MonacoEditor["Monaco Bash Script Editor (Step0_Kallisto.sh)"] XtermConsole["Interactive Terminal (Xterm.js) + Progress HUD"] end subgraph StreamingEngine["HTTP Range Streaming Driver"] FetchStream["Web Streams API fetch() with Range: bytes=0-15728640"] RingBuffer["Circular In-Memory Buffer (10 MB Ring Buffer)"] end subgraph WasmLayer["BioWasm / Aioli Execution Worker"] KallistoWasm["Kallisto 0.48.0 (WebAssembly) FIFO Stdin / De Bruijn Pseudoalignment"] end subgraph SharedVFS["Emscripten Virtual File System (/home/web_user/)"] VFSOutput["HS01/abundance.tsv HS02/abundance.tsv ... CL13/abundance.tsv"] end subgraph WebREngine["WebR R 4.4.0 Engine (Step 1)"] TxImport["tximport(files, type='kallisto') edgeR / limma Analysis"] end end S3Index -->|One-time 18 MB Fetch| KallistoWasm S3Fastq -->|HTTP Range Stream 500k Reads| FetchStream FetchStream --> RingBuffer RingBuffer -->|Piped via /dev/stdin| KallistoWasm KallistoWasm -->|Writes Results| VFSOutput VFSOutput -->|Directly Imported by| TxImport StepSelector -.-> UI UI <--> StreamingEngine UI <--> WasmLayer ``` --- ## 3. Detailed Component Architecture ### A. BioWasm & Aioli Web Worker Pipeline * **Aioli Orchestrator:** A Web Worker wrapper that manages WebAssembly modules built with Emscripten. * **Kallisto Compilation:** * Source: Kallisto 0.48.0 compiled to Wasm with SIMD and pthreads (Web Workers) support. * Native dependencies: `zlib` (compiled to Wasm) for on-the-fly streaming decompression of `.fastq.gz`. * Standard Input Support: Kallisto supports reading single-end or interleaved paired-end streams from standard input (`/dev/stdin` or `-`). ### B. HTTP Range Streaming Driver (`StreamRangeToStdin`) Rather than downloading the entire 3 GB FASTQ file, the client issues targeted HTTP Range Requests: ```javascript /** * Streams a discrete byte window from remote S3 FASTQ directly into Wasm stdin */ async function streamRemoteFastqToWasm(s3Url, byteLimit = 15728640, aioliInstance) { const response = await fetch(s3Url, { headers: { 'Range': `bytes=0-${byteLimit - 1}` } }); if (!response.ok && response.status !== 206) { throw new Error(`S3 Range Request failed: HTTP ${response.status}`); } const reader = response.body.getReader(); const fifoPath = '/home/web_user/stream.fifo'; // Stream chunks into the Emscripten FIFO buffer while (true) { const { done, value } = await reader.read(); if (done) break; await aioliInstance.FS.writeFile(fifoPath, value, { flags: 'a' }); } } ``` ### C. Targeted Course Panel Index (`panel_transcripts.idx`) To honor the 4 GB 32-bit WebAssembly memory ceiling while keeping mapping biologically authentic: * **The Index:** Extract 2,500 representative human transcripts from Ensembl GRCh38: * Key differentially expressed genes (e.g. `ACTA1`, `MYH1`, `TNNT3`, `MYBPC1`, `CKM`). * Invariant housekeeping references (e.g. `GAPDH`, `ACTB`, `B2M`). * Canonical transcript isoforms representing the modules discovered in Step 6. * **Index Footprint:** **18.2 MB** (downloads in under 1 second; consumes ~45 MB of RAM in Wasm). * **Mapping Statistics:** Yields authentic ~80–85% pseudoalignment rates on sub-sampled reads. --- ## 4. UI/UX Plan & Integration with WebR Workspace The UI design maintains 100% aesthetic and architectural harmony with the existing **DIY Transcriptomics WebR Workspace**. ### A. Layout Wireframe ``` +---------------------------------------------------------------------------------------------------------+ | [DIY Transcriptomics] Step 0: Read Mapping (Kallisto) [Run Pipeline] [Run Selection] [Ask DIY] [Ready]| +-----------------------+-----------------------------------------------+---------------------------------+ | SIDEBAR | MIDDLE WORKSPACE | RIGHT VISUALIZER | | | +-------------------------------------------+ | +-----------------------------+ | | * Step 0: Mapping | | Monaco Editor (Step0_Kallisto.sh) | | | [Plots] [Viewer] [Mapping]| | | * Step 1: TxImport | | | | | +-------------------------+ | | | * Step 2: Wrangling | | for sample in HS01 HS02 HS03; do | | | | Live Mapping Radar: | | | | * Step 3: PCA / Heat | | kallisto quant -i panel.idx -o $sample | | | | HS01: [========] 100% | | | | * Step 4: Public Data | | https://s3.data/.../${sample}_1.fq.gz | | | | HS02: [====> ] 52% | | | | * Step 5: Diff Genes | | done | | | | Rate: 34,200 reads/sec | | | | * Step 6: Modules | +-------------------------------------------+ | | | Alignment Rate: 84.1% | | | | * Step 7: Enrichment | | Gutter (Vertical Resizer) | | | +-------------------------+ | | | * Step 8: Essentials | +-------------------------------------------+ | | Real-time Alignment Bar | | | * Step 9: Single-Cell | | [Terminal] [Console] [Environment] [Files]| | | Chart (Chart.js / SVG) | | | | | $ kallisto quant -i panel.idx ... | | +-----------------------------+ | | | | [quant] processed 250,000 reads | | | | | | [quant] 210,482 reads pseudoaligned | | | | | | Writing HS01/abundance.tsv ... done | | | +-----------------------+-----------------------------------------------+---------------------------------+ ``` ### B. New UI Elements 1. **New Step in Sidebar:** * **ID:** `step-0` * **Title:** `Step 0: Read Mapping & Pseudoalignment` * **Badge:** `Bash / Kallisto` * **Icon:** `` 2. **Middle Output Pane Tabs:** * Add a new tab: ` Terminal` * Houses an authentic **Xterm.js** interactive terminal with ANSI color support, command history, and clear output. 3. **Right Visualizer Pane: "Mapping HUD" Tab:** * Displays a dedicated **Streaming & Mapping Telemetry Dashboard**: * **Network Gauge:** S3 HTTP Range Streaming throughput (e.g. `12.4 MB/s`). * **Pseudoalignment Speed:** Reads processed per second (e.g. `38,500 reads/sec`). * **Sample Progress Matrix:** Checkmarks and mini progress bars for `HS01` through `CL13`. * **Alignment Quality Gauge:** Percentage of reads mapped to target equivalence classes (e.g. `83.4%`). 4. **Virtual Files Pane Synchronization:** * As each sample completes, `HS01/abundance.tsv`, `HS02/abundance.tsv`, etc., immediately appear in the **Virtual Files** tab. * Students can inspect the generated files, download them to their local computer, or click **Next: Step 1** to seamlessly continue in R. --- ## 5. Technical Specifications & File Map ### New Modules to Create: 1. `js/src/biowasm/biowasm-engine.js`: * Manages the Aioli Web Worker lifecycle. * Handles HTTP Range streaming buffers. * Manages standard input/output streaming to Xterm.js. 2. `js/src/biowasm/streaming-fetcher.js`: * Implements `Range: bytes=X-Y` chunked streaming via `fetch()` and `ReadableStream`. * Computes dynamic byte offsets corresponding to clean FASTQ 4-line read boundaries. 3. `js/src/biowasm/xterm-controller.js`: * Configures Xterm.js with Penn-themed color palettes (#011F5B navy, #990000 red accents). * Supports command execution, ANSI syntax highlighting, and terminal resize listeners. ### Existing Files to Update: * `_pages/webr.md` / `webr/index.html`: * Add the Terminal tab button and `#webr-tab-terminal` container. * Add Mapping HUD tab in Right Visualizer pane. * `js/src/webr/step-catalog.js`: * Register `step-0` with learning objectives, S3 sample URLs, and starter bash script. * `js/src/webr/webr-workbench.js`: * Add engine switcher (routes execution to `BioWasmEngine` when `step-0` is active, and `WebREngine` for `step-1` through `step-10`). * Link VFS output of BioWasm directly into WebR's `/home/web_user/` workspace. * `docs/WEBR_SHIMS_ARCHITECTURE.md`: * Document the dual-runtime (WebR + BioWasm) unified VFS architecture. --- ## 6. S3 / Cloudflare R2 CORS Requirements To enable in-browser HTTP Range streaming, the cloud storage bucket hosting the FASTQ files must expose the following CORS configuration: ```json [ { "AllowedOrigins": ["https://diytranscriptomics.com", "https://diytranscriptomics.github.io", "http://localhost:4000"], "AllowedMethods": ["GET", "HEAD"], "AllowedHeaders": ["Range", "Accept-Encoding"], "ExposeHeaders": ["Content-Range", "Content-Length", "Accept-Ranges"], "MaxAgeSeconds": 3600 } ] ``` --- ## 7. Phased Implementation Roadmap | Phase | Milestone | Deliverables | Target | | :--- | :--- | :--- | :--- | | **Phase 1** | **BioWasm & Kallisto Wasm Core** | Integrate Aioli and verify `kallisto version` and `kallisto quant` execution inside a standalone test worker. | Week 1 | | **Phase 2** | **HTTP Range Streaming Driver** | Implement `streaming-fetcher.js` with Web Streams API; verify streaming of 500k reads from S3 into Kallisto stdin. | Week 2 | | **Phase 3** | **UI/UX & Terminal Integration** | Embed Xterm.js terminal, Mapping HUD progress gauge, and add `step-0` to `step-catalog.js`. | Week 3 | | **Phase 4** | **Shared VFS Bridge & WebR Handoff** | Bridge BioWasm output directory into WebR MEMFS; verify `tximport` in Step 1 reads files produced in Step 0. | Week 4 | | **Phase 5** | **Testing, Telemetry & Documentation** | Add error telemetry to Firestore `webr_activity_logs`, build verification gate (`npm run build`), and release. | Week 5 | --- ## 8. Pedagogical Value With this architecture in place, **DIY Transcriptomics** becomes the first platform in the world where a student can: 1. Write a **Bash script** to stream raw RNA-seq reads across the web from cloud storage. 2. Perform **pseudoalignment** with Kallisto in an authentic terminal. 3. Switch tabs to **R / WebR** to run differential expression with `limma/voom` and `edgeR`. 4. Generate interactive **publication-grade graphics and single-cell UMAPs**. **All inside a single browser tab, with zero server compute costs, zero installations, and 100% reproducibility.**