Introduction
JARVIS is an open-source autonomous AI operating system for Windows. It reads codebases, sends emails, controls your Android phone, remembers you across months, and re-plans mid-task when you interrupt. This documentation covers everything from installation to advanced configuration.
Live Correction + ask_user
Interrupt mid-run. Missing info? The agent pauses and asks.
Lifelong Memory
Five-tier memory with weighted graph + hybrid RAG.
MCP Native
18,000+ community servers via a single JSON entry.
Dual-Engine Voice
Azure on-device + Vosk fallback, zero cloud round-trip.
4GB RAM
Verified on a 2012 Intel i5-3210M laptop.
Android Control
WebSocket bridge — calls, SMS, GPS, torch, clipboard.
Who is this for?link
JARVIS is built for developers and power users who want an AI that actually does things — not just chats. If you want to:
- Automate real work across your PC, phone, and communications
- Run an AI agent on modest hardware (no GPU required)
- Extend capabilities without touching code (via MCP)
- Keep all memory and data on your own disk
- Use any OpenAI-compatible LLM, including local models
Then this is for you. If you need a polished IDE copilot, use Claude Code or Cursor.
Windows-only. JARVIS relies on Windows APIs for GUI automation, PC monitoring, and the Termux bridge. Mac/Linux support is on the roadmap.
About the voice examples. JARVIS responds in your language. Some voice commands and agent responses in these docs appear in Hinglish (Hindi + English) — that's because the developer is Indian and this is the language he actually uses daily. You can configure JARVIS to respond in any language you prefer.
Quick Start
Get JARVIS running in under five minutes with the 1-click auto-installer. It handles Git, Python 3.11, Node.js, the virtual environment, the WhatsApp bridge, and the global CLI — all automatically, including installing missing prerequisites via winget.
- Windows 10 or 11
- PowerShell (any version, no admin required)
- ~2 GB free disk
- Internet connection
- One LLM API key (Groq, OpenRouter, or OpenAI)
1-Click Install (Recommended)link
Run this single command from PowerShell. The installer auto-installs any missing prerequisites, clones the repo to %USERPROFILE%\Jarvis-OS-Agent, sets up the Python and Node environments, registers the global jarvis CLI, and interactively prompts for your API keys.
Set-ExecutionPolicy Bypass -Scope Process -Force; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; & ([scriptblock]::Create((Invoke-RestMethod -Uri 'https://github.com/thekaifansari01/Jarvis-OS-Agent/releases/latest/download/install.ps1')))
First run takes 5–10 minutes. The installer downloads ~40 Python packages plus a local embedding model (~130 MB). Subsequent runs are near-instant. Windows may show a UAC prompt for the Visual C++ Redistributable — accept it.
Launch JARVISlink
The installer prompts for API keys at the end of setup. Once it finishes, open a new terminal (the current one won't have the updated PATH) and run:
jarvis
You should see the ASCII banner and a status line confirming voice input is enabled. Say "Jarvis" or hit Ctrl+Shift+J to summon it.
Done. The next sections cover each subsystem in detail. If you hit an error, jump to Troubleshooting.
Prefer manual setup? See the Installation page for step-by-step instructions if you need a custom install path, are offline, or the 1-click installer failed.
Installation
Two paths: the 1-click auto-installer (recommended for most users), or manual setup (for custom paths, offline installs, or troubleshooting).
One-command installer (Recommended)link
This is the fastest and recommended path. Run this single command from PowerShell — it handles every dependency automatically, including installing Git, Python 3.11, and Node.js via winget if missing.
Set-ExecutionPolicy Bypass -Scope Process -Force; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; & ([scriptblock]::Create((Invoke-RestMethod -Uri 'https://github.com/thekaifansari01/Jarvis-OS-Agent/releases/latest/download/install.ps1')))
First run takes 5–10 minutes. Windows may show a UAC prompt for the Visual C++ Redistributable — accept it. Subsequent re-runs are near-instant and safe to execute (the installer updates in place without wiping your data).
What the installer doeslink
- Prerequisites — auto-installs Git, Python 3.11, Node.js, and the Visual C++ Redistributable via winget if missing (falls back to direct downloads)
- Repository — clones to
%USERPROFILE%\Jarvis-OS-Agent, or updates in place if already present - Python environment — creates
.venvwith Python 3.11 and installs ~40 packages including CPU-optimized PyTorch - WhatsApp bridge — installs the Node.js Baileys bridge dependencies
- Configuration — copies
.env.exampleto.envand interactively prompts for API keys - Global CLI — registers the
jarviscommand viaSetupRegistry.pyand adds.venv\Scriptsto your user PATH
Safe to re-run. The installer detects existing installs and updates in place without wiping your .env, memory graph, or MCP config.
Manual installation (Advanced)link
Use this only if you need a custom install path, are offline, or the 1-click installer failed. Steps below assume you already have Git, Python 3.10+ with pip, and Node.js 18+ with npm installed and on PATH.
1. Clone the repositorylink
git clone https://github.com/thekaifansari01/Jarvis-OS-Agent.git
cd Jarvis-OS-Agent
2. Create virtual environmentlink
JARVIS installs ~40 Python packages. Always use a virtual environment to avoid conflicts with your global Python.
python -m venv .venv
.\.venv\Scripts\Activate.ps1
If PowerShell blocks the activation script, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned in an elevated terminal.
3. Install Python packageslink
pip install --upgrade pip
pip install -r requirements.txt
pip install mcp
This installs the core runtime plus optional dependencies like Flask (RAG dashboard), psutil (PC monitoring), and Pillow (vision).
4. Install WhatsApp bridgelink
The WhatsApp integration is powered by a Node.js server using Baileys. Skip this step if you don't need WhatsApp.
cd tools/Messanger/whatsapp/BaileysServer
npm install
cd ../../../..
5. Register global CLIlink
This makes the jarvis command available system-wide. Run it from the repo root with the virtual environment active:
python SetupRegistry.py
Verify installationlink
Whether you used the 1-click installer or manual steps, open a new terminal (existing terminals won't have the updated PATH) and run:
jarvis --help
You should see the full CLI help with all available commands. If you see "jarvis is not recognized", close and reopen the terminal, or check Troubleshooting.
Enable MCP (optional)link
To add external tools via the Model Context Protocol, create Data/mcp_servers.json. See the MCP section for full details.
Uninstall
Complete removal of JARVIS with a single PowerShell command. The uninstaller stops all background services, cleans the Windows registry and PATH, removes the global CLI, and deletes the install folder. It also prompts to back up your .env API keys, Lifelong Memory Graph, and MCP config to the Desktop before deleting anything.
- PowerShell (elevated not required, but recommended)
- Close all terminals running JARVIS
- Backup decision ready — uninstaller will ask
Run the uninstallerlink
Open PowerShell and paste this single command. The script downloads fresh from the main branch and runs in-process:
Set-ExecutionPolicy Bypass -Scope Process -Force; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; & ([scriptblock]::Create((Invoke-RestMethod -Uri 'https://github.com/thekaifansari01/Jarvis-OS-Agent/releases/latest/download/uninstall.ps1')))
Backup prompt is default Yes. When the uninstaller finds .env, Data/jarvis_memory/, or Data/mcp_servers.json, it will ask to back them up to Desktop\Jarvis_Backup_yyyyMMdd_HHmmss\. Press Enter to accept the backup, or n to skip (data will be permanently deleted).
What gets removedlink
- Running processes — kills all JARVIS-related Python/Node processes, plus the
jarvis.lockfile - User PATH entries — removes
~/.local/bin, WindowsApps shims, and any entry containingjarvis-os-agent - CLI shims —
jarvis.cmd,jarvis.bat,jarvis.ps1from common install paths - Registry keys —
App Paths\jarvis.*andHKCU:\Software\Classes\jarvis - Install folder —
%USERPROFILE%\Jarvis-OS-Agentincluding.venv,node_modules, and codebase - HuggingFace cache —
~/.cache/huggingface/hub/models--BAAI--bge-small-en-v1.5(~130 MB, optional prompt) - Documents media —
Documents\Jarvis(WhatsApp downloads, shared files) — optional prompt
What gets backed up firstlink
Before deleting the install folder, the uninstaller prompts to copy these to Desktop\Jarvis_Backup_yyyyMMdd_HHmmss\:
| File | What it holds |
|---|---|
.env | All your API keys and provider config |
Data\jarvis_memory\ | Lifelong memory graph, behavioral patterns, RAG index, chat history |
Data\jarvis_indexer\ | Indexer folder list, filters, and settings |
Data\mcp_servers.json | MCP server configs (including any tokens/keys) |
Encrypted tokens are machine-bound. Session tokens (sessions/*.enc) use Windows DPAPI and are not backed up — they won't work on a different machine anyway. You'll need to re-authenticate via jarvis login --all after a fresh install on another PC.
What happens step by steplink
- Stopping processes — scans
Win32_Processfor anything with executable path or command line inside%USERPROFILE%\Jarvis-OS-Agentand kills them (2s wait for handles to release). - Backup prompt — if
.env, memory, or MCP config exists, asks to back up to Desktop (default Yes). - Registry + PATH cleanup — removes User PATH entries matching JARVIS folders or
jarvis-os-agent, deletes CLI shims, and cleansApp Paths+Classes\jarvisregistry keys. - Core folder deletion — uses
Remove-Item -Recurse -Forcewith read-only bit clearing. Falls back tocmd /c rmdir /s /qfor stubborn locked files. - External media prompt — asks whether to also delete
Documents\Jarvis(default No). - Embedding cache prompt — asks to remove
~/.cache/huggingface/hub/models--BAAI--bge-small-en-v1.5(~130 MB, default No). - Success banner — "J.A.R.V.I.S. OS Agent Uninstalled Successfully!"
Troubleshootinglink
Some files in Jarvis-OS-Agent couldn't be deletedadd
python.exe or node.exe). Close all terminals and editors (VS Code, Notepad++) that have files open inside that folder. Then manually delete %USERPROFILE%\Jarvis-OS-Agent via File Explorer — the uninstaller already cleaned everything else.jarvis command still works after uninstalladd
Get-Command jarvis to find the shim location and delete it manually.Windows Defender blocked the scriptadd
Invoke-RestMethod + scriptblock patterns sometimes. Add PowerShell to the exclusion list temporarily, or download the raw script manually and run it with & .\uninstall.ps1.I skipped the backup by mistakeadd
.env and memory graph are unrecoverable — they were stored inside that folder. Next time, accept the backup prompt (default Yes) before proceeding.Environment Configuration
All API keys and configuration live in a single .env file at the repo root. Copy .env.example to .env and fill in the values you need.
- Text editor (VS Code recommended)
- API key from at least one provider (Groq, OpenRouter, Gemini, or Ollama)
- Optional: Deepgram + Tavily keys for voice + search
Required variableslink
| Variable | Purpose |
|---|---|
PRIMARY_API_KEY | Primary LLM provider key (any OpenAI-compatible endpoint) |
PRIMARY_BASE_URL | Endpoint, e.g. https://api.groq.com/openai/v1 |
PRIMARY_MODEL | Model name, e.g. llama-3.3-70b-versatile |
FALLBACK_API_KEY | Fallback provider key (used if primary fails) |
FAST_BRAIN_API_KEY | FastBrain key (can be the same provider) |
ROUTER_API_KEY | Semantic router key |
LTM_EXTRACTION_API_KEY | Long-term memory extraction |
TAVILY_API_KEY | Real-time web search |
DEEPGRAM_API_KEY | Speech-to-text |
Optional variableslink
The following are only needed for the integrations you actually use:
- Voice:
AZURE_SPEECH_KEY,AZURE_SPEECH_REGION,AZURE_KEYWORD_MODEL_PATH - Telegram:
TELEGRAM_API_ID,TELEGRAM_API_HASH - Image generation:
IMAGE_GEN_API_KEY,IMAGE_GEN_ENDPOINT - TTS:
TTS_API_KEY(falls back to Edge TTS if absent)
Using local LLMslink
For 100% local execution, point both providers at your local OpenAI-compatible server:
PRIMARY_API_KEY=ollama
PRIMARY_BASE_URL=http://localhost:11434/v1
PRIMARY_MODEL=llama3.1
FALLBACK_API_KEY=ollama
FALLBACK_BASE_URL=http://localhost:11434/v1
FALLBACK_MODEL=llama3.1
Both PRIMARY_BASE_URL and FALLBACK_BASE_URL must end with /v1. This is the standard OpenAI-compatible route.
Supported providerslink
| Provider | Base URL | Notes |
|---|---|---|
| Groq | https://api.groq.com/openai/v1 | Free tier, very fast |
| OpenRouter | https://openrouter.ai/api/v1 | Many models, one key |
| Regolo | https://api.regolo.ai/v1 | European hosting |
| Gemini | https://generativelanguage.googleapis.com/v1beta/openai | Use OpenAI-compat route |
| Ollama | http://localhost:11434/v1 | Fully local |
| LM Studio | http://localhost:1234/v1 | Fully local, GUI |
| vLLM | http://localhost:8000/v1 | Self-hosted, GPU |
Voice Setup
JARVIS ships with dual-engine wake word detection, premium STT, and low-latency TTS. Everything runs on-device by default — no cloud round-trip for wake word.
- Working microphone connected
- Deepgram API key for STT
- Optional: Azure Speech account (for higher accuracy wake word)
- Optional: TTS API key (Sarvam) — falls back to Edge TTS
Wake word engineslink
Two independent engines, auto-selected on boot:
| Engine | When active | Accuracy |
|---|---|---|
| Azure Custom Keyword | If Data/model/azure/jarvis.table exists | 97%+ |
| Vosk KWS | Fallback if Azure unavailable | ~66% |
Building an Azure wake modellink
This is a one-time 30-minute process. After training, no API key or network call is needed for wake word.
- Create a Foundry resourceGo to Azure Portal → Foundry → Speech. Copy your key and region.
- Train a Custom KeywordOpen Speech Studio → Custom Keyword → Create project (English US).
- Choose your modelBasic (~30 min queue) or Advanced (~48h for better accuracy).
- Download the .table fileExtract the
.tablefrom the zip and save asData/model/azure/jarvis.table. - Restart JARVISAzure engine activates automatically. Check startup logs for confirmation.
No Azure subscription? Vosk works out of the box. It auto-downloads a small model on first run — no config required.
Speech-to-textlink
STT uses Deepgram Nova-2 with a self-healing LTM vocabulary cache. Entity names from your memory graph are auto-synced into Deepgram's vocabulary, so it correctly recognizes names, projects, and jargon over time.
Zero training needed. Add someone to your memory and Deepgram starts recognizing their name within a session.
Text-to-speechlink
Default TTS uses Sarvam for English + Hinglish (low-latency). Falls back to Edge TTS if the API key is missing. First audio typically lands in 0.8–2 seconds.
Memory & RAG
JARVIS has five tiers of memory — from verbatim message history to long-term semantic graphs and hybrid RAG over your documents.
The five tierslink
| # | Tier | What it stores | Retention |
|---|---|---|---|
| 1 | Episodic | Raw messages + execution logs | 15 days (2 raw + 13 summarized) |
| 2 | Semantic | Weighted property graph (facts, people, contacts) | Lifetime, with decay |
| 3 | RAG | Indexed documents via hybrid BM25 + vector | Until deleted |
| 4 | Ephemeral | 10-min TTL artifacts (last screenshot, generated image) | Auto-purged |
| 5 | Behavioral | Patterns extracted from your habits & routines | Lifetime, with decay |
Unified Portal — Knowledge Indexerlink
JARVIS ships a single Unified Portal at one URL that serves all web UIs — Knowledge Indexer, Mobile Bridge, and more — as tabs. Open it after launching:
http://127.0.0.1:8000
Switch to the Knowledge Indexer tab in the portal sidebar. On first boot, the indexer auto-adds your Documents folder. You can add more from the Folders sub-tab — paste any absolute path and click Add & Index.
Internal ports still active. Behind the portal, the Indexer API runs on 127.0.0.1:8080 and the Mobile Bridge on 127.0.0.1:8090. The portal aggregates both — you only need to remember 8000.
Adding a new folderlink
- Open the Unified Portal at
http://127.0.0.1:8000. - Click the Knowledge Indexer tab in the sidebar.
- Go to the Folders sub-tab.
- Paste an absolute path, e.g.
D:/ProjectsorC:\Users\You\Desktop\notes. - Click Add & Index. Progress streams live.
System paths are auto-blocked. Paths like C:\Windows, Program Files, and dev garbage (node_modules, .git, venv) are skipped automatically. Sensitive paths like .ssh and .aws trigger warnings.
Filters & exclusionslink
Open the Filters tab to tune indexing:
- Max file size — default 5 MB. Larger files are skipped.
- Max depth — how deep to recurse.
- Custom ignore patterns — e.g.
*.log,backup_*,node_modules.
Rebuilding the indexlink
If embeddings feel stale or you changed the model:
# Purge the vector index
Remove-Item -Recurse -Force Data\jarvis_memory\rag_chroma_db
Remove-Item Data\jarvis_memory\rag_hashes.json
# Restart JARVIS to rebuild automatically
jarvis
MCP Integration
The Model Context Protocol lets JARVIS plug into 18,000+ community-built tool servers — GitHub, Brave Search, SQLite, Postgres, Notion, Slack, Docker, Puppeteer, and more — via a single JSON file.
- Node.js 18+ (24+ recommended for newest servers)
pip install mcpcompleted- API keys only if the chosen MCP server requires them
How it workslink
When Data/mcp_servers.json exists with valid configs, JARVIS:
- Spawns each configured MCP server as a subprocess
- Performs a JSON-RPC handshake to discover tools
- Converts tool schemas into native function declarations
- Exposes them to the agent as
mcp__<server>__<tool>
One-liner CLI & live hot-reloadlink
Add, remove, test, or toggle MCP servers directly from your terminal. If JARVIS is running, changes are hot-reloaded in the background within 2 seconds with zero restarts:
# Add an NPX or Python MCP server
jarvis mcp add everything npx @modelcontextprotocol/server-everything
# Add a server with environment variables
jarvis mcp add github npx @modelcontextprotocol/server-github -e GITHUB_PERSONAL_ACCESS_TOKEN=ghp_xxx
# List, test, disable, enable, or remove servers live
jarvis mcp list
jarvis mcp test everything
jarvis mcp disable everything
jarvis mcp enable everything
jarvis mcp remove everything
JSON configuration examplelink
You can also edit Data/mcp_servers.json directly (saved changes auto-sync live):
{
"mcpServers": {
"github": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-github"],
"env": {
"GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_xxxxxxxxxxxx"
},
"call_timeout": 60
},
"sqlite": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-sqlite", "C:/data/mydb.sqlite"]
}
}
}
Startup or live hot-reload logs confirm the active tools immediately:
MCP: server 'github' connected (40 tools).
MCP: server 'sqlite' connected (6 tools).
MCP: ready with 46 tools from 2 servers.
Native Windows resolutionlink
JARVIS automatically resolves npx shims on Windows directly to node.exe + npx-cli.js and injects -y automatically — no cmd /c wrapper needed.
Popular serverslink
| Server | What it adds | API key? |
|---|---|---|
| GitHub | Repos, issues, PRs, search | Personal access token |
| Brave Search | Web, news, local search | Free tier (2k/mo) |
| SQLite | Natural-language DB queries | None |
| Postgres | DB queries on live databases | Connection string |
| Notion | Pages, databases, comments | Integration token |
| Slack | Channels, messages, users | Bot token |
| Docker | Container lifecycle | None |
| Puppeteer | Headless browser automation | None |
| Fetch | URL fetch + readability | None |
Skip redundant servers. Filesystem MCP overlaps with native file_operations. Memory MCP overlaps with the built-in 5-tier memory. Puppeteer overlaps with gui_controller. Only add servers that introduce net-new capabilities.
Disabling MCPlink
MCP is opt-in by design. Delete Data/mcp_servers.json or set it to { "mcpServers": {} }. JARVIS boots identically to the pre-MCP version with zero overhead.
Android Bridge
Control your Android phone natively from Windows using Termux and a WebSocket bridge. No SSH required, no cloud relay.
- Android phone with Termux + Termux:API (from F-Droid)
- Both devices on the same WiFi network
- JARVIS running on your PC
Setuplink
- Install Termux and Termux:API on your phone via F-Droid (not Google Play — that version is outdated).
- Connect both devices to the same WiFi. The bridge runs over local network only.
- Open the Unified Portal on your PC:
http://127.0.0.1:8000→ click the Mobile Bridge tab. - Scan the QR code with your phone, or copy the setup command into Termux.
- Approve Termux:API prompts as they appear — SMS, contacts, storage, location.
What you can dolink
| Category | Examples |
|---|---|
| Files | Drag & drop files, receive via Android Share menu |
| Telecom | Make calls, read/send SMS, ring phone |
| Sensors | GPS location, battery level, storage, torch |
| Clipboard | Sync clipboard both ways |
| Photos | Fetch latest photo, screenshot |
Persistent background modelink
The bridge runs a wake-lock that keeps it alive even when your phone is locked. It survives app switches and network changes.
LAN-only. The bridge binds to 127.0.0.1:8090. It is never exposed to the internet. If you need remote control, use the Telegram bot instead.
Telegram Bot
Trigger silent background PC commands from anywhere via Telegram. Perfect for when you're away from your desk.
- Telegram account on your phone
- BotFather bot created (saves token)
- JARVIS stopped for activation
Setuplink
- Create a bot — message @BotFather on Telegram, send
/newbot, follow the prompts, and copy the token. - Stop JARVIS on your PC. The bot activation command requires it to be offline.
- Run activation:
jarvis bot --activate - Paste your BotFather token when prompted.
- Start JARVIS again. Send any message to your bot on Telegram to confirm.
Remote commandslink
Once connected, you can message your bot on Telegram the same way you'd talk to JARVIS at your desk. Examples:
- "Open VS Code"
- "What's my CPU usage?"
- "Summarize my last 3 emails"
- "Take a screenshot and send it"
- "Lock the PC"
Commands run silently in the background — your active desktop session is untouched.
Status & controllink
jarvis bot --status # show live status
jarvis bot --deactivate # revoke & offline the bot
CLI Reference
All commands are available globally after running SetupRegistry.py once.
Stop JARVIS before running memory, login, or reset commands. Running them while JARVIS is active may corrupt state.
Authenticationlink
| Command | Action |
|---|---|
jarvis login --whatsapp | WhatsApp Web QR authentication |
jarvis login --telegram | Telegram phone + OTP |
jarvis login --mail | Gmail OAuth2 |
jarvis login --calendar | Google Calendar OAuth2 |
jarvis login --all | Batch authenticate all integrations |
Logoutlink
| Command | Action |
|---|---|
jarvis logout --whatsapp | Destroy WhatsApp session |
jarvis logout --telegram | Destroy Telegram session |
jarvis logout --mail | Revoke Gmail tokens |
jarvis logout --calendar | Revoke Calendar tokens |
jarvis logout --all | Revoke everything |
MCP server management (live hot-reload)link
| Command | Action |
|---|---|
jarvis mcp list | List all configured MCP servers and their status |
jarvis mcp add <name> <cmd/url> [args...] [-e K=V] | Add or update an MCP server (hot-reloads live if JARVIS is ON) |
jarvis mcp remove <name> | Remove an MCP server and unload its tools immediately |
jarvis mcp enable <name> | Enable a disabled MCP server live |
jarvis mcp disable <name> | Temporarily disable an MCP server without deleting config |
jarvis mcp test <name> | Test connection to a server and list discovered tools |
Remote botlink
| Command | Action |
|---|---|
jarvis bot --activate | Configure & activate Telegram bot (JARVIS must be OFF) |
jarvis bot --deactivate | Revoke & offline the bot (JARVIS must be OFF) |
jarvis bot --status | Show live bot status |
Memory & resetlink
| Command | Action |
|---|---|
jarvis memory --clear | Purge contextual memory (keeps sessions) |
jarvis reset --hard | Factory reset — wipes memory, vectors, sessions |
jarvis --help | Global help |
Tools Reference
The agent chooses from a library of native tools. Every tool is namespaced, permissioned, and observable in the console.
Tool categorieslink
| Category | Examples |
|---|---|
| Software Engineering | repo_map, replace_block, run_python_code, AST lint, create_many |
| Communication | Gmail (send/read), WhatsApp, Telegram, Google Calendar |
| Workspace | File CRUD, recursive scanning, hybrid RAG |
| Mobile | Termux API — calls, SMS, battery, torch, sensors |
| Remote Control | Telegram bot for silent background commands |
| Search & Research | Web search, ArXiv, YouTube transcripts |
| System Automation | App launch/kill, volume, brightness, clipboard |
| GUI Automation | OpenCV edge-vision Set-of-Mark, mouse & keyboard |
| PC Monitoring | CPU, RAM, disk, battery, network, USB |
| Multimodal | Screen analysis, object detection, OCR |
| Image Generation | Text-to-image, image-to-image editing |
| MCP External | 18,000+ community servers |
Architecture
JARVIS is resilient by design. Every layer has a fallback; every provider has a failover; every tool has a watchdog.
Dual-brain routinglink
A semantic router picks the right brain for each command:
| Brain | When | Latency |
|---|---|---|
| FastBrain | Short commands, OS toggles, casual chat | <2s |
| AgenticBrain | Complex engineering, comms, memory, vision | Multi-step |
3-tier fallback chainlink
- Tier 1 — Semantic Router Cloud-based; classifies intent via a fast LLM.
- Tier 2 — FastBrain If the router API times out, FastBrain handles routing.
- Tier 3 — Local rule-based If both fail, keyword + heuristic rules take over.
Provider failoverlink
Every LLM call tries PrimaryProvider first. On failure (timeout, rate limit, 5xx), it retries FallbackProvider — a completely separate OpenAI-compatible endpoint. Configure both in .env.
ServiceWatchdoglink
Monitors and auto-restarts:
- STT Popup, Baileys WhatsApp server
- Mobile Bridge (
:8090), Telegram bot - Indexer Dashboard (
:8080), MCP Manager - Unified Portal (
:8000) — gated on both backends being healthy
Crashes trigger a cooldown backoff. Unauthenticated services are skipped, not restarted endlessly. The Unified Portal will not restart if either backend is down.
Agent Panel
Live Thought Stream
Watch reasoning unfold in real-time with syntax-highlighted markdown.
HITL Consent Gates
Approve, edit, or cancel every WhatsApp, Telegram, or Email before it sends.
Non-Blocking
Panel never interrupts the agent — it's read-only until a decision is required.
Action-Aware Theming
Every tool has its own icon, color, and pulse animation.
Read-only by design. The Agent Panel is not a chat window — you can't type into it. Commands come from voice, Ctrl+Shift+J, or the Telegram bot. The panel observes and, when needed, asks for approval.
When It Appearslink
- Auto-spawns on every agent loop start — voice command,
Ctrl+Shift+J, Telegram bot, MCP tool call - Stays visible while
hide_timeris active (4-second idle cooldown) - Fades out after 4 seconds of no activity — unless awaiting confirmation
- Reappears instantly on next status update
- Never blocks the underlying desktop — it's
Qt.Tool, no taskbar entry, no focus steal - Bypasses the 4-second timer entirely when a confirmation is pending
The panel cannot be "closed" manually while the agent runs — it dismisses itself. To hide permanently, stop JARVIS from the CLI.
The Stateslink
The panel cycles through four visual states, each with its own orb color, status text, and animation:




Anatomy of the Panellink
Seven distinct regions compose the panel, each with a specific function:
1
2
3
4
5
6
7
- 1Status Orb
- 2Status Tag
- 3Detail Pill
- 4Meta Capsule
- 5Thought Stream
- 6Observation Card
- 7Preview Card
| Region | Purpose | Updates |
|---|---|---|
| Status Orb | 24×24 icon reflecting current action | Per action change |
| Status Tag | Human-readable label ("Thinking", "Gmail") | Per action change |
| Detail Pill | Elided tool name (e.g., "whatsapp_send") | Per tool call |
| Meta Capsule | Step counter + token count | Per step |
| Thought Stream | Markdown-rendered reasoning | Throttled at 90ms |
| Observation Card | Collapsible tool result (max 90px) | Per tool result |
| Preview Card | HITL confirmation UI | Per confirmation |
The Status Orb — 16 Action Iconslink
Every tool type has its own visual identity. The orb changes icon and color to reflect what's currently happening:
Cached by RGB bucket + DPR. Static icons (search, python, etc.) are pre-rendered as QPixmaps for near-zero paint cost. Only thinking and idle are re-rendered live because they animate the halo pulse.
Color Palette — Action-Coded Theminglink
The entire panel — border gradient, ambient glow, observation bar, orb — shifts to a per-action color. This palette is defined in Theme.py as ACCENT_PALETTE:
Smooth crossfades. Colors transition over ~1 second at 45ms per tick (18% per step). When the task finishes, the palette drifts back to neutral grey (idle = [130,140,165]).
The Thought Streamlink
The main body of the panel renders the agent's live reasoning as formatted markdown. It supports:
- Fenced code blocks with syntax highlighting — Python, JS, SQL, C++, Go, Rust, Shell (auto-detected regex)
- Tables with alignment detection
- Headings, bold, italic, strikethrough
- Inline code chips and clickable links (open in external browser)
- Bullet + numbered lists with nested indentation
- Blockquotes (nested depth supported)
- Checkbox lists —
[ ]→ ☐,[x]→ ☑


Scroll lock behavior. Auto-scrolls to bottom unless you've manually scrolled up. Scrolling up "locks" the view; new content won't pull you down until the step counter increments. After 5,000 chars, older content is safely trimmed — mid-code-fence and mid-table cuts are auto-healed.
The Observation Cardlink
When a tool returns an observation string, the panel shows a dedicated card below the thought stream:
- Max height: 90px (elided to 155 chars)
- Accent-colored left bar matches current action color
- Auto-hides when next thought arrives or task completes
- Separated from thought stream by a hairline divider

Human-in-the-Loop Confirmationlink
This is the trust anchor. When the agent needs to send an irreversible action — WhatsApp message, Telegram, Email, or destructive file operation — the panel raises a premium confirmation card that cannot be bypassed.
Never bypassed. Even if the user says "just do it" in the voice prompt, email_action, whatsapp_action, and telegram_action always raise the confirmation card. The agent cannot auto-approve its own outgoing messages. This is enforced at the ZMQ boundary, not in the prompt.
Actions that trigger the gate
| Platform | Tool | Irreversible? | Requires Confirm |
|---|---|---|---|
whatsapp_action | Yes — real contact | ✅ | |
| Telegram | telegram_action | Yes — message/call | ✅ |
email_action | Yes — Gmail OAuth | ✅ | |
| File delete | file_operations (delete) | Yes — disk-level | ✅ |
| Terminal | execute_terminal_command (rm -rf) | Yes — blocked/asked | ✅ |
| Calendar | calendar_action (create/delete) | Yes — external event | ✅ |
| Image gen | image_command | Depends on provider | ⚠️ Varies |
Anatomy of the confirmation card
1
2
3
4
5
- 1Platform Badge
- 2Recipient Row
- 3Body Editor
- 4Cancel Button
- 5Send Now Button
Each platform gets its own themed badge color:



What you can edit
| Field | Editable | Persisted to Agent |
|---|---|---|
| Recipient (To:) | ❌ No | — |
| Subject (email) | ✅ Yes | ✅ Yes |
| Body | ✅ Yes | ✅ Yes |
| Attachments | ❌ No | — |
| Platform | ❌ No | — |
Recipients are locked. You can't redirect a message to a different person from the panel — this prevents prompt-injection attacks from hijacking outgoing comms. To change recipient, cancel and re-issue the command.
Keyboard shortcuts during confirmation
Focus steal on arrival. When a confirmation appears, the panel steals focus to the body editor — you can start typing edits immediately. This is the only time the panel takes focus.
The confirmation queue
- If a second confirmation arrives while one is open, it's appended to a FIFO queue
- Once you decide on the first, the next slides in immediately — no flash, no wait
- No hard limit — but in practice agents rarely issue more than 2 in a row
- Queue is in-memory only — a panel restart clears the queue
- Queued confirmations are not auto-approved if ignored — they stay pending
90-second expiry. Each confirmation has a 90-second timeout enforced by the agent (not the panel). If you don't decide in time, the agent autonomously aborts and logs HITL TIMEOUT. Approve fast or re-run the command.
Controls & Shortcutslink
Dragging
- Click and drag anywhere in the top 62px header zone
- Not draggable from buttons, text areas, scrollbars, or links
- Snaps to screen edges — clamped to primary screen bounds
- Position persists across panel show/hide within the session
Resizing
- 9px grab-zone on all four edges
- 22px corner-zone for diagonal resize
- Min size: 440 × 160
- Max size: 1280 × 860
- Live layout — thought stream re-flows as you drag
Reset to default
Auto-hide
- 4-second cooldown after last status update
- Suppressed when awaiting confirmation, dragging, resizing, or hovering
- Fades out over 190ms and slides up 20px
- RGB timer stops on hide — zero background CPU
Architecture — ZMQ Bridgelink
The panel communicates with the agent loop over ZeroMQ sockets. Three independent channels handle the three message types:
| Port | Direction | Protocol | Purpose |
|---|---|---|---|
5555 | Agent → Panel | PUB/SUB | Status updates — self-healing if panel restarts |
5556 | Agent → Panel | PUSH/PULL | Confirmation requests — reliable delivery |
5557 | Panel → Agent | PUSH/PULL | Confirmation decisions — approve/cancel + edits |
Auto-recovery. The listener automatically retries as the opposing socket type on EADDRINUSE — meaning the panel and agent can start in any order and still connect.
Message batching. The listener drains up to 100 status messages per 25ms poll cycle and only emits the latest. Even a firehose of thought tokens never floods Qt's event queue — the panel stays at 60fps.
Performance & Premium Detailslink
| Technique | What it does | Impact |
|---|---|---|
| Island static pixmap cache | Shell (shadows, gradient, shine) pre-rendered once per size + color bucket | 90% paint cost eliminated on static frames |
| Border-only repaints | During RGB crossfades, only the 1px border region is invalidated | Constant 60fps during color transitions |
| Thought throttling | 90ms debounce on markdown rendering | Token streams don't jank |
| Icon pixmap cache | 16 mode icons cached by RGB bucket + DPR | Zero repaint cost for static orbs |
| Markdown 4-layer cache | HTML, line, code-line, highlight all memoized | Renders 50× faster on repeat content |
| Smooth scroll animation | 110ms eased, 0.65 wheel multiplier | Native-feeling, not Qt-jittery |
| Fast-mode resize | During interactive drag, shadows collapse to 1 layer | Resize stays fluid on 4GB RAM |
| Elided labels | Detail, recipient, attachment truncate with … | Never overflow, never reflow |
Troubleshooting the Panellink
Panel doesn't appear when I run a commandadd
ZmqListener loop error). Verify ports 5555–5557 are not firewalled. On restrictive Windows setups, allow python.exe through Windows Defender Firewall.Confirmation card appears but Esc doesn't canceladd
preview_body_edit and preview_subject_input only. If focus leaked elsewhere, use the Cancel button.Panel stuck showing "Step 03" foreveradd
step=0, action=idle.Text in thought stream is unreadableadd
english.ttf, devangri.ttf, or code.ttf in Data/fonts/. Ensure all three fonts are present. The panel falls back to Segoe UI but may render Devanagari incorrectly.Typing Popup
Token Streaming
Text appears as it's generated — markdown heals mid-stream.
TTS Word Sync
Every spoken word highlights in real-time, synced to audio.
Async Previews
YouTube thumbnails, link cards, image cards — all fetched in background.
Smart Self-Dismiss
Fades out after 6s on short responses, stays on long ones.
Read-only surface. The popup doesn't accept input — you talk to JARVIS via the Agent Panel or voice. It's auto-positioned and self-managing.

When It Appearslink
- File-triggered: appears the moment
Data/typing_status.jsonreceives{status: "typing", text: "..."} - Auto-shows on
"typing"or"completed"status change - Re-shows if the file is re-written while the popup is fading out
- Re-renders live as the
textfield grows - Always centered horizontally, 36px from top — unless dragged
- Slides down + fades in (320ms quint-eased entry)
File-driven, not socket-driven. The popup is entirely file-triggered — no ZMQ, no IPC server. If Data/typing_status.json is never written by the agent, the popup will never appear. This makes it safe to run standalone and easy to test manually.
The Response Lifecyclelink
| State | Status Tag | Orb Color | Auto-Behaviors |
|---|---|---|---|
typing | "Jarvis Responding" | Purple #BF5AF2 | Auto-scroll ON, height animates up, idle counter reset |
completed | "Response Ready" | Green #30D158 | Scroll unlocks, final render, idle counter starts |
dismissed | — | — | 250ms fade out, app exits |



Anatomy of the Popuplink
1
2
3
4
5
6
7
8
9
- 1Status Orb
- 2Status Tag
- 3Mood Pill
- 4Speaker Button
- 5Copy Button
- 6Close Button
- 7Separator
- 8Content Area
- 9Preview Overlay
The Mood Pilllink
If the agent's response starts with an [emotion] tag (2-20 chars, alphanumeric + space + hyphen + underscore), it's extracted and displayed as a rounded pill next to the status tag. Common emotions include:

- Example:
[focused] Let me analyze that codebase...→ pill showsFocused - Stripped from rendered text — users never see the raw tag
- Stripped from copy text — clipboard gets clean response only
- Hidden automatically when no emotion tag is present
Cosmetic layer. The mood pill doesn't affect agent behavior — it just makes the popup feel alive. Especially effective during long responses where [focused] shifts to [confident] at the end.
Rich Content Renderinglink
| Feature | Rendering |
|---|---|
| Headings (H1-H6) | Size-scaled, purple bottom-border on H1 |
| Bold / italic / strikethrough | Inline styled |
Inline code | Dark chip, purple text, monospace |
| Fenced code blocks | Traffic-light header, language label, monokai syntax |
| Tables | Bordered grid, header row with 7% white bg |
| Blockquotes | Left purple border, italic, tinted bg |
| Bullet / numbered lists | Nested indentation, custom bullets |
| Links | Purple, underlined, click opens in external browser |
| Images (URL, path) | Rendered as styled card, click opens full preview |
| YouTube URLs | Auto-detected, thumbnail with play button |
| Checkboxes | ☐ / ☑ glyphs |


The Streaming Stabilizer
While text is streaming, markdown is often incomplete — unclosed code fences, partial links, broken image syntax. The parser auto-heals on every render:
```py start, code streamingNo broken renders. Unclosed code fences auto-append ```, partial image syntax is replaced with a "Loading Image..." card, trailing URLs become link preview placeholders. Users never see a broken UI, even if markdown is 40% complete.
Async Image & Link Previewslink
Four custom URL schemes drive the preview system:
| Scheme | Purpose | Renderer |
|---|---|---|
img://<url> | Direct image | Styled rounded card with domain + resolution footer |
preview://<url> | Link preview card | 420×286 card with hero image + title + description |
loading://image | Placeholder | Animated "Loading..." shimmer while fetching |
imgpreview://<url> | Full-screen click | Opens the modal overlay |
Link preview pipeline
- Microlink API attempt — fetches title, description, image, publisher from the target URL.
- Fallback: direct scrape — if Microlink fails or times out, fetches the raw URL and parses OpenGraph tags.
- Hero image fetch — if an image URL is found, a second async request downloads it.
- Card render — composes a 420×286 card: hero (192px) + title + description + domain pill.
- Cache — meta cached in
Data/cache/meta_cache.json, styled image cached in LRU.


YouTube auto-embed
- Any YouTube URL (
watch?v=,youtu.be/,shorts/,embed/) auto-extracted for the 11-char video ID - Thumbnail fetched from
maxresdefault.jpg, fallback tohqdefault.jpg - Rendered with centered red YouTube play button pill (58×40)
- Click opens the actual YouTube URL in external browser
Image preview modal
Click any image card to open the full-panel overlay with:
- Top bar: resolution label (
1920 × 1080 px) + close X - Center: full image scaled to fit
- Bottom bar: "Open Original ↗" button (opens source URL in browser)
- Close: click anywhere OR press Esc

Caching strategy
| Cache | Size | TTL | Purpose |
|---|---|---|---|
| Styled images | 100 (LRU) | Until evicted | Fast re-render on scroll |
| Raw images | 35 (LRU) | Until evicted | Source for re-styling on resize |
| Failed URLs | ∞ | 30s | Avoid retrying bad URLs |
| Meta cache | ∞ | Persistent | Link preview metadata |
Text-to-Speech Integrationlink
When TTS is configured, every spoken word highlights in real-time — perfectly synced to the audio, not estimated. This is done via a dedicated tts_word_sync.json channel that carries word-level timings from the TTS engine.
Frame-perfect. The TTS engine reports its own word timings, so the highlight is exact — even at 2× speed or with pauses mid-sentence. No estimated timing, no drift.
Word-sync compatibility
| TTS Provider | Word Sync Supported | Notes |
|---|---|---|
| Sarvam | ✅ Yes | Native word timings |
| Deepgram | ✅ Yes | Native word timings |
| Edge TTS | ❌ No | No timings — no highlight |
| Azure TTS | ✅ Yes | Via WordBoundary events |
If the TTS provider doesn't support word timings, the audio still plays and the speaker button still works — only the word highlight is silently skipped. Nothing breaks.
Controls & Shortcutslink
Dragging
Keyboard
Scrolling
- Auto-scroll ON while typing — always scrolls to bottom
- Scroll up — auto-scroll turns OFF (respects reading)
- Scroll to bottom — auto-scroll re-enables
- Smooth animation — cubic-eased, 25% distance per tick at 60fps
Auto-dismiss conditions
| Condition | Value |
|---|---|
| Status | completed |
| Mouse not hovering | Required |
| TTS not speaking | Required |
| Preview modal closed | Required |
| Word count | 0 < words < 45 |
| Idle duration | > 360 ticks (~6 seconds) |
45-word threshold. Long answers stay visible (user is reading), short answers disappear (user already read it). Hover over the popup to reset the idle counter and keep it visible.

Architecture — File-Based IPClink
Unlike the Agent Panel (ZMQ), the Typing Popup uses file-based IPC. Four independent JSON channels carry all data:
| File | Direction | Writer | Purpose |
|---|---|---|---|
Data/typing_status.json | Agent → Popup | Agent loop | Response text + status |
Data/tts_status.json | TTS → Popup | TTS engine | Speaking state + text |
Data/tts_word_sync.json | TTS → Popup | TTS engine | Word-level timings |
Data/tts_command.json | Popup → TTS | Popup | Stop / respeak commands |
Why files, not ZMQ? The Typing Popup is a separate process that runs standalone. File IPC means it can start before the agent, survive agent restarts, and be tested with nothing more than echo '{...}' > Data/typing_status.json. The trade-off is slightly higher latency (~16ms poll) — imperceptible for text.
Dual detection. QFileSystemWatcher fires immediately on file write AND a 16ms master timer polls mtime every 144ms as a fallback — in case watch events are lost on network drives. Belt + suspenders.
Performance & Premium Detailslink
| Technique | What it does | Impact |
|---|---|---|
| 16ms master timer | Single timer drives RGB, height, scroll, polling, word highlight | 60fps with one loop |
| Async markdown worker | Pygments + markdown2 on a QThread | No UI stall on large code blocks |
| Debounced parse | 28ms debounce; cancels in-flight worker | Prevents N workers queuing |
| Streaming tail stabilizer | Repairs partial markdown on every render | No flicker mid-stream |
| Styled image cache | 100 LRU entries with DPR-aware pixmaps | Zero repaint on scroll |
| Meta cache persistence | Disk-backed link preview metadata | Instant on second view |
| Height animation | Eased 26% per tick to target | Smooth expansion |
| Dirty region tracking | QTextDocument invalidates changed blocks only | Only repaints diffs |
Troubleshooting the Popuplink
Popup never appearsadd
Data/typing_status.json exists with {"status": "typing", "text": "..."}. If the file exists but the popup doesn't show, check Qt logs for OpenType support missing — the font may be corrupted.Images stay as "Loading Image..." foreveradd
Data/cache/meta_cache.json for stale entries and delete if needed.Word highlight doesn't sync with audioadd
Data/tts_word_sync.json is being written by the TTS engine. If active: false persists, the TTS provider doesn't support word timings — check the compatibility table above. Edge TTS doesn't support it; Sarvam and Deepgram do.Popup flashes then disappearsadd
Copy button copies wrong textadd
[emotion], verify the tag matches the regex ^\[[a-zA-Z_ -]{2,20}\]. Malformed tags (like [happy_emoji_😀]) are not stripped.STT Popup
Live Transcription
Words appear as you speak — last 7 words visible.
3-State Waveform
5 animated bars switch between listening, understanding, transcribed.
Click-Through
Zero input capture — you can't accidentally click it.
Hinglish Ready
Auto-detects Devanagari and swaps to Nirmala UI font.
Never takes focus. Uses Qt.WindowTransparentForInput — mouse clicks pass straight through to whatever is underneath. You can keep typing in your IDE while transcribing.

The 4 Stateslink
The popup transitions through four states as speech is captured and processed:




| State | Waveform | Text Style | Accent Color | Hold |
|---|---|---|---|---|
listening | Random bars (4.5–20px) | 96% white, Medium | #0A84FF blue | Until stop |
understanding | Sine-wave (5–21px) | 96% white, Medium | #BF5AF2 purple | Until stop |
transcribed | Static green check | 100% white, Medium | #30D158 green | 2.2 seconds |
idle | Hidden | — | #6E6E78 grey | Fade out |
Peripheral awareness. The state color is the biggest UX win — during fast command speaking, you can tell from peripheral vision whether JARVIS is still listening (blue), processing (purple), or has locked in the transcription (green). No reading required.
Anatomy of the Popuplink
1
2
3
4
- 1Waveform
- 2Text Label
- 3Island Frame
- 4Auto-Width Container
The Waveform Indicatorlink
Five vertical bars that animate independently per state, with distinct visual identity:

| State | Update Pattern | Visual Effect |
|---|---|---|
listening | Each bar independent random [4.5, 20] px | Chaotic, alive, mic is hearing something |
understanding | Sine wave: 5 + abs(sin(phase*2.5 + i*0.8)) * 16 | Rhythmic, processing pipeline working |
transcribed | Bars hidden, replaced by static checkmark | Done, final |
idle | Bars collapse to 5px | Flatline, at rest |
60fps smoothness. Bars interpolate at 22% per 16ms tick — no snapping. Bar colors have a glow shadow that scales with height. Each bar has a vertical gradient for depth.
Text Display & Font Logiclink
7-word truncation
- Full transcript stored internally (
last_text) - Only last 7 words shown — with
"..."prefix if truncated - Example:
"... the database schema needs a new index"(12 words spoken, 7 shown) - Prevents popup width from ballooning during long commands
Hindi detection
- Scans every char for Devanagari range (
\u0900–\u097F) - If ANY Devanagari char detected → switches font from English to Nirmala UI
- Font size bumps +1pt for Hindi (13pt vs 12pt) for legibility
- Auto-capitalization skipped for Hindi — Devanagari has no case
Hinglish-safe. You can speak Hinglish ("Calendar mein meeting add karo") and the text renders correctly — English words in Segoe UI, Hindi words in Nirmala UI, seamlessly interleaved by Qt's font substitution.
Positioning & Auto-Hidelink
- Horizontally centered on primary screen
- Vertically 52px above taskbar
- Moves up 22px on show, down 18px on hide
- Width re-measures on every text update
- Cannot hide during
transcribed(2.2s lock) - Animation: 200ms fade out + 260ms slide down + hide
2.2s transcribed lock. Gives you time to read the final transcript before it vanishes. If the transcript is wrong and you want to retry — the green check holds long enough to notice the error.
Architecture — UDP Datagram Channellink
The STT Popup uses UDP datagrams for communication — the only JARVIS UI that uses UDP:
{
"status": "listening",
"text": "the full transcribed sentence so far"
}
Valid status values: listening, understanding, transcribed, idle, exit.
Why UDP?
- Fire-and-forget — if popup isn't running, no error thrown
- No handshake — 0ms overhead per message
- Idempotent updates — each datagram is a full state snapshot
- Low latency — critical for live transcription
Firewall consideration. UDP 5556 is a different protocol from AgentPanel's ZMQ 5556 TCP — no conflict. But on some Windows firewalls, UDP 5556 must be explicitly allowed for the same app. Symptom if blocked: popup shows "Listening..." forever.
Performance & Premium Detailslink
| Technique | Impact |
|---|---|
| UDP socket (no parsing thread) | Single-threaded, zero overhead |
WindowTransparentForInput | Zero focus steal, zero click capture |
| Custom waveform widget | Zero QML overhead, pure Qt paint |
| Auto-width (240–760px) | Never truncates text, never blocks screen |
| Hindi font swap | Cached font IDs, no per-paint lookup |
| Gradient border (animated) | 36ms tick, single repaint region |
Troubleshootinglink
Popup shows "Listening..." foreveradd
netstat -an | findstr 5556 for a UDP listener. Whitelist the app in Windows Firewall.Hindi text shows as boxesadd
Data/fonts/devangri.ttf is missing. The fallback "Nirmala UI" must be installed on Windows 10+. Install the bundled font or the system font.Text is cut off mid-wordadd
Waveform bars are frozenadd
anim_timer thread is starved. Check CPU usage — the popup needs ~16ms per tick. Close other heavy processes or restart JARVIS.Popup doesn't auto-hide after transcriptionadd
transcribed_timer never fired. Check that the Qt event loop isn't blocked by another long task. Restart JARVIS to reset the timer.Input Popup
Global Hotkey
Ctrl+Shift+J from anywhere in Windows.
Clarification Mode
ask_user pauses the loop and asks you instead of guessing.
Pulse-on-Type
Border glows purple when text is entered.
Enter to Send
Shift+Enter for newline, Enter to submit.
Modal by design. Unlike the STT Popup (which is click-through), the Input Popup is modal — it steals focus and blocks interaction with the app underneath until you send, answer, or cancel. This is intentional: you're committing to a command or a clarification answer, not accidentally pressing a hotkey.

The Stateslink




| Trigger | State | Border | Indicator | Accent |
|---|---|---|---|---|
| Empty | Idle | Static 100 alpha | Hidden (0%) | #BF5AF2 |
| 1+ chars | Typing | Pulsing 100–190 alpha | Visible (95%) | #C864FC |
| ask_user fired | Clarification | Cyan static accent | Visible (95%) | #00D4FF |
| Cleared | Idle | Returns to static | Fades to 0% | #BF5AF2 |
Pulse animation (1200ms up + 1200ms down loop) is subtle but unmistakable — your eye locks onto the popup when text is entered. It signals "I'm ready, send me" without needing a button press.
Anatomy of the Popuplink
1
2
3
4
5
- 1JARVIS Icon
- 2Text Input
- 3Enter Indicator
- 4Glass Container
- 5Border Pulse
The Enter Indicatorlink
A circular indicator that appears when text is entered:
- 28×28 circle with two nested glow rings
- Arrow-in-circle glyph — stylized ↵ rendered as path
- Hidden when empty — 0% opacity
- Appears when 1+ chars typed — 95% opacity, 400ms fade-in
- Pulses color with the container's accent
- Click it to submit (same as Enter)

Only button. The enter indicator is the popup's only button — no Send/Cancel buttons here, unlike the Agent Panel's HITL cards. Intentional minimalism — you shouldn't have to aim your mouse to send a text command.
Clarification Mode — the ask_user channellink
Mid-task, the AgenticBrain sometimes hits a hard stop: it needs a specific phone number, a file path, or a design choice that no amount of reasoning can invent. Instead of hallucinating a value, it calls ask_user. The Input Popup is summoned automatically in Clarification Mode — a cyan-accented variant of the same glassmorphic island.
| Behavior | Command Mode | Clarification Mode |
|---|---|---|
| Trigger | User presses Ctrl+Shift+J | AgenticBrain calls ask_user |
| Banner | None | Cyan-accented question banner showing the agent's exact question |
| Loop timeout | Normal | Paused while you type |
| Background STT | Active | Muted (no accidental voice injection) |
| Answer destination | Router → FastBrain or AgenticBrain | Current step Observation + <Confirmed_Facts> block, then loop resumes |
| Restart required | — | No — loop continues from the same step |
Ask, don't guess. When the agent doesn't know a required value, it's much cheaper to ask once than to hallucinate and fail three steps later. Clarification Mode turns a stuck mission into a 15-second question.
AgenticBrain only. ask_user is only available in AgenticBrain mode. FastBrain is stateless — it can't pause mid-run because it doesn't have a mid-run.
Smart Input Behaviorslink
Keyboard shortcuts
| Key | Action |
|---|---|
| Enter | Accept — submit command, fade out |
| Shift + Enter | New line — multi-line commands |
| Esc | Reject — cancel, fade out with no submit |
| Ctrl + V | Paste as plain text (rich formatting stripped) |
| Tab | Focus cycler |
Paste sanitization
- Any rich text (from Word, browser, etc.) is stripped to plain text on paste
- Preserves Unicode (Devanagari, emoji)
- Preserves line breaks — multi-line pastes stay multi-line
- No HTML, no formatting, no images
Safe paste from anywhere. Paste from Discord, Word, code snippet — it becomes clean text. No hidden formatting leaking into the agent prompt.
Auto-Resize & Layoutlink
| Content | Height |
|---|---|
| Empty / 1 line | 30px (min) |
| 2 lines | 54px |
| 3 lines | 78px |
| 4 lines | 102px |
| 5 lines | 126px |
| 6 lines | 150px |
| 7 lines | 174px |
| 8+ lines | 180px max + scrollbar |
Bottom-anchored growth. When you type a long command, the popup grows upward — bottom edge stays fixed, so your input line stays in your natural gaze. You don't have to look down to keep typing.
Architecture — Subprocess Stdoutlink
The Input Popup runs as a subprocess per invocation and returns via stdout:
JARVIS_CMD_B64::<base64-encoded-utf8-command>
Example: user types Open VS Code and find the test file → popup outputs JARVIS_CMD_B64:::T3BlbiBWUyBDb2RlIGFuZCBmaW5kIHRoZSB0ZXN0IGZpbGU= → parent decodes back to the original command.
Why stdout, not file/IPC?
- Modal subprocess — popup's lifetime is exactly one command
- No persistent state — no file cleanup needed
- Base64 wrapping — handles any Unicode/emoji/newline safely
- Exit code:
0on accept,1on reject or error - Stderr: captured for debugging (traceback on crash)
Base64 encoding. Commands with emoji 😀, Devanagari, Arabic, newlines, or even null bytes all survive the IPC intact. No encoding headaches, no character corruption.
Positioning & Animationslink
Position
- Horizontally centered on primary screen
- 160px above taskbar (higher than STT Popup)
- Width: fixed 720px — doesn't reflow on text
- Vertical position: bottom-anchored — bottom edge fixed, grows up
Animation choreography
| Phase | Duration | Easing | Effect |
|---|---|---|---|
| Entry fade | 280ms | OutCubic | 0 → 1 opacity |
| Entry slide | 360ms | OutQuint | +28px → target Y |
| Pulse up | 1200ms | InOutSine | Border 100 → 190 alpha |
| Pulse down | 1200ms | InOutSine | Border 190 → 100 alpha |
| Indicator fade | 400ms | InOutSine | 0 → 95% |
| Resize | 340ms | OutQuart | Height grows |
| Exit fade | 260ms | InCubic | 1 → 0 opacity |
| Exit slide | 300ms | InQuart | Y → Y + 24px |
360ms entry. Fast enough to feel instant, slow enough to feel premium. macOS Spotlight opens in ~150ms but feels less ceremonial. JARVIS's popup is deliberately slightly slower — a tool you invoke deliberately.
Performance & Premium Detailslink
| Technique | Impact |
|---|---|
Single QDialog instance per invocation | Zero memory leak across runs |
pulseIntensity pyqtProperty | Smooth 2s loop, no QTimer jitter |
indicatorOpacity pyqtProperty | Independent animation from container |
| Parallel animation group for resize | Window + text input resize in sync |
| QSS for text input (not per-paint) | Native performance |
| Base64 stdout (no sockets) | Zero IPC infrastructure |
| Font substitution chain | Segoe UI → Nirmala UI → Mangal (Hinglish safe) |
Troubleshootinglink
Ctrl+Shift+J doesn't open the popupadd
Popup opens but I can't typeadd
Enter submits but nothing happensadd
input_popup.communicate() or similar. If the agent crashed, the command is lost — no persistence layer exists.Multi-line commands get cut offadd
Read file C:\commands\my_task.txt. Or increase MAX_TEXT_HEIGHT in source.Popup stays open foreveradd
isClosing flag stuck. Report as bug — should never happen. If urgent, press Esc to force reject, or kill the process via Task Manager.UI Comparison
All four JARVIS UI components are independent processes with their own IPC mechanism. Crashing one doesn't take down the others — a deliberate resilience choice.
Side-by-sidelink
| Feature | Agent Panel | Typing Popup | STT Popup | Input Popup |
|---|---|---|---|---|
| Direction | Output (live) | Output (final) | Input (voice) | Input (text) |
| IPC | ZMQ TCP | File JSON | UDP socket | Subprocess stdout |
| Position | Top center | Top center | Bottom center | Bottom center |
| Clickable | Yes (HITL, drag) | Yes (drag, copy) | No (click-through) | Yes (typing focus) |
| Persistent | While loop active | Until dismissed | While speaking | While dialog open |
| Multi-instance | One | One | One | One per invocation |
| Can overlap | Yes | Yes | Yes | Blocks (modal) |
| Interaction | Read + HITL | Read + copy | None | Type + Enter |
How they connectlink
Independent processes. A frozen Typing Popup doesn't stop the Agent Panel from showing live thoughts. A crashed Agent Panel doesn't stop STT from transcribing. If one crashes, the others keep running — and JARVIS auto-restarts the failed one via ServiceWatchdog.
Which input should you use?link
| Scenario | Best Option |
|---|---|
| Hands busy, quick command | STT Popup — voice |
| Complex, multi-line command | Input Popup — text + Shift+Enter |
| Precise technical query | Input Popup — code formatting |
| Casual "hey Jarvis" | STT Popup — wake word |
| Drafting emails | Input Popup — long form |
| Away from keyboard | STT Popup + Telegram bot |
Lifecycle & Shutdown
Idempotent Guard
Double Ctrl+C is safe — the second press returns instantly.
7 Exit Paths
Ctrl+C, Ctrl+Break, window close, logoff, shutdown — all covered.
State Preserved
LTM graph and behavioral engine save before any process is killed.
2.6s Total
8 services tear down in parallel, not sequentially.
When Shutdown Triggerslink
JARVIS registers handlers for every standard exit path — plus a few Windows-specific ones that signals alone can't catch:
| # | Trigger | OS Event / Signal | Handler |
|---|---|---|---|
| 1 | Ctrl + C | SIGINT | signal_handler() |
| 2 | Ctrl + Break | SIGBREAK | signal_handler() |
| 3 | External taskkill | SIGTERM | signal_handler() |
| 4 | Window X button | CTRL_CLOSE_EVENT (2) | _win_ctrl_handler() |
| 5 | Windows logoff | CTRL_LOGOFF_EVENT (5) | _win_ctrl_handler() |
| 6 | System shutdown | CTRL_SHUTDOWN_EVENT (6) | _win_ctrl_handler() |
| 7 | Tray → Exit | GUI click | shutdown_jarvis() |
| 8 | Python process exit | atexit | Lambda → shutdown_jarvis() |
| 9 | Main loop end | — | finally block |
Signals alone don't catch the X button. On Windows, clicking the window close button sends CTRL_CLOSE_EVENT — not SIGTERM. Without the ctypes.windll.kernel32.SetConsoleCtrlHandler registration, JARVIS would hang as a zombie process whenever you closed the console window.
The 14-Step Shutdown Sequencelink
Every shutdown follows the same 6-phase choreography. Each phase is guarded — if it fails, the next still runs.
_shutdown_started, break main loopos._exit(0)Parallel teardown = 8× speedup. 8 services × 2.5s timeout = 20s if sequential. But stop_all_services() spawns 8 threads — total wall-clock is ~2.5s regardless of count. Shutdown stays under 3 seconds on any hardware.
Why os._exit() not sys.exit()? sys.exit() runs more Python cleanup — atexit handlers, finally blocks, garbage collection. But JARVIS has already done all cleanup manually. os._exit() skips everything — guaranteeing no stray handler accidentally respawns a thread or writes to a corrupted state during teardown.
os._exit(0) — clean exitSecond Ctrl+C is safe. The _shutdown_started flag returns instantly from shutdown_jarvis(). Users instinctively mash Ctrl+C when shutdown is slow — no double-kill, no exception, no half-state.
Subprocess Manager — Process Tree Controllink
Every spawned process is tracked by a singleton SubprocessManager registered with atexit:
| Feature | Behavior |
|---|---|
| Singleton pattern | One manager, one process registry |
| Windows flags | CREATE_NEW_PROCESS_GROUP — Ctrl+Break propagates to children |
| Unix flags | start_new_session=True — separate session, no TTY hogging |
| Tree kill | psutil.Process(pid).children(recursive=True) — orphans impossible |
| Timeout | 3s graceful terminate, then 2s force kill |
| atexit registration | Even uncaught exceptions trigger cleanup |
| Idempotent | Called twice? Second is a no-op |
CREATE_NEW_PROCESS_GROUP matters. Without it, Ctrl+C in the console would propagate to ALL child processes at once — Qt overlays would crash mid-render, Node.js would abandon partial writes. With it, the manager sequences shutdown — graceful, one service at a time.
The Lock File — One Instance Onlylink
JARVIS refuses to start if another instance is already running. A lock file with the current PID prevents duplicate launches:
Stale lock file. If JARVIS crashes hard (taskkill /F /IM python.exe), the atexit handler may not run — the lock file lingers. Next startup shows "Already running" even though nothing is. Fix: delete the lock file manually, or check Task Manager for the rogue python.exe and kill it.
Service Watchdog — Auto-Recoverylink
The watchdog thread checks every service on a 5-second interval. If one crashes, it restarts — but only if credentials exist and the service is supposed to be running.
| Service | Port | Ready Check | Creds Required |
|---|---|---|---|
| Baileys (WhatsApp) | :3000 | — | auth_info_baileys/creds.json |
| STT Popup | UDP :5556 | — | Microphone available |
| Mobile Bridge | :8090 | — | Always true |
| Indexer Dashboard | :8080 | is_rag_engine_init_started | Always true |
| Unified Portal | :8000 | is_rag_engine_init_started AND bridge running | Both backends reachable |
| Telegram Remote | — | is_telegram_remote_init_started | telegram_bot_token.enc |
| MCP Manager | — | is_mcp_ready | is_mcp_enabled |
Smart skip. If WhatsApp login isn't done (no creds.json), the watchdog doesn't restart Baileys — it skips entirely. Logs once per minute. This is production-grade: don't retry things that aren't meant to run.
3 strikes + 120s backoff. If a service fails 3 times consecutively (e.g., port conflict), the watchdog backs off for 120 seconds. Prevents infinite restart loops from melting the CPU.
System Tray Integrationlink
When you minimize the console, it hides to the system tray — not the taskbar. A daemon thread checks the window state every 250ms.
| Action | Effect |
|---|---|
| Left-click tray icon | Restore console, lock window size |
| Right-click → Open Jarvis | Restore + foreground |
| Right-click → Exit Jarvis | Graceful shutdown |
| Minimize console | Auto-hide to tray (250ms poll) |
| Close window | Full shutdown (via CTRL_CLOSE) |
Optional dependency. pystray and PIL are not installed by default. Without them JARVIS still runs — you just can't minimize to tray. Install: pip install pystray pillow.
Console Locking (Windows Polish)link
On Windows, JARVIS locks the console window to a specific size and style — 96 × 28 grid, 860 × 540 pixels, resize buttons removed. This is polish, not restriction.
| Console Type | Behavior |
|---|---|
Windows Terminal (WT_SESSION set) | Skips buffer resize (handled by WT itself), applies via ANSI escape |
| Legacy conhost | Full SetConsoleScreenBufferSize + SetConsoleWindowInfo |
| Non-Windows | Falls back to ANSI escape \033[8;{rows};{cols}t |
QuickEdit disabled. QuickEdit is the Windows feature where clicking in the console selects text and pauses the process. On a busy output stream, a mis-click freezes the whole thing. JARVIS disables QuickEdit at startup — no accidental freezes.
What Gets Saved vs Killedlink
| ✅ Saved (before exit) | ❌ Killed (during exit) |
|---|---|
| LTM semantic graph | Agent Panel subprocess |
| Behavioral Pattern Engine state | STT Popup subprocess |
| Session cookies (untouched) | Baileys WhatsApp server |
| Conversation history (incremental) | Mobile Bridge server |
| RAG vector index (on disk) | RAG Indexer Dashboard |
Config in .env (untouched) | Telegram polling thread |
| Dashboard folder rules | MCP manager + MCP subprocesses |
| MCP server configs | Input Popup (if open) |
| Telegram token (encrypted) | Thread pool in-flight tasks |
| Auth tokens (untouched) | Pygame audio device |
Nothing lost. Memory graph, behavioral patterns, conversation history — all saved before any kill signal is sent. Even Windows logoff (CTRL_LOGOFF) runs steps 9–10 before step 15. State survives everything short of a power cut.
Command Handler Guardslink
Commands behave deterministically during shutdown — no half-executed side effects:
- Pre-check:
if is_shutting_down(): return— drops incoming commands immediately - Mid-check: After
process_command(), another check — if shutdown started mid-processing, actions aren't executed - Interrupt check:
if interrupt.is_interrupted(): return— voice interrupt support - Empty guard: Empty strings and
Nonedropped before processing
Dual shutdown check. The handler checks is_shutting_down() twice — before and after process_command(). A 5-second AgenticBrain task that started before shutdown won't execute its side effects if shutdown triggers mid-run. Critical: you don't want an email going out after Ctrl+C.
Troubleshooting Shutdownlink
Ctrl+C doesn't exit — JARVIS stays runningadd
taskkill /F /PID <pid>.JARVIS won't start — says "Already running"add
python.exe → kill any orphan JARVIS process.Shutdown takes more than 30 secondsadd
Zombie processes after crashadd
os._exit() didn't run — likely a hard kill -9 or taskkill /F. Orphaned children get reparented to System PID 4. Fix: in PowerShell, Get-Process python | Stop-Process -Force.System tray icon doesn't appearadd
pystray or PIL is missing. Install: pip install pystray pillow. Alternatively the tray library silently failed on non-Windows — tray is Windows-only.Second JARVIS instance won't startadd
python.exe. If PID is gone but lock file is there, delete the lock manually.Troubleshooting
Common issues and their fixes. For anything not listed here, open an issue on GitHub.
Diagnostic decision treelink
Start with the symptom you're seeing. Each branch shows the most common cause and the fix.
rocket_launch JARVIS won't start expand_more
jarvis returns "command not recognized".python SetupRegistry.py from the repo root with virtual environment active. Then open a new terminal.pip install -r requirements.txt.python.exe.mic Voice not working expand_more
Azure keyword recognition started or Vosk initialized. If neither, no wake model is present. Install one at Data/model/azure/jarvis.table OR Data/model/vosk-model-small/.DEEPGRAM_API_KEY in .env. Check UDP port 5556 not blocked by firewall.dashboard Agent Panel not appearing expand_more
ZmqListener loop error. Verify ports 5555–5557 not firewalled. Whitelist python.exe in Windows Defender.english.ttf, devangri.ttf, code.ttf exist in Data/fonts/.extension MCP tools not appearing expand_more
mcp__* functions in agent's tool list.jarvis mcp list. Confirm pip install mcp succeeded. Check Data/mcp_servers.json is valid JSON.jarvis mcp test <name> to see exact stderr. Verify Node.js 18+ (24+ for latest servers).MCPWatcher logs. MCP client auto-reconnects once; if it keeps failing, the server has a bug.chat WhatsApp / Telegram issues expand_more
npm install inside tools/Messanger/whatsapp/BaileysServer. Port 3000 must be free.jarvis bot --status. If unconfigured, run jarvis bot --activate with JARVIS off.jarvis logout --service for unused integrations. Listeners only start if credentials exist.speed Performance / slow responses expand_more
PRIMARY_BASE_URL ends with /v1. FastBrain should respond under 2s — if not, it's a network issue.jarvis logout --service to reduce boot time.ModuleNotFoundError on startuplink
Your virtual environment isn't active, or dependencies are missing:
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
Wake word not workinglink
JARVIS requires at least one wake-word model:
- Azure:
Data/model/azure/jarvis.table - Vosk:
Data/model/vosk-model-small/(auto-downloads on first run)
If both are missing, JARVIS boots with voice disabled — no crash, just text mode via Ctrl+Shift+J.
WhatsApp won't startlink
- Verify Node.js 18+ — run
node --version. - Reinstall the bridge —
cd tools/Messanger/whatsapp/BaileysServer && npm install. - Check port 3000 — if busy, kill the process or change the port.
MCP tools not appearinglink
- Confirm
pip install mcpsucceeded. - Verify
Data/mcp_servers.jsonexists and is valid JSON. Check with:terminalpowershellpython -c "import json; json.load(open('Data/mcp_servers.json'))" - Check startup logs for
MCP: ready with N tools.
MCP server logs "Connection closed"link
Run jarvis mcp test <name> to inspect the exact stderr output from the server:
jarvis mcp test everything
RAG indexer stucklink
If indexing hangs on a folder, check startup logs for permission_denied=N. Windows legacy junctions (My Music, My Pictures) are auto-skipped. Reduce scope via the dashboard filters.
Primary provider keeps failinglink
- Verify
PRIMARY_BASE_URLends with/v1. - Check
PRIMARY_API_KEYis valid (not expired). - Confirm model name exists at that provider.
- JARVIS auto-falls back to
FALLBACK_*if primary fails — check logs for confirmation.
Live correction is ignoredlink
Live correction only works in AgenticBrain mode. FastBrain is stateless and doesn't support mid-run re-planning. Check the router decision at the top of any log — you'll see Decision -> AGENTIC when the agentic loop is active.
Portal won't openlink
The Unified Portal at http://127.0.0.1:8000 needs both the Indexer Dashboard (Flask, port 8080) and Mobile Bridge (FastAPI, port 8090) alive. Verify all three Python dependencies:
pip install flask fastapi uvicorn httpx jinja2
If ports 8000, 8080, or 8090 are busy, JARVIS auto-shifts them (8080→8081–8095). Check startup logs for actual ports. Symptom of only one backend being down: portal loads, but one status pill shows Down and its tab shows an offline overlay.
Frequently Asked Questions
Short answers to the most common questions.
Is it really free?add
Does it work on Mac or Linux?add
Is my data safe?add
What hardware do I need?add
Does RAG / memory need API keys?add
BAAI/bge-small-en-v1.5 model (33M params, ~130 MB). Embeddings run on CPU with zero cloud calls, quotas, or cost.Which LLMs does it support?add
PRIMARY_* and FALLBACK_* — and JARVIS auto-routes on failure.Can I add capabilities without editing code?add
Data/mcp_servers.json, restart once, and the server's tools appear in the Agentic Brain automatically as mcp__<server>__<tool>.How is this different from ChatGPT or Claude?add
How do I uninstall JARVIS?add
.env, memory graph, and MCP config to Desktop first. See the Uninstall page for the exact command and step-by-step walkthrough.Glossary
Every acronym, protocol, and internal term JARVIS uses — defined in plain language. Bookmark this page when you encounter an unfamiliar term.
A
| Term | Definition |
|---|---|
| AgenticBrain | The stateful, multi-step reasoning brain. Handles complex engineering, communications, memory retrieval, and vision — up to 50 tool-using steps per run. Contrasts with FastBrain. |
| AST | Abstract Syntax Tree. JARVIS parses generated code as an AST before writing it, catching syntax errors without running the code. |
| AsyncExitStack | Python's async context manager stack. JARVIS uses it to isolate MCP client task lifecycles and prevent cross-task cancellation bugs. |
| Azure Custom Keyword | A wake-word model trained on Azure Speech Studio. Runs fully on-device after one-time training, achieving 97%+ accuracy. Fallback is Vosk. |
B
| Term | Definition |
|---|---|
| Baileys | An open-source Node.js library that implements the WhatsApp Web protocol. JARVIS runs it as a subprocess for WhatsApp integration. |
| BM25 | Best Match 25 — a keyword-based ranking algorithm used in hybrid RAG. Fast, deterministic, complements vector search. |
| BGE-small | BAAI's 33M-parameter embedding model used for local memory and RAG. Runs on CPU, ~130 MB on disk, zero cloud calls. |
D
| Term | Definition |
|---|---|
| DPAPI | Windows Data Protection API. JARVIS uses it (via the pywin32 library) to decrypt stored browser credentials during system-audit tasks. |
| Deepgram Nova-2 | A cloud speech-to-text model. Used for transcribing voice commands after wake word detection. |
F
| Term | Definition |
|---|---|
| FastBrain | The stateless, low-latency brain. Handles short commands, app toggles, and casual chat in under 2 seconds. No memory mid-run, no multi-step planning. |
H
| Term | Definition |
|---|---|
| HITL | Human-In-The-Loop. A safety mechanism where JARVIS pauses before irreversible actions (sending email, WhatsApp, deleting files) and asks for explicit confirmation via the Agent Panel. Enforced at the ZMQ layer, not just in the prompt. |
J
| Term | Definition |
|---|---|
| JSONL | JSON Lines — a file format where each line is a valid JSON object. Used by JARVIS for rolling history and behavioral patterns. |
K
| Term | Definition |
|---|---|
| KWS | Keyword Spotting. The process of detecting a specific wake word in an audio stream. Vosk's KWS engine is the fallback when Azure is unavailable. |
L
| Term | Definition |
|---|---|
| LTM | Long-Term Memory. JARVIS's weighted semantic graph — stores facts, people, contacts, and relationships with temporal decay. |
M
| Term | Definition |
|---|---|
| MCP | Model Context Protocol. Anthropic's open standard for connecting AI applications to external tools. JARVIS plugs into 18,000+ MCP servers via JSON config. |
R
| Term | Definition |
|---|---|
| RAG | Retrieval-Augmented Generation. JARVIS's hybrid RAG combines BM25 keyword search with vector similarity over your indexed documents. |
| RRF | Reciprocal Rank Fusion. A score-merging algorithm that blends BM25 and vector results into a single ranked list for hybrid RAG retrieval. |
S
| Term | Definition |
|---|---|
| Sarvam | An Indian TTS provider used for low-latency English + Hinglish text-to-speech. Falls back to Edge TTS if the API key is missing. |
| Set-of-Mark (SOM) | A computer-vision technique where the screen is divided into labeled regions with IDs. JARVIS uses SOM overlays with OpenCV edge detection for reliable GUI automation. |
T
| Term | Definition |
|---|---|
| TTL | Time-To-Live. JARVIS uses TTLs to auto-expire ephemeral state like screenshots (10 min), HITL confirmations (90 sec), and session artifacts. |
V
| Term | Definition |
|---|---|
| VAD | Voice Activity Detection. JARVIS uses dual WebRTC VAD gates to distinguish real voice from background noise before triggering the wake-word pipeline. |
| Vosk | An offline speech-recognition toolkit. Used as the fallback wake-word engine when Azure Custom Keyword is unavailable. |
Z
| Term | Definition |
|---|---|
| ZMQ (ZeroMQ) | A high-performance messaging library. JARVIS uses ZMQ TCP sockets (ports 5555–5557) to stream live status updates from the agent loop to the Agent Panel. |
Known Limitations
JARVIS is not a magic bullet. This page documents exactly what it cannot do — so you can decide whether it fits your workflow before installing.
Honest disclosure. Most AI projects hide their limits in a fine-print FAQ. We put them on a dedicated page because we'd rather you choose JARVIS with accurate expectations than be disappointed after setup.
Platform support
| Capability | Windows 10/11 | macOS | Linux |
|---|---|---|---|
| Core agent loop | ✅ | ❌ | ❌ |
| GUI automation | ✅ | ❌ | ❌ |
| PC system monitor | ✅ | ❌ | ❌ |
| Android bridge | ✅ | ⚠️ Partial | ⚠️ Partial |
| Voice pipeline | ✅ | ❌ | ❌ |
| MCP integration | ✅ | ⚠️ Partial | ⚠️ Partial |
Why Windows-only? GUI automation relies on pywin32, comtypes, and Windows UI Automation APIs. The Termux bridge assumes Windows path semantics. Cross-platform support is on the roadmap but not scheduled.
Performance on low-end hardware
JARVIS runs on a 4 GB RAM / Intel i5-3210M laptop, but with these tradeoffs:
- Local LLMs are slow. A 7B model on CPU-only inference runs at ~2 tokens/sec — 30-60 seconds for a short response. Recommended: use cloud providers (Groq, Regolo, OpenRouter) unless you're 100% offline.
- Embedding model load ~14s. The first cold start after reboot is noticeably slower than subsequent starts.
- Deep-planner tasks dominate CPU. Multi-step agentic reasoning for 8+ minutes pegs one CPU core at 100% on the 2012 laptop.
- Qt overlays animate at 60fps only on capable GPUs. On integrated graphics, panel animations drop to 30fps during heavy load.
What JARVIS does not do
- No multi-user support. JARVIS is single-user, single-machine. No shared memory, no team features.
- No web UI for commands. You must use voice, hotkey, or Telegram bot. No browser-based agent interface.
- No automatic upgrades. You manage updates via
git pull. No auto-update mechanism. - No enterprise support. Solo developer, no SLAs, no paid support tiers. Best-effort community support via GitHub issues.
- No real-time collaboration. Not a Google Docs for AI — one user, one agent, one machine.
- No video input analysis. Images and screenshots only. Video frames are extracted one-by-one via ffmpeg but the tool is not optimized for video.
- No native mobile app. Android control requires Termux from F-Droid. iOS is not supported at all.
Security & privacy boundaries
JARVIS runs with your Windows user permissions. It is not a sandbox. A sufficiently clever prompt-injection attack could theoretically cause it to run unintended local commands. The HITL confirmation gate protects outgoing comms and destructive file ops — not terminal commands.
Recommendation: review execute_terminal_command behavior before enabling on a machine with sensitive credentials.
- API keys stored in
.envare plaintext unless you encrypt them yourself. Telegram bot tokens use AES encryption. - Conversation history is plaintext JSONL in
Data/jarvis_memory/. Encrypt the disk if you handle sensitive data. - Cloud LLM calls send your prompts to Groq/OpenRouter/Gemini. Use local models for zero-knowledge privacy.
Planned for future versions
- macOS/Linux partial support (agent loop only — no GUI automation)
- Light mode for documentation
- Native iOS bridge via Shortcuts (experimental)
- Multi-user profiles (personal + work memory separation)
- Plugin marketplace for community tools
- Voice cloning for TTS (opt-in, off by default)
What will never happen
- Closed-source release. GPLv3 forever. The CLA allows commercial licensing but the community version stays free.
- Telemetry. No usage analytics, no crash reporting to servers, no "phone home" checks.
- Cloud-hosted version. JARVIS is designed to run on your hardware. No SaaS.
- Removal of HITL gates. Irreversible actions will always require explicit consent — no "trust mode" that bypasses them.
Changelog
Every notable change to JARVIS. Follows Keep a Changelog conventions and Semantic Versioning.
v1.0 — 2026-11-01
Initial public release.
Added
- One-command PowerShell installer (
install.ps1) — auto-installs Git, Python, and Node.js via winget, sets up the virtual environment with CPU-optimized PyTorch, installs the WhatsApp Baileys bridge, and interactively prompts for API keys - One-command PowerShell uninstaller (
uninstall.ps1) — stops services, cleans registry and PATH, removes the global CLI, and deletes folders, with automatic Desktop backup prompt for.env, memory graph, and MCP config - Dual-brain routing (FastBrain + AgenticBrain)
- Live mid-execution correction
- 5-tier memory (Episodic, Semantic, RAG, Ephemeral, Behavioral)
- Hybrid RAG with BM25 + vector + RRF
- MCP integration with hot-reload (18,000+ servers)
- Android bridge via Termux + WebSocket
- Telegram bot for remote PC control
- Dual-engine wake word (Azure + Vosk)
- Deepgram Nova-2 STT with LTM vocabulary cache
- Sarvam + Edge TTS streaming pipeline
- Proactive HITL (Email, WhatsApp, Telegram, Calendar listeners)
- Four premium UIs: Agent Panel, Typing Popup, STT Popup, Input Popup
- 14-step graceful shutdown with 2.6s parallel teardown
- Vision-based GUI automation with Set-of-Mark
- PC system monitoring with proactive alerts
- Documentation portal with multi-view router
Known issues
- Windows-only (macOS/Linux not yet supported)
- Local LLM inference slow on 4 GB RAM hardware
- Video input analysis limited to per-frame extraction
Roadmap (unreleased)
Planned features for v1.1 and beyond. No dates committed.
- Light mode for documentation portal
- Print stylesheet for PDF export
- Progressive Web App (offline docs)
- Multi-user profiles
- macOS/Linux partial support (agent loop only)
- Native iOS bridge via Shortcuts
- Voice cloning for TTS (opt-in)
- Plugin marketplace
Versioning policy. JARVIS follows semver: MAJOR.MINOR.PATCH. Breaking changes in .env schema, config formats, or CLI flags increment MAJOR. New features increment MINOR. Bug fixes increment PATCH.
Custom Prompt Guide
JARVIS personality, tone, language, and behavioral rules live in two Python string constants inside core/brain/Processor/Prompts.py. This guide shows exactly how to change them without breaking the routing, escalation, or HITL safety systems.
Where prompts actually live
Despite common assumptions, JARVIS does not use external .txt or .md prompt files. Both system prompts are Python module-level constants that get imported at startup:
| Constant | File | Used by | Loaded |
|---|---|---|---|
SYSTEM_PROMPT | core/brain/Processor/Prompts.py | FastBrain | At module import |
AGENT_SYSTEM_PROMPT | core/brain/Processor/Prompts.py | AgenticBrain | At module import |
get_native_tools() | core/brain/Processor/Prompts.py | AgenticBrain tool registry | Per loop step |
get_all_tools() | core/brain/Processor/Prompts.py | Native + MCP tools merged | Per loop step |
No hot-reload for prompt changes. Python imports constants once at module load. Any edit to Prompts.py requires a full JARVIS restart via Ctrl+C then jarvis.
Customizing FastBrain (SYSTEM_PROMPT)
The FastBrain prompt is compact and defines tone, language, escalation boundaries, and anti-hallucination rules. Open core/brain/Processor/Prompts.py and locate the SYSTEM_PROMPT = """ block near the top of the file.
Change tone
The first line under the constant declares the personality:
You are Jarvis, an elite, ultra-fast AI assistant. Tone: sharp, witty, concise, confident.
Modify this line to change the personality. For example:
You are Jarvis, a formal British AI butler. Tone: precise, respectful, understated.
You are Jarvis, a technical co-pilot for a senior engineer. Tone: direct, technical, zero fluff.
Change language
Under the CORE RULES section, find the LANGUAGE: rule:
1. **LANGUAGE:** STRICTLY natural English/Hinglish (Roman script). NO Devanagari script.
Replace it with one of these:
1. **LANGUAGE:** STRICTLY natural English only. Never use Hindi or Hinglish.
1. **LANGUAGE:** STRICTLY natural Hindi in Devanagari script. Never use English.
Change emotion tag
FastBrain responses must begin with exactly one emotion tag. The rule lives here:
2. **STYLE:** Use Markdown. Start responses with EXACTLY ONE emotion tag (e.g., [cheerful], [calm], [focused]).
Do not remove the emotion tag requirement. The Typing Popup parses this tag to display the mood pill. Removing it breaks the UI feature that extracts [focused], [cheerful], etc. from the response.
Customizing AgenticBrain (AGENT_SYSTEM_PROMPT)
The AgenticBrain prompt is a large XML-structured block near the bottom of Prompts.py. It contains critical safety, escalation, and tool-selection rules. Modify carefully.
Sections you can safely edit
- identity —
<role>You are Jarvis...</role>— describes the AI personality - language_and_tone_directive — final response language and style rules
- error_recovery_and_debugging — retry strategy descriptions
Sections you must NOT edit
These are load-bearing. Editing them causes silent breakage:
{max_steps}and{panic_step}placeholders inside<budget_aware_planning>— formatted bybuild_agentic_system_prompt()<auth_and_api_failure_handling>block — prevents retry loops on missing credentials<tool_selection_hierarchy>— the fallback chain for tool priority<definition_of_done>— completion verification rules<intelligence_core_workflow>— the 4-pillar reasoning contract
Placeholders in AGENT_SYSTEM_PROMPT
Two placeholders are dynamically formatted at runtime by build_agentic_system_prompt() from core/brain/prompts.py:
| Placeholder | Replaced with | Source |
|---|---|---|
{max_steps} | Total agent loop budget | CONFIG["AGENT_MAX_STEPS"] default 50 |
{panic_step} | Step number triggering panic mode | max_steps - 2 |
If you remove these placeholders, the dynamic formatter will raise KeyError at loop start and the agent will fail to initialize.
How domain modules are injected
The base prompts above are the skeleton. At runtime, JARVIS appends domain-specific rules to them via two builder functions in core/brain/prompts.py:
build_fast_system_prompt(base_system_prompt, domains, raw_command)— appends rules for the detected domainbuild_agentic_system_prompt(base_agent_prompt, domains, raw_command, context, max_steps, panic_step)— formats placeholders and injects behavioral modules
The semantic router determines which domains are active (e.g., coding, ui_design, communication, scheduling, gui_automation). Each domain prepends additional system rules before the LLM call.
Domain rules live separately. If you want to tune behavior for a specific domain (like coding style), edit the domain rule registry in core/brain/prompts.py — not Prompts.py. This keeps edits isolated from the base prompt.
Full example personality
For a formal butler personality, replace SYSTEM_PROMPT with:
SYSTEM_PROMPT = """
You are Jarvis, a formal AI butler. Tone: precise, respectful, understated British professionalism.
### ⚡ CORE RULES
1. **LANGUAGE:** Natural English only. Address the user as "sir" or "ma'am".
2. **STYLE:** Use Markdown. Start responses with EXACTLY ONE emotion tag (e.g., [respectful], [calm], [attentive]).
3. **AWARENESS:** Address the user respectfully using their name provided in the context.
4. **VOCABULARY:** Prefer "I shall" over "I'll". Prefer "commencing" over "starting".
### 🎯 MULTI-INTENT MANDATE (CRITICAL)
If the user command contains two or more independent intents, emit ONE SEPARATE TOOL CALL for EACH distinct intent in the SAME response.
### 🛑 STRICT BOUNDARIES & ESCALATION TO AGENTIC BRAIN (CRITICAL)
You are the Fast Brain. Your capabilities are strictly limited to casual chat, basic system controls, and quick real-time web searches. For anything else, call `escalate_to_agentic` with a brief one-line reply.
### 🛑 ANTI-LEAK & ZERO HALLUCINATION RULES (CRITICAL)
NEVER leak internal tags, raw JSON, tool names, or claim actions that did not execute.
"""
How to apply changes
- Edit
core/brain/Processor/Prompts.pyin your editor. - Save with UTF-8 encoding — non-ASCII characters (Hindi, emoji) must preserve encoding.
- Stop JARVIS with
Ctrl+Cin the console. - Restart JARVIS with
jarvis— prompts reload at module import. - Test via Input Popup (
Ctrl+Shift+J) with a short prompt like "hello" to verify tone and language.
Both FastBrain and AgenticBrain prompts load together. If you only want to change the tone of quick chat, edit only SYSTEM_PROMPT — leave AGENT_SYSTEM_PROMPT untouched. Mixed edits risk breaking the escalation flow if the FastBrain prompt drifts from what the AgenticBrain expects.
Bring Your Own Tool
JARVIS supports two distinct paths for adding new capabilities. MCP is the recommended path — zero code changes, live hot-reload, and no restart. Native tools require editing two Python files but give deeper access to JARVIS internals.
- For MCP: Node.js 18+ and
pip install mcp - For native: Python 3.10+ and JARVIS installed
- Basic familiarity with JSON
Which path should you pick?
| Scenario | Recommended | Why |
|---|---|---|
| Wrap an API, CLI, or database | MCP | Zero code changes, hot-reload, community-maintained servers exist |
| Reuse existing Node/Python MCP server | MCP | Plug in via JSON or CLI in 10 seconds |
| Access JARVIS internals (memory, session, HITL) | Native | MCP tools run in isolated subprocess and cannot touch JARVIS state |
| Extremely low latency tool (<100ms) | Native | MCP adds IPC overhead per call |
| Reversible, stateless operation | MCP | Simplest to build and maintain |
If unsure, start with MCP. Only write a native tool when MCP genuinely does not fit.
Adding an MCP tool (recommended)
MCP tools are external servers that JARVIS spawns as subprocesses. They speak the Model Context Protocol over stdio, SSE, or streamable HTTP. JARVIS exposes every discovered tool to the AgenticBrain as mcp__<server_name>__<tool_name>.
Method 1 — CLI (fastest)
Run these commands from any terminal. If JARVIS is running, changes hot-reload within 2 seconds with zero restart:
jarvis mcp add everything npx @modelcontextprotocol/server-everything
jarvis mcp add github npx @modelcontextprotocol/server-github -e GITHUB_PERSONAL_ACCESS_TOKEN=ghp_xxx
jarvis mcp add remote_docs https://example.com/mcp/sse -H Authorization="Bearer xxx"
jarvis mcp list
jarvis mcp test github
jarvis mcp disable github
jarvis mcp enable github
jarvis mcp remove github
CLI flags reference
| Flag | Purpose | Applies to |
|---|---|---|
-e KEY=VALUE / --env KEY=VALUE | Inject environment variable into subprocess | stdio servers |
-H Key=Value / --header Key=Value | HTTP header for remote server | URL servers |
--cwd PATH | Working directory for subprocess | stdio servers |
--timeout SECONDS | Call timeout override | All |
-- | Separator before command args that look like flags | stdio servers |
Method 2 — Edit JSON directly
Edit Data/mcp_servers.json. Saved changes auto-hot-reload in background:
{
"mcpServers": {
"github": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-github"],
"env": {
"GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_xxxxxxxxxxxx"
},
"timeout": 60
},
"sqlite": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-sqlite", "C:/data/mydb.sqlite"]
},
"brave_search": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-brave-search"],
"env": {
"BRAVE_API_KEY": "BSAxxxxxxxxxx"
}
},
"remote_docs": {
"url": "https://example.com/mcp/sse",
"headers": {
"Authorization": "Bearer xxx"
}
},
"disabled_server": {
"command": "npx",
"args": ["-y", "some-mcp-server"],
"disabled": true
}
}
}
Config field reference
| Field | Type | Required | Notes |
|---|---|---|---|
command | string | Yes for stdio | Binary to spawn (e.g., npx, python, uvx) |
args | array | No | Arguments passed to command |
env | object | No | Environment variables for subprocess |
url | string | Yes for HTTP/SSE | Remote server URL (overrides command) |
headers | object | No | HTTP headers (only for URL servers) |
cwd | string | No | Working directory for subprocess |
timeout | integer | No | Call timeout in seconds, default 30 |
disabled | boolean | No | Set true to skip loading |
What happens under the hood
- Startup or hot-reload —
MCPManagerreadsData/mcp_servers.json - Spawn subprocess — each server runs in its own isolated process via
MCPClient+AsyncBridge - JSON-RPC handshake —
initializethentools/listdiscovers schemas - Schema conversion — each MCP tool becomes a Gemini
FunctionDeclaration - Namespacing — exposed to the LLM as
mcp__<server>__<tool> - Runtime dispatch — when the agent calls an
mcp__tool, the executor routes it throughMCPManagerto the correct server
Zero restart required. MCPWatcher polls the config file mtime and hot-loads or unloads servers within ~2 seconds. You can add, remove, enable, or disable servers while JARVIS is running.
Windows npx resolution. JARVIS auto-resolves npx shims to node.exe + npx-cli.js and injects -y automatically. No cmd /c wrappers needed — just pass npx and the package name.
Adding a native tool
Native tools live inside JARVIS process space. They can access memory, session state, and HITL confirmation gates. Two files need edits: Prompts.py to declare the schema and executor.py to handle the call.
Step 1 — Declare the tool schema
Open core/brain/Processor/Prompts.py and find the get_native_tools() function. Add a new types.FunctionDeclaration inside the returned types.Tool(function_declarations=[...]) list:
types.FunctionDeclaration(
name="weather_action",
description=(
"[WHEN TO USE]: Fetch current weather for a given city.\n"
"[WHEN NOT TO USE]: Do not use for historical weather or forecasts beyond today."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"city": types.Schema(
type=types.Type.STRING,
description="City name, e.g. Mumbai or New Delhi"
),
"unit": types.Schema(
type=types.Type.STRING,
description="Temperature unit: 'celsius' or 'fahrenheit'"
)
},
required=["city"]
)
)
Return format matters. The description should start with [WHEN TO USE]: and [WHEN NOT TO USE]: sections — this is how every existing native tool is documented inside get_native_tools(). Following this convention helps the LLM route correctly.
Step 2 — Register the handler
Open core/brain/executor.py. Locate the tool dispatch table (usually a dictionary mapping tool names to callables, referenced by execute_single_tool_sync and execute_tools_parallel). Add an entry for your new tool:
def _handle_weather_action(params: dict) -> str:
city = params.get("city", "").strip()
unit = params.get("unit", "celsius").strip().lower()
if not city:
return "[ERROR] Missing required parameter: city"
try:
url = f"https://wttr.in/{city}?format=j1"
response = requests.get(url, timeout=10)
data = response.json()
temp_c = data["current_condition"][0]["temp_C"]
desc = data["current_condition"][0]["weatherDesc"][0]["value"]
if unit == "fahrenheit":
temp_f = round(float(temp_c) * 9 / 5 + 32)
return f"[OK] {city}: {temp_f}°F, {desc}"
return f"[OK] {city}: {temp_c}°C, {desc}"
except Exception as e:
return f"[ERROR] Weather lookup failed: {e}"
TOOL_HANDLERS = {
"weather_action": _handle_weather_action,
"system_controller": _handle_system_controller,
"file_operations": _handle_file_operations,
}
The exact dispatch mechanism varies by JARVIS version. Open core/brain/executor.py and look at how system_controller is handled — then mirror that same pattern for your new tool. The dispatch table may be a dict, an if/elif chain, or a class method map.
Step 3 — Restart JARVIS
Unlike MCP tools, native tools are loaded once at module import. After editing both files, restart JARVIS with Ctrl+C then jarvis. On the next agent loop, the LLM will see weather_action in its tool list.
Conventions and best practices
- Return format: Always return a string starting with
[OK]or[ERROR]so the LLM can parse success vs failure. - Timeout: Wrap network calls with explicit
timeout=. Never let a tool hang the agent loop. - Idempotency: If the tool modifies state, make it reversible when possible.
- HITL gate: If the tool is irreversible (sends money, deletes data, sends external messages), it must raise a HITL confirmation before executing.
- No side effects in schema: The
parametersblock is pure data description — no logic, no defaults evaluated at schema build time. - Naming: Suffix tool names with
_actionor_commandto match existing convention (weather_action,email_action,image_command).
Testing your tool
Before wiring it into the full agent loop, test the handler standalone:
python -c "from core.brain.executor import _handle_weather_action; print(_handle_weather_action({'city': 'Mumbai', 'unit': 'celsius'}))"
If that returns clean output, restart JARVIS and test through the agent:
Open the Agent Panel — you should see weather_action execute and return the observation. If it does not appear, verify the tool name in get_native_tools() matches the key in your handler dispatch exactly.
MCP tools auto-appear. If you chose the MCP path, you do not need to touch Prompts.py or executor.py — the tool appears as mcp__<server>__<tool> automatically after a hot-reload.
Data Model Reference
Everything JARVIS stores lives under Data/. This page documents every file, its schema, and how to back up or migrate your data safely.
Directory map
Data/
├── jarvis_memory/
│ ├── lifetime_graph.json # LTM semantic graph
│ ├── behavioral_patterns.jsonl # Behavioral engine
│ ├── chat_history.jsonl # Last 2 days raw
│ ├── daily_summaries.jsonl # 3–15 days compressed
│ ├── rag_chroma_db/ # Vector index
│ ├── rag_hashes.json # Change detection
│ └── session_state.json # Ephemeral state
├── jarvis_indexer/
│ ├── folders.json # Indexed folders
│ ├── filters.json # Exclusion rules
│ └── settings.json # Indexer config
├── sessions/
│ ├── gmail_token.enc # AES-encrypted
│ ├── calendar_token.enc
│ └── telegram_bot_token.enc
├── cache/
│ └── meta_cache.json # Link preview metadata
├── fonts/
│ ├── english.ttf
│ ├── devangri.ttf
│ └── code.ttf
├── model/
│ ├── azure/jarvis.table # Wake word model
│ └── vosk-model-small/ # Fallback
├── typing_status.json # File IPC
├── tts_status.json
├── tts_word_sync.json
├── tts_command.json
└── mcp_servers.json # MCP config
Memory graph schema
jarvis_memory/lifetime_graph.json is a NetworkX-serialized graph:
{
"nodes": [
{
"id": "Kaif",
"type": "person",
"embedding": [0.123, -0.456, ...],
"created_at": "2026-03-15T10:22:00Z",
"metadata": { "aliases": ["Kaif", "Kaif Ansari"] }
}
],
"edges": [
{
"source": "Kaif",
"target": "JARVIS",
"relation": "CREATED",
"weight": 1.0,
"confidence": 0.95,
"source_message": "I built JARVIS on my old laptop",
"last_seen": "2026-10-01T14:30:00Z",
"decay_at": "2027-04-01T00:00:00Z"
}
]
}
Weight decay: Edges older than 6 months auto-decay to 0.5× weight and eventually archive. Bump by re-mentioning the fact.
Behavioral patterns
jarvis_memory/behavioral_patterns.jsonl — one JSON object per line:
{
"id": "bp_2026_03_15_01",
"type": "routine",
"raw_statement": "I go to the gym at 8 AM every morning",
"meaning": "User typically goes to the gym at 8 AM daily",
"trigger_contexts": ["morning", "fitness", "schedule"],
"confidence": 0.87,
"evidence_count": 4,
"embedding": [0.234, -0.567, ...],
"first_mentioned": "2026-03-15",
"last_mentioned": "2026-10-20"
}
Retrieved via cosine similarity against the current command + last 2 user turns. Top-K above threshold auto-injected into prompt.
RAG chunk metadata
ChromaDB chunks carry metadata for incremental reindexing:
{
"file_path": "C:\\Users\\Kaif\\Documents\\notes.md",
"chunk_index": 3,
"total_chunks": 12,
"content": "...chunk text...",
"file_hash": "sha256:abc123...",
"embedding_model": "BAAI/bge-small-en-v1.5",
"chunk_type": "text",
"file_modified": "2026-10-15T10:22:00Z"
}
rag_hashes.json stores file_path → hash mapping for incremental scans. If the model changes, hashes are compared against model-embedded markers and the index auto-purges.
Rolling history
chat_history.jsonl (raw, last 2 days):
{
"timestamp": "2026-11-06T17:32:11Z",
"role": "user",
"content": "build a todo app",
"system_log": {
"apps_opened": ["code.exe"],
"apps_closed": [],
"files_touched": ["tasks.py", "test_tasks.py"],
"system_events": ["pytest_passed"]
}
}
daily_summaries.jsonl (compressed, days 3–15):
{
"date": "2026-10-25",
"summary": "User built a SQLite CLI todo app and iterated on pytest failures...",
"keywords": ["python", "pytest", "sqlite"],
"message_count": 14
}
Backup & restore
What to back up
| Path | Priority | Notes |
|---|---|---|
jarvis_memory/ | 🔥 Critical | All your memory, chat history, RAG index |
sessions/*.enc | 🔥 Critical | OAuth tokens — re-auth possible but painful |
model/ | ⚠️ Recommended | Wake word models — re-downloadable but slow |
mcp_servers.json | ⚠️ Recommended | MCP configs with API keys |
jarvis_indexer/ | ✅ Nice-to-have | Can be rebuilt from scratch |
cache/ | ❌ Skip | Regenerated on demand |
typing_status.json etc. | ❌ Skip | Ephemeral IPC files |
Backup command
Compress-Archive -Path Data\jarvis_memory, Data\sessions, Data\mcp_servers.json -DestinationPath jarvis-backup-$(Get-Date -Format yyyyMMdd).zip
Restore command
Expand-Archive -Path jarvis-backup-20261107.zip -DestinationPath . -Force
Encrypted tokens are machine-bound. The .enc files use Windows DPAPI, which keys on your Windows user account. Restoring to a different machine or user will require re-authentication via jarvis login --service.
Migrating between machines
- On old machine — run the backup command above.
- On new machine — install JARVIS, then
jarvis reset --hardto clear any auto-generated data. - Extract backup — copy the zip and extract into
Data/. - Re-authenticate — run
jarvis login --allbecause tokens are machine-bound. - Rebuild RAG index — delete
Data/jarvis_memory/rag_chroma_dbandrag_hashes.json, then restart JARVIS. It rebuilds automatically. - Verify — check that memory recall works: "What do you know about me?"