v1.0 GitHub

Introduction

JARVIS is an open-source autonomous AI operating system for Windows. It reads codebases, sends emails, controls your Android phone, remembers you across months, and re-plans mid-task when you interrupt. This documentation covers everything from installation to advanced configuration.

sync

Live Correction + ask_user

Interrupt mid-run. Missing info? The agent pauses and asks.

memory

Lifelong Memory

Five-tier memory with weighted graph + hybrid RAG.

extension

MCP Native

18,000+ community servers via a single JSON entry.

record_voice_over

Dual-Engine Voice

Azure on-device + Vosk fallback, zero cloud round-trip.

developer_board

4GB RAM

Verified on a 2012 Intel i5-3210M laptop.

phone_android

Android Control

WebSocket bridge — calls, SMS, GPS, torch, clipboard.

Who is this for?link

JARVIS is built for developers and power users who want an AI that actually does things — not just chats. If you want to:

  • Automate real work across your PC, phone, and communications
  • Run an AI agent on modest hardware (no GPU required)
  • Extend capabilities without touching code (via MCP)
  • Keep all memory and data on your own disk
  • Use any OpenAI-compatible LLM, including local models

Then this is for you. If you need a polished IDE copilot, use Claude Code or Cursor.

info

Windows-only. JARVIS relies on Windows APIs for GUI automation, PC monitoring, and the Termux bridge. Mac/Linux support is on the roadmap.

translate

About the voice examples. JARVIS responds in your language. Some voice commands and agent responses in these docs appear in Hinglish (Hindi + English) — that's because the developer is Indian and this is the language he actually uses daily. You can configure JARVIS to respond in any language you prefer.

Quick Start

Get JARVIS running in under five minutes with the 1-click auto-installer. It handles Git, Python 3.11, Node.js, the virtual environment, the WhatsApp bridge, and the global CLI — all automatically, including installing missing prerequisites via winget.

checklist
Before you start
  • Windows 10 or 11
  • PowerShell (any version, no admin required)
  • ~2 GB free disk
  • Internet connection
  • One LLM API key (Groq, OpenRouter, or OpenAI)

1-Click Install (Recommended)link

Run this single command from PowerShell. The installer auto-installs any missing prerequisites, clones the repo to %USERPROFILE%\Jarvis-OS-Agent, sets up the Python and Node environments, registers the global jarvis CLI, and interactively prompts for your API keys.

terminalpowershell
Set-ExecutionPolicy Bypass -Scope Process -Force; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; & ([scriptblock]::Create((Invoke-RestMethod -Uri 'https://github.com/thekaifansari01/Jarvis-OS-Agent/releases/latest/download/install.ps1')))
info

First run takes 5–10 minutes. The installer downloads ~40 Python packages plus a local embedding model (~130 MB). Subsequent runs are near-instant. Windows may show a UAC prompt for the Visual C++ Redistributable — accept it.

Launch JARVISlink

The installer prompts for API keys at the end of setup. Once it finishes, open a new terminal (the current one won't have the updated PATH) and run:

terminalpowershell
jarvis

You should see the ASCII banner and a status line confirming voice input is enabled. Say "Jarvis" or hit Ctrl+Shift+J to summon it.

check_circle

Done. The next sections cover each subsystem in detail. If you hit an error, jump to Troubleshooting.

info

Prefer manual setup? See the Installation page for step-by-step instructions if you need a custom install path, are offline, or the 1-click installer failed.

Installation

Two paths: the 1-click auto-installer (recommended for most users), or manual setup (for custom paths, offline installs, or troubleshooting).

One-command installer (Recommended)link

This is the fastest and recommended path. Run this single command from PowerShell — it handles every dependency automatically, including installing Git, Python 3.11, and Node.js via winget if missing.

terminalpowershell
Set-ExecutionPolicy Bypass -Scope Process -Force; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; & ([scriptblock]::Create((Invoke-RestMethod -Uri 'https://github.com/thekaifansari01/Jarvis-OS-Agent/releases/latest/download/install.ps1')))
info

First run takes 5–10 minutes. Windows may show a UAC prompt for the Visual C++ Redistributable — accept it. Subsequent re-runs are near-instant and safe to execute (the installer updates in place without wiping your data).

What the installer doeslink

  • Prerequisites — auto-installs Git, Python 3.11, Node.js, and the Visual C++ Redistributable via winget if missing (falls back to direct downloads)
  • Repository — clones to %USERPROFILE%\Jarvis-OS-Agent, or updates in place if already present
  • Python environment — creates .venv with Python 3.11 and installs ~40 packages including CPU-optimized PyTorch
  • WhatsApp bridge — installs the Node.js Baileys bridge dependencies
  • Configuration — copies .env.example to .env and interactively prompts for API keys
  • Global CLI — registers the jarvis command via SetupRegistry.py and adds .venv\Scripts to your user PATH
check_circle

Safe to re-run. The installer detects existing installs and updates in place without wiping your .env, memory graph, or MCP config.

Manual installation (Advanced)link

Use this only if you need a custom install path, are offline, or the 1-click installer failed. Steps below assume you already have Git, Python 3.10+ with pip, and Node.js 18+ with npm installed and on PATH.

1. Clone the repositorylink

terminalpowershell
git clone https://github.com/thekaifansari01/Jarvis-OS-Agent.git
cd Jarvis-OS-Agent

2. Create virtual environmentlink

JARVIS installs ~40 Python packages. Always use a virtual environment to avoid conflicts with your global Python.

terminalpowershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
warning

If PowerShell blocks the activation script, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned in an elevated terminal.

3. Install Python packageslink

terminalpowershell
pip install --upgrade pip
pip install -r requirements.txt
pip install mcp

This installs the core runtime plus optional dependencies like Flask (RAG dashboard), psutil (PC monitoring), and Pillow (vision).

4. Install WhatsApp bridgelink

The WhatsApp integration is powered by a Node.js server using Baileys. Skip this step if you don't need WhatsApp.

terminalpowershell
cd tools/Messanger/whatsapp/BaileysServer
npm install
cd ../../../..

5. Register global CLIlink

This makes the jarvis command available system-wide. Run it from the repo root with the virtual environment active:

terminalpowershell
python SetupRegistry.py

Verify installationlink

Whether you used the 1-click installer or manual steps, open a new terminal (existing terminals won't have the updated PATH) and run:

terminalpowershell
jarvis --help

You should see the full CLI help with all available commands. If you see "jarvis is not recognized", close and reopen the terminal, or check Troubleshooting.

Enable MCP (optional)link

To add external tools via the Model Context Protocol, create Data/mcp_servers.json. See the MCP section for full details.

Uninstall

Complete removal of JARVIS with a single PowerShell command. The uninstaller stops all background services, cleans the Windows registry and PATH, removes the global CLI, and deletes the install folder. It also prompts to back up your .env API keys, Lifelong Memory Graph, and MCP config to the Desktop before deleting anything.

checklist
Before you start
  • PowerShell (elevated not required, but recommended)
  • Close all terminals running JARVIS
  • Backup decision ready — uninstaller will ask

Run the uninstallerlink

Open PowerShell and paste this single command. The script downloads fresh from the main branch and runs in-process:

terminalpowershell
Set-ExecutionPolicy Bypass -Scope Process -Force; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; & ([scriptblock]::Create((Invoke-RestMethod -Uri 'https://github.com/thekaifansari01/Jarvis-OS-Agent/releases/latest/download/uninstall.ps1')))
warning

Backup prompt is default Yes. When the uninstaller finds .env, Data/jarvis_memory/, or Data/mcp_servers.json, it will ask to back them up to Desktop\Jarvis_Backup_yyyyMMdd_HHmmss\. Press Enter to accept the backup, or n to skip (data will be permanently deleted).

What gets removedlink

  • Running processes — kills all JARVIS-related Python/Node processes, plus the jarvis.lock file
  • User PATH entries — removes ~/.local/bin, WindowsApps shims, and any entry containing jarvis-os-agent
  • CLI shims — jarvis.cmd, jarvis.bat, jarvis.ps1 from common install paths
  • Registry keys — App Paths\jarvis.* and HKCU:\Software\Classes\jarvis
  • Install folder — %USERPROFILE%\Jarvis-OS-Agent including .venv, node_modules, and codebase
  • HuggingFace cache — ~/.cache/huggingface/hub/models--BAAI--bge-small-en-v1.5 (~130 MB, optional prompt)
  • Documents media — Documents\Jarvis (WhatsApp downloads, shared files) — optional prompt

What gets backed up firstlink

save

Before deleting the install folder, the uninstaller prompts to copy these to Desktop\Jarvis_Backup_yyyyMMdd_HHmmss\:

FileWhat it holds
.envAll your API keys and provider config
Data\jarvis_memory\Lifelong memory graph, behavioral patterns, RAG index, chat history
Data\jarvis_indexer\Indexer folder list, filters, and settings
Data\mcp_servers.jsonMCP server configs (including any tokens/keys)
info

Encrypted tokens are machine-bound. Session tokens (sessions/*.enc) use Windows DPAPI and are not backed up — they won't work on a different machine anyway. You'll need to re-authenticate via jarvis login --all after a fresh install on another PC.

What happens step by steplink

  1. Stopping processes — scans Win32_Process for anything with executable path or command line inside %USERPROFILE%\Jarvis-OS-Agent and kills them (2s wait for handles to release).
  2. Backup prompt — if .env, memory, or MCP config exists, asks to back up to Desktop (default Yes).
  3. Registry + PATH cleanup — removes User PATH entries matching JARVIS folders or jarvis-os-agent, deletes CLI shims, and cleans App Paths + Classes\jarvis registry keys.
  4. Core folder deletion — uses Remove-Item -Recurse -Force with read-only bit clearing. Falls back to cmd /c rmdir /s /q for stubborn locked files.
  5. External media prompt — asks whether to also delete Documents\Jarvis (default No).
  6. Embedding cache prompt — asks to remove ~/.cache/huggingface/hub/models--BAAI--bge-small-en-v1.5 (~130 MB, default No).
  7. Success banner — "J.A.R.V.I.S. OS Agent Uninstalled Successfully!"

Troubleshootinglink

Some files in Jarvis-OS-Agent couldn't be deletedadd
A process is still holding the files (usually python.exe or node.exe). Close all terminals and editors (VS Code, Notepad++) that have files open inside that folder. Then manually delete %USERPROFILE%\Jarvis-OS-Agent via File Explorer — the uninstaller already cleaned everything else.
jarvis command still works after uninstalladd
Open a new terminal — the old one still has the stale PATH in memory. If it still runs, check Get-Command jarvis to find the shim location and delete it manually.
Windows Defender blocked the scriptadd
Antivirus flags Invoke-RestMethod + scriptblock patterns sometimes. Add PowerShell to the exclusion list temporarily, or download the raw script manually and run it with & .\uninstall.ps1.
I skipped the backup by mistakeadd
If the install folder is already gone, your .env and memory graph are unrecoverable — they were stored inside that folder. Next time, accept the backup prompt (default Yes) before proceeding.

Environment Configuration

All API keys and configuration live in a single .env file at the repo root. Copy .env.example to .env and fill in the values you need.

checklist
Before you start
  • Text editor (VS Code recommended)
  • API key from at least one provider (Groq, OpenRouter, Gemini, or Ollama)
  • Optional: Deepgram + Tavily keys for voice + search

Required variableslink

VariablePurpose
PRIMARY_API_KEYPrimary LLM provider key (any OpenAI-compatible endpoint)
PRIMARY_BASE_URLEndpoint, e.g. https://api.groq.com/openai/v1
PRIMARY_MODELModel name, e.g. llama-3.3-70b-versatile
FALLBACK_API_KEYFallback provider key (used if primary fails)
FAST_BRAIN_API_KEYFastBrain key (can be the same provider)
ROUTER_API_KEYSemantic router key
LTM_EXTRACTION_API_KEYLong-term memory extraction
TAVILY_API_KEYReal-time web search
DEEPGRAM_API_KEYSpeech-to-text

Optional variableslink

The following are only needed for the integrations you actually use:

  • Voice: AZURE_SPEECH_KEY, AZURE_SPEECH_REGION, AZURE_KEYWORD_MODEL_PATH
  • Telegram: TELEGRAM_API_ID, TELEGRAM_API_HASH
  • Image generation: IMAGE_GEN_API_KEY, IMAGE_GEN_ENDPOINT
  • TTS: TTS_API_KEY (falls back to Edge TTS if absent)

Using local LLMslink

For 100% local execution, point both providers at your local OpenAI-compatible server:

description.env (Ollama example)
PRIMARY_API_KEY=ollama
PRIMARY_BASE_URL=http://localhost:11434/v1
PRIMARY_MODEL=llama3.1

FALLBACK_API_KEY=ollama
FALLBACK_BASE_URL=http://localhost:11434/v1
FALLBACK_MODEL=llama3.1
info

Both PRIMARY_BASE_URL and FALLBACK_BASE_URL must end with /v1. This is the standard OpenAI-compatible route.

Supported providerslink

ProviderBase URLNotes
Groqhttps://api.groq.com/openai/v1Free tier, very fast
OpenRouterhttps://openrouter.ai/api/v1Many models, one key
Regolohttps://api.regolo.ai/v1European hosting
Geminihttps://generativelanguage.googleapis.com/v1beta/openaiUse OpenAI-compat route
Ollamahttp://localhost:11434/v1Fully local
LM Studiohttp://localhost:1234/v1Fully local, GUI
vLLMhttp://localhost:8000/v1Self-hosted, GPU

Voice Setup

JARVIS ships with dual-engine wake word detection, premium STT, and low-latency TTS. Everything runs on-device by default — no cloud round-trip for wake word.

checklist
Before you start
  • Working microphone connected
  • Deepgram API key for STT
  • Optional: Azure Speech account (for higher accuracy wake word)
  • Optional: TTS API key (Sarvam) — falls back to Edge TTS

Wake word engineslink

Two independent engines, auto-selected on boot:

EngineWhen activeAccuracy
Azure Custom KeywordIf Data/model/azure/jarvis.table exists97%+
Vosk KWSFallback if Azure unavailable~66%

Building an Azure wake modellink

This is a one-time 30-minute process. After training, no API key or network call is needed for wake word.

  1. Create a Foundry resourceGo to Azure Portal → Foundry → Speech. Copy your key and region.
  2. Train a Custom KeywordOpen Speech Studio → Custom Keyword → Create project (English US).
  3. Choose your modelBasic (~30 min queue) or Advanced (~48h for better accuracy).
  4. Download the .table fileExtract the .table from the zip and save as Data/model/azure/jarvis.table.
  5. Restart JARVISAzure engine activates automatically. Check startup logs for confirmation.
info

No Azure subscription? Vosk works out of the box. It auto-downloads a small model on first run — no config required.

Speech-to-textlink

STT uses Deepgram Nova-2 with a self-healing LTM vocabulary cache. Entity names from your memory graph are auto-synced into Deepgram's vocabulary, so it correctly recognizes names, projects, and jargon over time.

check_circle

Zero training needed. Add someone to your memory and Deepgram starts recognizing their name within a session.

Text-to-speechlink

Default TTS uses Sarvam for English + Hinglish (low-latency). Falls back to Edge TTS if the API key is missing. First audio typically lands in 0.8–2 seconds.

Memory & RAG

JARVIS has five tiers of memory — from verbatim message history to long-term semantic graphs and hybrid RAG over your documents.

The five tierslink

#TierWhat it storesRetention
1EpisodicRaw messages + execution logs15 days (2 raw + 13 summarized)
2SemanticWeighted property graph (facts, people, contacts)Lifetime, with decay
3RAGIndexed documents via hybrid BM25 + vectorUntil deleted
4Ephemeral10-min TTL artifacts (last screenshot, generated image)Auto-purged
5BehavioralPatterns extracted from your habits & routinesLifetime, with decay

Unified Portal — Knowledge Indexerlink

JARVIS ships a single Unified Portal at one URL that serves all web UIs — Knowledge Indexer, Mobile Bridge, and more — as tabs. Open it after launching:

languagebrowser
http://127.0.0.1:8000

Switch to the Knowledge Indexer tab in the portal sidebar. On first boot, the indexer auto-adds your Documents folder. You can add more from the Folders sub-tab — paste any absolute path and click Add & Index.

info

Internal ports still active. Behind the portal, the Indexer API runs on 127.0.0.1:8080 and the Mobile Bridge on 127.0.0.1:8090. The portal aggregates both — you only need to remember 8000.

Adding a new folderlink

  1. Open the Unified Portal at http://127.0.0.1:8000.
  2. Click the Knowledge Indexer tab in the sidebar.
  3. Go to the Folders sub-tab.
  4. Paste an absolute path, e.g. D:/Projects or C:\Users\You\Desktop\notes.
  5. Click Add & Index. Progress streams live.
warning

System paths are auto-blocked. Paths like C:\Windows, Program Files, and dev garbage (node_modules, .git, venv) are skipped automatically. Sensitive paths like .ssh and .aws trigger warnings.

Filters & exclusionslink

Open the Filters tab to tune indexing:

  • Max file size — default 5 MB. Larger files are skipped.
  • Max depth — how deep to recurse.
  • Custom ignore patterns — e.g. *.log, backup_*, node_modules.

Rebuilding the indexlink

If embeddings feel stale or you changed the model:

terminalpowershell
# Purge the vector index
Remove-Item -Recurse -Force Data\jarvis_memory\rag_chroma_db
Remove-Item Data\jarvis_memory\rag_hashes.json

# Restart JARVIS to rebuild automatically
jarvis

MCP Integration

The Model Context Protocol lets JARVIS plug into 18,000+ community-built tool servers — GitHub, Brave Search, SQLite, Postgres, Notion, Slack, Docker, Puppeteer, and more — via a single JSON file.

checklist
Before you start
  • Node.js 18+ (24+ recommended for newest servers)
  • pip install mcp completed
  • API keys only if the chosen MCP server requires them

How it workslink

When Data/mcp_servers.json exists with valid configs, JARVIS:

  • Spawns each configured MCP server as a subprocess
  • Performs a JSON-RPC handshake to discover tools
  • Converts tool schemas into native function declarations
  • Exposes them to the agent as mcp__<server>__<tool>

One-liner CLI & live hot-reloadlink

Add, remove, test, or toggle MCP servers directly from your terminal. If JARVIS is running, changes are hot-reloaded in the background within 2 seconds with zero restarts:

terminalpowershell
# Add an NPX or Python MCP server
jarvis mcp add everything npx @modelcontextprotocol/server-everything

# Add a server with environment variables
jarvis mcp add github npx @modelcontextprotocol/server-github -e GITHUB_PERSONAL_ACCESS_TOKEN=ghp_xxx

# List, test, disable, enable, or remove servers live
jarvis mcp list
jarvis mcp test everything
jarvis mcp disable everything
jarvis mcp enable everything
jarvis mcp remove everything

JSON configuration examplelink

You can also edit Data/mcp_servers.json directly (saved changes auto-sync live):

descriptionmcp_servers.json
{
  "mcpServers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_xxxxxxxxxxxx"
      },
      "call_timeout": 60
    },
    "sqlite": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-sqlite", "C:/data/mydb.sqlite"]
    }
  }
}

Startup or live hot-reload logs confirm the active tools immediately:

terminalstartup log
MCP: server 'github' connected (40 tools).
MCP: server 'sqlite' connected (6 tools).
MCP: ready with 46 tools from 2 servers.

Native Windows resolutionlink

info

JARVIS automatically resolves npx shims on Windows directly to node.exe + npx-cli.js and injects -y automatically — no cmd /c wrapper needed.

ServerWhat it addsAPI key?
GitHubRepos, issues, PRs, searchPersonal access token
Brave SearchWeb, news, local searchFree tier (2k/mo)
SQLiteNatural-language DB queriesNone
PostgresDB queries on live databasesConnection string
NotionPages, databases, commentsIntegration token
SlackChannels, messages, usersBot token
DockerContainer lifecycleNone
PuppeteerHeadless browser automationNone
FetchURL fetch + readabilityNone
info

Skip redundant servers. Filesystem MCP overlaps with native file_operations. Memory MCP overlaps with the built-in 5-tier memory. Puppeteer overlaps with gui_controller. Only add servers that introduce net-new capabilities.

Disabling MCPlink

MCP is opt-in by design. Delete Data/mcp_servers.json or set it to { "mcpServers": {} }. JARVIS boots identically to the pre-MCP version with zero overhead.

Android Bridge

Control your Android phone natively from Windows using Termux and a WebSocket bridge. No SSH required, no cloud relay.

checklist
Before you start
  • Android phone with Termux + Termux:API (from F-Droid)
  • Both devices on the same WiFi network
  • JARVIS running on your PC

Setuplink

  1. Install Termux and Termux:API on your phone via F-Droid (not Google Play — that version is outdated).
  2. Connect both devices to the same WiFi. The bridge runs over local network only.
  3. Open the Unified Portal on your PC: http://127.0.0.1:8000 → click the Mobile Bridge tab.
  4. Scan the QR code with your phone, or copy the setup command into Termux.
  5. Approve Termux:API prompts as they appear — SMS, contacts, storage, location.

What you can dolink

CategoryExamples
FilesDrag & drop files, receive via Android Share menu
TelecomMake calls, read/send SMS, ring phone
SensorsGPS location, battery level, storage, torch
ClipboardSync clipboard both ways
PhotosFetch latest photo, screenshot

Persistent background modelink

The bridge runs a wake-lock that keeps it alive even when your phone is locked. It survives app switches and network changes.

security

LAN-only. The bridge binds to 127.0.0.1:8090. It is never exposed to the internet. If you need remote control, use the Telegram bot instead.

Telegram Bot

Trigger silent background PC commands from anywhere via Telegram. Perfect for when you're away from your desk.

checklist
Before you start
  • Telegram account on your phone
  • BotFather bot created (saves token)
  • JARVIS stopped for activation

Setuplink

  1. Create a bot — message @BotFather on Telegram, send /newbot, follow the prompts, and copy the token.
  2. Stop JARVIS on your PC. The bot activation command requires it to be offline.
  3. Run activation: jarvis bot --activate
  4. Paste your BotFather token when prompted.
  5. Start JARVIS again. Send any message to your bot on Telegram to confirm.

Remote commandslink

Once connected, you can message your bot on Telegram the same way you'd talk to JARVIS at your desk. Examples:

  • "Open VS Code"
  • "What's my CPU usage?"
  • "Summarize my last 3 emails"
  • "Take a screenshot and send it"
  • "Lock the PC"

Commands run silently in the background — your active desktop session is untouched.

Status & controllink

terminalpowershell
jarvis bot --status      # show live status
jarvis bot --deactivate  # revoke & offline the bot

CLI Reference

All commands are available globally after running SetupRegistry.py once.

warning

Stop JARVIS before running memory, login, or reset commands. Running them while JARVIS is active may corrupt state.

Authenticationlink

CommandAction
jarvis login --whatsappWhatsApp Web QR authentication
jarvis login --telegramTelegram phone + OTP
jarvis login --mailGmail OAuth2
jarvis login --calendarGoogle Calendar OAuth2
jarvis login --allBatch authenticate all integrations

Logoutlink

CommandAction
jarvis logout --whatsappDestroy WhatsApp session
jarvis logout --telegramDestroy Telegram session
jarvis logout --mailRevoke Gmail tokens
jarvis logout --calendarRevoke Calendar tokens
jarvis logout --allRevoke everything

MCP server management (live hot-reload)link

CommandAction
jarvis mcp listList all configured MCP servers and their status
jarvis mcp add <name> <cmd/url> [args...] [-e K=V]Add or update an MCP server (hot-reloads live if JARVIS is ON)
jarvis mcp remove <name>Remove an MCP server and unload its tools immediately
jarvis mcp enable <name>Enable a disabled MCP server live
jarvis mcp disable <name>Temporarily disable an MCP server without deleting config
jarvis mcp test <name>Test connection to a server and list discovered tools

Remote botlink

CommandAction
jarvis bot --activateConfigure & activate Telegram bot (JARVIS must be OFF)
jarvis bot --deactivateRevoke & offline the bot (JARVIS must be OFF)
jarvis bot --statusShow live bot status

Memory & resetlink

CommandAction
jarvis memory --clearPurge contextual memory (keeps sessions)
jarvis reset --hardFactory reset — wipes memory, vectors, sessions
jarvis --helpGlobal help

Tools Reference

The agent chooses from a library of native tools. Every tool is namespaced, permissioned, and observable in the console.

Tool categorieslink

CategoryExamples
Software Engineeringrepo_map, replace_block, run_python_code, AST lint, create_many
CommunicationGmail (send/read), WhatsApp, Telegram, Google Calendar
WorkspaceFile CRUD, recursive scanning, hybrid RAG
MobileTermux API — calls, SMS, battery, torch, sensors
Remote ControlTelegram bot for silent background commands
Search & ResearchWeb search, ArXiv, YouTube transcripts
System AutomationApp launch/kill, volume, brightness, clipboard
GUI AutomationOpenCV edge-vision Set-of-Mark, mouse & keyboard
PC MonitoringCPU, RAM, disk, battery, network, USB
MultimodalScreen analysis, object detection, OCR
Image GenerationText-to-image, image-to-image editing
MCP External18,000+ community servers

Architecture

JARVIS is resilient by design. Every layer has a fallback; every provider has a failover; every tool has a watchdog.

Dual-brain routinglink

A semantic router picks the right brain for each command:

BrainWhenLatency
FastBrainShort commands, OS toggles, casual chat<2s
AgenticBrainComplex engineering, comms, memory, visionMulti-step

3-tier fallback chainlink

  1. Tier 1 — Semantic Router Cloud-based; classifies intent via a fast LLM.
  2. Tier 2 — FastBrain If the router API times out, FastBrain handles routing.
  3. Tier 3 — Local rule-based If both fail, keyword + heuristic rules take over.

Provider failoverlink

Every LLM call tries PrimaryProvider first. On failure (timeout, rate limit, 5xx), it retries FallbackProvider — a completely separate OpenAI-compatible endpoint. Configure both in .env.

ServiceWatchdoglink

Monitors and auto-restarts:

  • STT Popup, Baileys WhatsApp server
  • Mobile Bridge (:8090), Telegram bot
  • Indexer Dashboard (:8080), MCP Manager
  • Unified Portal (:8000) — gated on both backends being healthy

Crashes trigger a cooldown backoff. Unauthenticated services are skipped, not restarted endlessly. The Unified Portal will not restart if either backend is down.

Agent Panel

Live boltZMQ Powered verified_userHITL Gates palette16 Orb Modes

The Agent Panel

The Agent Panel is JARVIS's floating command center — a premium always-on-top overlay that shows exactly what the agent is thinking, which tool it's running, how many tokens it has consumed, and when it needs your permission before doing something irreversible. It is not a chat window. It is a live observability layer over the autonomous loop.

psychology

Live Thought Stream

Watch reasoning unfold in real-time with syntax-highlighted markdown.

verified_user

HITL Consent Gates

Approve, edit, or cancel every WhatsApp, Telegram, or Email before it sends.

sync

Non-Blocking

Panel never interrupts the agent — it's read-only until a decision is required.

palette

Action-Aware Theming

Every tool has its own icon, color, and pulse animation.

info

Read-only by design. The Agent Panel is not a chat window — you can't type into it. Commands come from voice, Ctrl+Shift+J, or the Telegram bot. The panel observes and, when needed, asks for approval.

Agent Panel showing live thinking state with purple status orb, thought stream, step counter, and token count
LiveAgent Panel in active thinking state — Step 03, 1.2k tokens, purple orb

When It Appearslink

  • Auto-spawns on every agent loop start — voice command, Ctrl+Shift+J, Telegram bot, MCP tool call
  • Stays visible while hide_timer is active (4-second idle cooldown)
  • Fades out after 4 seconds of no activity — unless awaiting confirmation
  • Reappears instantly on next status update
  • Never blocks the underlying desktop — it's Qt.Tool, no taskbar entry, no focus steal
  • Bypasses the 4-second timer entirely when a confirmation is pending
warning

The panel cannot be "closed" manually while the agent runs — it dismisses itself. To hide permanently, stop JARVIS from the CLI.

The Stateslink

The panel cycles through four visual states, each with its own orb color, status text, and animation:

Anatomy of the Panellink

Seven distinct regions compose the panel, each with a specific function:

Agent Panel anatomy with numbered pins overlaid on screenshot 1 2 3 4 5 6 7
  • 1Status Orb
  • 2Status Tag
  • 3Detail Pill
  • 4Meta Capsule
  • 5Thought Stream
  • 6Observation Card
  • 7Preview Card
RegionPurposeUpdates
Status Orb24×24 icon reflecting current actionPer action change
Status TagHuman-readable label ("Thinking", "Gmail")Per action change
Detail PillElided tool name (e.g., "whatsapp_send")Per tool call
Meta CapsuleStep counter + token countPer step
Thought StreamMarkdown-rendered reasoningThrottled at 90ms
Observation CardCollapsible tool result (max 90px)Per tool result
Preview CardHITL confirmation UIPer confirmation

The Status Orb — 16 Action Iconslink

Every tool type has its own visual identity. The orb changes icon and color to reflect what's currently happening:

blur_on
Thinking
Initial planning
search
Search
search_actions
manage_search
Deep Task
deep_research
terminal
Terminal
execute_terminal
code
Python
run_python_code
mouse
GUI Control
gui_controller
hub
Memory
memory_actions
folder
Files
file_operations
phone_android
Mobile
mobile_action
swap_horiz
Transfer
file_transfer_action
image
Image Gen
image_command
content_paste
Clipboard
clipboard_action
calendar_today
Calendar
calendar_action
desktop_windows
Workspace
system_controller
mail
Comms
communication
check_circle
Done
step = 0
info

Cached by RGB bucket + DPR. Static icons (search, python, etc.) are pre-rendered as QPixmaps for near-zero paint cost. Only thinking and idle are re-rendered live because they animate the halo pulse.

Color Palette — Action-Coded Theminglink

The entire panel — border gradient, ambient glow, observation bar, orb — shifts to a per-action color. This palette is defined in Theme.py as ACCENT_PALETTE:

Search Deep Task Python Thinking Workspace Comms WhatsApp Telegram GUI Mobile File Ops Terminal Clipboard Image Gen Calendar Memory
auto_awesome

Smooth crossfades. Colors transition over ~1 second at 45ms per tick (18% per step). When the task finishes, the palette drifts back to neutral grey (idle = [130,140,165]).

The Thought Streamlink

The main body of the panel renders the agent's live reasoning as formatted markdown. It supports:

  • Fenced code blocks with syntax highlighting — Python, JS, SQL, C++, Go, Rust, Shell (auto-detected regex)
  • Tables with alignment detection
  • Headings, bold, italic, strikethrough
  • Inline code chips and clickable links (open in external browser)
  • Bullet + numbered lists with nested indentation
  • Blockquotes (nested depth supported)
  • Checkbox lists — [ ] → ☐, [x] → ☑
info

Scroll lock behavior. Auto-scrolls to bottom unless you've manually scrolled up. Scrolling up "locks" the view; new content won't pull you down until the step counter increments. After 5,000 chars, older content is safely trimmed — mid-code-fence and mid-table cuts are auto-healed.

The Observation Cardlink

When a tool returns an observation string, the panel shows a dedicated card below the thought stream:

  • Max height: 90px (elided to 155 chars)
  • Accent-colored left bar matches current action color
  • Auto-hides when next thought arrives or task completes
  • Separated from thought stream by a hairline divider
Observation card showing test result summary with accent-colored left border
ResultTool observation — 12 tests passed, accent-bordered

Human-in-the-Loop Confirmationlink

This is the trust anchor. When the agent needs to send an irreversible action — WhatsApp message, Telegram, Email, or destructive file operation — the panel raises a premium confirmation card that cannot be bypassed.

gpp_maybe

Never bypassed. Even if the user says "just do it" in the voice prompt, email_action, whatsapp_action, and telegram_action always raise the confirmation card. The agent cannot auto-approve its own outgoing messages. This is enforced at the ZMQ boundary, not in the prompt.

Actions that trigger the gate

PlatformToolIrreversible?Requires Confirm
WhatsAppwhatsapp_actionYes — real contact✅
Telegramtelegram_actionYes — message/call✅
Emailemail_actionYes — Gmail OAuth✅
File deletefile_operations (delete)Yes — disk-level✅
Terminalexecute_terminal_command (rm -rf)Yes — blocked/asked✅
Calendarcalendar_action (create/delete)Yes — external event✅
Image genimage_commandDepends on provider⚠️ Varies

Anatomy of the confirmation card

WhatsApp confirmation card with badge, recipient, editable body, and Send/Cancel buttons 1 2 3 4 5
  • 1Platform Badge
  • 2Recipient Row
  • 3Body Editor
  • 4Cancel Button
  • 5Send Now Button

Each platform gets its own themed badge color:

What you can edit

FieldEditablePersisted to Agent
Recipient (To:)❌ No—
Subject (email)✅ Yes✅ Yes
Body✅ Yes✅ Yes
Attachments❌ No—
Platform❌ No—
warning

Recipients are locked. You can't redirect a message to a different person from the panel — this prevents prompt-injection attacks from hijacking outgoing comms. To change recipient, cancel and re-issue the command.

Keyboard shortcuts during confirmation

Esc → Cancel & reject
Ctrl + Enter → Approve & send
info

Focus steal on arrival. When a confirmation appears, the panel steals focus to the body editor — you can start typing edits immediately. This is the only time the panel takes focus.

The confirmation queue

  • If a second confirmation arrives while one is open, it's appended to a FIFO queue
  • Once you decide on the first, the next slides in immediately — no flash, no wait
  • No hard limit — but in practice agents rarely issue more than 2 in a row
  • Queue is in-memory only — a panel restart clears the queue
  • Queued confirmations are not auto-approved if ignored — they stay pending
timer

90-second expiry. Each confirmation has a 90-second timeout enforced by the agent (not the panel). If you don't decide in time, the agent autonomously aborts and logs HITL TIMEOUT. Approve fast or re-run the command.

Controls & Shortcutslink

Dragging

  • Click and drag anywhere in the top 62px header zone
  • Not draggable from buttons, text areas, scrollbars, or links
  • Snaps to screen edges — clamped to primary screen bounds
  • Position persists across panel show/hide within the session

Resizing

  • 9px grab-zone on all four edges
  • 22px corner-zone for diagonal resize
  • Min size: 440 × 160
  • Max size: 1280 × 860
  • Live layout — thought stream re-flows as you drag

Reset to default

Double-click header → Restore 580 × 196, re-center

Auto-hide

  • 4-second cooldown after last status update
  • Suppressed when awaiting confirmation, dragging, resizing, or hovering
  • Fades out over 190ms and slides up 20px
  • RGB timer stops on hide — zero background CPU

Architecture — ZMQ Bridgelink

The panel communicates with the agent loop over ZeroMQ sockets. Three independent channels handle the three message types:

Status Agent PUB :5555 → Panel SUB PUB/SUB · fire-and-forget
Confirm In Agent PUSH :5556 → Panel PULL PUSH/PULL · reliable
Confirm Out Panel PUSH → Agent PULL :5557 PUSH/PULL · decision reply
PortDirectionProtocolPurpose
5555Agent → PanelPUB/SUBStatus updates — self-healing if panel restarts
5556Agent → PanelPUSH/PULLConfirmation requests — reliable delivery
5557Panel → AgentPUSH/PULLConfirmation decisions — approve/cancel + edits
info

Auto-recovery. The listener automatically retries as the opposing socket type on EADDRINUSE — meaning the panel and agent can start in any order and still connect.

speed

Message batching. The listener drains up to 100 status messages per 25ms poll cycle and only emits the latest. Even a firehose of thought tokens never floods Qt's event queue — the panel stays at 60fps.

Performance & Premium Detailslink

TechniqueWhat it doesImpact
Island static pixmap cacheShell (shadows, gradient, shine) pre-rendered once per size + color bucket90% paint cost eliminated on static frames
Border-only repaintsDuring RGB crossfades, only the 1px border region is invalidatedConstant 60fps during color transitions
Thought throttling90ms debounce on markdown renderingToken streams don't jank
Icon pixmap cache16 mode icons cached by RGB bucket + DPRZero repaint cost for static orbs
Markdown 4-layer cacheHTML, line, code-line, highlight all memoizedRenders 50× faster on repeat content
Smooth scroll animation110ms eased, 0.65 wheel multiplierNative-feeling, not Qt-jittery
Fast-mode resizeDuring interactive drag, shadows collapse to 1 layerResize stays fluid on 4GB RAM
Elided labelsDetail, recipient, attachment truncate with …Never overflow, never reflow

Troubleshooting the Panellink

Panel doesn't appear when I run a commandadd
Check that ZmqListener is running (log line: ZmqListener loop error). Verify ports 5555–5557 are not firewalled. On restrictive Windows setups, allow python.exe through Windows Defender Firewall.
Confirmation card appears but Esc doesn't canceladd
Ensure focus is on the panel — click the panel first. Esc is bound to preview_body_edit and preview_subject_input only. If focus leaked elsewhere, use the Cancel button.
Panel stuck showing "Step 03" foreveradd
The agent likely crashed mid-loop. Restart JARVIS. The panel resets on the next status update with step=0, action=idle.
Text in thought stream is unreadableadd
Missing english.ttf, devangri.ttf, or code.ttf in Data/fonts/. Ensure all three fonts are present. The panel falls back to Segoe UI but may render Devanagari incorrectly.

Typing Popup

Live descriptionFile IPC graphic_eqTTS Word Sync imageAsync Previews

The Typing Popup

The Typing Popup is JARVIS's response surface — the premium floating window where the agent's final answer streams in character-by-character, with live markdown rendering, syntax-highlighted code blocks, async image previews, and word-by-word text-to-speech highlighting. Where the Agent Panel shows what JARVIS is thinking, the Typing Popup shows what it's saying.

stream

Token Streaming

Text appears as it's generated — markdown heals mid-stream.

graphic_eq

TTS Word Sync

Every spoken word highlights in real-time, synced to audio.

image

Async Previews

YouTube thumbnails, link cards, image cards — all fetched in background.

auto_awesome

Smart Self-Dismiss

Fades out after 6s on short responses, stays on long ones.

info

Read-only surface. The popup doesn't accept input — you talk to JARVIS via the Agent Panel or voice. It's auto-positioned and self-managing.

Typing Popup showing streaming response with code block and image card rendering
LiveTyping Popup mid-stream — code block + image card

When It Appearslink

  • File-triggered: appears the moment Data/typing_status.json receives {status: "typing", text: "..."}
  • Auto-shows on "typing" or "completed" status change
  • Re-shows if the file is re-written while the popup is fading out
  • Re-renders live as the text field grows
  • Always centered horizontally, 36px from top — unless dragged
  • Slides down + fades in (320ms quint-eased entry)
warning

File-driven, not socket-driven. The popup is entirely file-triggered — no ZMQ, no IPC server. If Data/typing_status.json is never written by the agent, the popup will never appear. This makes it safe to run standalone and easy to test manually.

The Response Lifecyclelink

State 1TYPING→Purple orb, auto-scroll, height animates up
State 2COMPLETED→Green check, final render, scroll unlocks
State 3DISMISSED→250ms fade out, app exits if standalone
StateStatus TagOrb ColorAuto-Behaviors
typing"Jarvis Responding"Purple #BF5AF2Auto-scroll ON, height animates up, idle counter reset
completed"Response Ready"Green #30D158Scroll unlocks, final render, idle counter starts
dismissed——250ms fade out, app exits

Anatomy of the Popuplink

Typing Popup anatomy with numbered pins overlaid 1 2 3 4 5 6 7 8 9
  • 1Status Orb
  • 2Status Tag
  • 3Mood Pill
  • 4Speaker Button
  • 5Copy Button
  • 6Close Button
  • 7Separator
  • 8Content Area
  • 9Preview Overlay

The Mood Pilllink

If the agent's response starts with an [emotion] tag (2-20 chars, alphanumeric + space + hyphen + underscore), it's extracted and displayed as a rounded pill next to the status tag. Common emotions include:

  • Example: [focused] Let me analyze that codebase... → pill shows Focused
  • Stripped from rendered text — users never see the raw tag
  • Stripped from copy text — clipboard gets clean response only
  • Hidden automatically when no emotion tag is present
check_circle

Cosmetic layer. The mood pill doesn't affect agent behavior — it just makes the popup feel alive. Especially effective during long responses where [focused] shifts to [confident] at the end.

Rich Content Renderinglink

FeatureRendering
Headings (H1-H6)Size-scaled, purple bottom-border on H1
Bold / italic / strikethroughInline styled
Inline codeDark chip, purple text, monospace
Fenced code blocksTraffic-light header, language label, monokai syntax
TablesBordered grid, header row with 7% white bg
BlockquotesLeft purple border, italic, tinted bg
Bullet / numbered listsNested indentation, custom bullets
LinksPurple, underlined, click opens in external browser
Images (URL, path)Rendered as styled card, click opens full preview
YouTube URLsAuto-detected, thumbnail with play button
Checkboxes☐ / ☑ glyphs

The Streaming Stabilizer

While text is streaming, markdown is often incomplete — unclosed code fences, partial links, broken image syntax. The parser auto-heals on every render:

Frame 1
User sees: ```py start, code streaming
→
Frame 2
Partial fence auto-closed, renders as block
→
Frame 3
Full render, syntax highlighting active
check_circle

No broken renders. Unclosed code fences auto-append ```, partial image syntax is replaced with a "Loading Image..." card, trailing URLs become link preview placeholders. Users never see a broken UI, even if markdown is 40% complete.

Async Image & Link Previewslink

Four custom URL schemes drive the preview system:

SchemePurposeRenderer
img://<url>Direct imageStyled rounded card with domain + resolution footer
preview://<url>Link preview card420×286 card with hero image + title + description
loading://imagePlaceholderAnimated "Loading..." shimmer while fetching
imgpreview://<url>Full-screen clickOpens the modal overlay

Link preview pipeline

  1. Microlink API attempt — fetches title, description, image, publisher from the target URL.
  2. Fallback: direct scrape — if Microlink fails or times out, fetches the raw URL and parses OpenGraph tags.
  3. Hero image fetch — if an image URL is found, a second async request downloads it.
  4. Card render — composes a 420×286 card: hero (192px) + title + description + domain pill.
  5. Cache — meta cached in Data/cache/meta_cache.json, styled image cached in LRU.

YouTube auto-embed

  • Any YouTube URL (watch?v=, youtu.be/, shorts/, embed/) auto-extracted for the 11-char video ID
  • Thumbnail fetched from maxresdefault.jpg, fallback to hqdefault.jpg
  • Rendered with centered red YouTube play button pill (58×40)
  • Click opens the actual YouTube URL in external browser

Image preview modal

Click any image card to open the full-panel overlay with:

  • Top bar: resolution label (1920 × 1080 px) + close X
  • Center: full image scaled to fit
  • Bottom bar: "Open Original ↗" button (opens source URL in browser)
  • Close: click anywhere OR press Esc
Image preview modal with resolution label and Open Original button
ModalFull-panel image preview with resolution + Open button

Caching strategy

CacheSizeTTLPurpose
Styled images100 (LRU)Until evictedFast re-render on scroll
Raw images35 (LRU)Until evictedSource for re-styling on resize
Failed URLs∞30sAvoid retrying bad URLs
Meta cache∞PersistentLink preview metadata

Text-to-Speech Integrationlink

When TTS is configured, every spoken word highlights in real-time — perfectly synced to the audio, not estimated. This is done via a dedicated tts_word_sync.json channel that carries word-level timings from the TTS engine.

Step 1TTS Status→Popup starts 33ms highlight ticker
Step 2Word Sync→Popup builds char-span map for sentence
Step 3Highlight→Current word gets accent bg via ExtraSelection
0.00s
"The" highlighted
→
0.15s
"function" highlighted
→
0.48s
"returns" highlighted
→
0.72s
"a" highlighted
bolt

Frame-perfect. The TTS engine reports its own word timings, so the highlight is exact — even at 2× speed or with pauses mid-sentence. No estimated timing, no drift.

Word-sync compatibility

TTS ProviderWord Sync SupportedNotes
Sarvam✅ YesNative word timings
Deepgram✅ YesNative word timings
Edge TTS❌ NoNo timings — no highlight
Azure TTS✅ YesVia WordBoundary events
warning

If the TTS provider doesn't support word timings, the audio still plays and the speaker button still works — only the word highlight is silently skipped. Nothing breaks.

Controls & Shortcutslink

Dragging

Click & drag→Reposition popup (except on buttons/links)

Keyboard

Esc→Close modal, then fade out popup

Scrolling

  • Auto-scroll ON while typing — always scrolls to bottom
  • Scroll up — auto-scroll turns OFF (respects reading)
  • Scroll to bottom — auto-scroll re-enables
  • Smooth animation — cubic-eased, 25% distance per tick at 60fps

Auto-dismiss conditions

ConditionValue
Statuscompleted
Mouse not hoveringRequired
TTS not speakingRequired
Preview modal closedRequired
Word count0 < words < 45
Idle duration> 360 ticks (~6 seconds)
info

45-word threshold. Long answers stay visible (user is reading), short answers disappear (user already read it). Hover over the popup to reset the idle counter and keep it visible.

Copy button showing transition from Copy to Copied state
CopyCopy button in "✓ Copied" feedback state

Architecture — File-Based IPClink

Unlike the Agent Panel (ZMQ), the Typing Popup uses file-based IPC. Four independent JSON channels carry all data:

FileDirectionWriterPurpose
Data/typing_status.jsonAgent → PopupAgent loopResponse text + status
Data/tts_status.jsonTTS → PopupTTS engineSpeaking state + text
Data/tts_word_sync.jsonTTS → PopupTTS engineWord-level timings
Data/tts_command.jsonPopup → TTSPopupStop / respeak commands
info

Why files, not ZMQ? The Typing Popup is a separate process that runs standalone. File IPC means it can start before the agent, survive agent restarts, and be tested with nothing more than echo '{...}' > Data/typing_status.json. The trade-off is slightly higher latency (~16ms poll) — imperceptible for text.

check_circle

Dual detection. QFileSystemWatcher fires immediately on file write AND a 16ms master timer polls mtime every 144ms as a fallback — in case watch events are lost on network drives. Belt + suspenders.

Performance & Premium Detailslink

TechniqueWhat it doesImpact
16ms master timerSingle timer drives RGB, height, scroll, polling, word highlight60fps with one loop
Async markdown workerPygments + markdown2 on a QThreadNo UI stall on large code blocks
Debounced parse28ms debounce; cancels in-flight workerPrevents N workers queuing
Streaming tail stabilizerRepairs partial markdown on every renderNo flicker mid-stream
Styled image cache100 LRU entries with DPR-aware pixmapsZero repaint on scroll
Meta cache persistenceDisk-backed link preview metadataInstant on second view
Height animationEased 26% per tick to targetSmooth expansion
Dirty region trackingQTextDocument invalidates changed blocks onlyOnly repaints diffs

Troubleshooting the Popuplink

Popup never appearsadd
Check that Data/typing_status.json exists with {"status": "typing", "text": "..."}. If the file exists but the popup doesn't show, check Qt logs for OpenType support missing — the font may be corrupted.
Images stay as "Loading Image..." foreveradd
Microlink API may be down or rate-limited. Images still load directly via fallback scraper. Check Data/cache/meta_cache.json for stale entries and delete if needed.
Word highlight doesn't sync with audioadd
Verify Data/tts_word_sync.json is being written by the TTS engine. If active: false persists, the TTS provider doesn't support word timings — check the compatibility table above. Edge TTS doesn't support it; Sarvam and Deepgram do.
Popup flashes then disappearsadd
Word count is under 45 and you're not hovering. Hover to lock it open, or increase the response length. This is intended behavior for short answers.
Copy button copies wrong textadd
If the response starts with [emotion], verify the tag matches the regex ^\[[a-zA-Z_ -]{2,20}\]. Malformed tags (like [happy_emoji_😀]) are not stripped.

STT Popup

Live wifiUDP Socket touch_appClick-Through translateHinglish Ready

The STT Popup

The STT Popup is JARVIS's live voice input surface — a small floating pill at the bottom of your screen that shows exactly what the speech-to-text engine is hearing, in real time, with a state-aware animated waveform. It's the only JARVIS UI component that is completely transparent to clicks — you can keep typing in your IDE while the waveform dances at the bottom of your screen.

mic

Live Transcription

Words appear as you speak — last 7 words visible.

graphic_eq

3-State Waveform

5 animated bars switch between listening, understanding, transcribed.

touch_app

Click-Through

Zero input capture — you can't accidentally click it.

translate

Hinglish Ready

Auto-detects Devanagari and swaps to Nirmala UI font.

info

Never takes focus. Uses Qt.WindowTransparentForInput — mouse clicks pass straight through to whatever is underneath. You can keep typing in your IDE while transcribing.

STT Popup in listening state at bottom center of screen with blue waveform
LiveSTT Popup — voice input surface

The 4 Stateslink

The popup transitions through four states as speech is captured and processed:

StateWaveformText StyleAccent ColorHold
listeningRandom bars (4.5–20px)96% white, Medium#0A84FF blueUntil stop
understandingSine-wave (5–21px)96% white, Medium#BF5AF2 purpleUntil stop
transcribedStatic green check100% white, Medium#30D158 green2.2 seconds
idleHidden—#6E6E78 greyFade out
check_circle

Peripheral awareness. The state color is the biggest UX win — during fast command speaking, you can tell from peripheral vision whether JARVIS is still listening (blue), processing (purple), or has locked in the transcription (green). No reading required.

Anatomy of the Popuplink

STT Popup anatomy with numbered pins 1 2 3 4
  • 1Waveform
  • 2Text Label
  • 3Island Frame
  • 4Auto-Width Container

The Waveform Indicatorlink

Five vertical bars that animate independently per state, with distinct visual identity:

Waveform close-up showing all four state variants side by side
WaveformAll 4 state variants side-by-side
StateUpdate PatternVisual Effect
listeningEach bar independent random [4.5, 20] pxChaotic, alive, mic is hearing something
understandingSine wave: 5 + abs(sin(phase*2.5 + i*0.8)) * 16Rhythmic, processing pipeline working
transcribedBars hidden, replaced by static checkmarkDone, final
idleBars collapse to 5pxFlatline, at rest
info

60fps smoothness. Bars interpolate at 22% per 16ms tick — no snapping. Bar colors have a glow shadow that scales with height. Each bar has a vertical gradient for depth.

Text Display & Font Logiclink

7-word truncation

  • Full transcript stored internally (last_text)
  • Only last 7 words shown — with "..." prefix if truncated
  • Example: "... the database schema needs a new index" (12 words spoken, 7 shown)
  • Prevents popup width from ballooning during long commands

Hindi detection

  • Scans every char for Devanagari range (\u0900–\u097F)
  • If ANY Devanagari char detected → switches font from English to Nirmala UI
  • Font size bumps +1pt for Hindi (13pt vs 12pt) for legibility
  • Auto-capitalization skipped for Hindi — Devanagari has no case
translate

Hinglish-safe. You can speak Hinglish ("Calendar mein meeting add karo") and the text renders correctly — English words in Segoe UI, Hindi words in Nirmala UI, seamlessly interleaved by Qt's font substitution.

Positioning & Auto-Hidelink

  • Horizontally centered on primary screen
  • Vertically 52px above taskbar
  • Moves up 22px on show, down 18px on hide
  • Width re-measures on every text update
  • Cannot hide during transcribed (2.2s lock)
  • Animation: 200ms fade out + 260ms slide down + hide
check_circle

2.2s transcribed lock. Gives you time to read the final transcript before it vanishes. If the transcript is wrong and you want to retry — the green check holds long enough to notice the error.

Architecture — UDP Datagram Channellink

The STT Popup uses UDP datagrams for communication — the only JARVIS UI that uses UDP:

Speech STT Engine → UDP:5556 → STT Popup
descriptionJSON payload
{
  "status": "listening",
  "text": "the full transcribed sentence so far"
}

Valid status values: listening, understanding, transcribed, idle, exit.

Why UDP?

  • Fire-and-forget — if popup isn't running, no error thrown
  • No handshake — 0ms overhead per message
  • Idempotent updates — each datagram is a full state snapshot
  • Low latency — critical for live transcription
warning

Firewall consideration. UDP 5556 is a different protocol from AgentPanel's ZMQ 5556 TCP — no conflict. But on some Windows firewalls, UDP 5556 must be explicitly allowed for the same app. Symptom if blocked: popup shows "Listening..." forever.

Performance & Premium Detailslink

TechniqueImpact
UDP socket (no parsing thread)Single-threaded, zero overhead
WindowTransparentForInputZero focus steal, zero click capture
Custom waveform widgetZero QML overhead, pure Qt paint
Auto-width (240–760px)Never truncates text, never blocks screen
Hindi font swapCached font IDs, no per-paint lookup
Gradient border (animated)36ms tick, single repaint region

Troubleshootinglink

Popup shows "Listening..." foreveradd
UDP 5556 is blocked by firewall, or the speech engine isn't sending datagrams. Check netstat -an | findstr 5556 for a UDP listener. Whitelist the app in Windows Firewall.
Hindi text shows as boxesadd
Data/fonts/devangri.ttf is missing. The fallback "Nirmala UI" must be installed on Windows 10+. Install the bundled font or the system font.
Text is cut off mid-wordadd
Check window scaling (125%, 150%) isn't pushing width calculation off. The 240px min width should always be respected — report as an issue if not.
Waveform bars are frozenadd
The anim_timer thread is starved. Check CPU usage — the popup needs ~16ms per tick. Close other heavy processes or restart JARVIS.
Popup doesn't auto-hide after transcriptionadd
transcribed_timer never fired. Check that the Qt event loop isn't blocked by another long task. Restart JARVIS to reset the timer.

Input Popup

Dual-Mode keyboard_command_keyCtrl+Shift+J helpask_user Clarification codeSubprocess IPC

The Input Popup

The Input Popup is JARVIS's dual-mode glassmorphic command surface. In Command Mode, it appears at the bottom of your screen when you press Ctrl+Shift+J, accepts a text command, and closes with Enter. In Clarification Mode, it's summoned automatically by the AgenticBrain's ask_user tool when critical parameters are missing mid-mission — pausing the loop timeout, muting background STT, and pinning your answer to confirmed facts without restarting the loop.

bolt

Global Hotkey

Ctrl+Shift+J from anywhere in Windows.

help

Clarification Mode

ask_user pauses the loop and asks you instead of guessing.

auto_awesome

Pulse-on-Type

Border glows purple when text is entered.

check_circle

Enter to Send

Shift+Enter for newline, Enter to submit.

info

Modal by design. Unlike the STT Popup (which is click-through), the Input Popup is modal — it steals focus and blocks interaction with the app underneath until you send, answer, or cancel. This is intentional: you're committing to a command or a clarification answer, not accidentally pressing a hotkey.

Input Popup in typing state with pulsing border, enter indicator, and text visible
LiveInput Popup — keyboard command surface

The Stateslink

TriggerStateBorderIndicatorAccent
EmptyIdleStatic 100 alphaHidden (0%)#BF5AF2
1+ charsTypingPulsing 100–190 alphaVisible (95%)#C864FC
ask_user firedClarificationCyan static accentVisible (95%)#00D4FF
ClearedIdleReturns to staticFades to 0%#BF5AF2
check_circle

Pulse animation (1200ms up + 1200ms down loop) is subtle but unmistakable — your eye locks onto the popup when text is entered. It signals "I'm ready, send me" without needing a button press.

Anatomy of the Popuplink

Input Popup anatomy with numbered pins 1 2 3 4 5
  • 1JARVIS Icon
  • 2Text Input
  • 3Enter Indicator
  • 4Glass Container
  • 5Border Pulse

The Enter Indicatorlink

A circular indicator that appears when text is entered:

  • 28×28 circle with two nested glow rings
  • Arrow-in-circle glyph — stylized ↵ rendered as path
  • Hidden when empty — 0% opacity
  • Appears when 1+ chars typed — 95% opacity, 400ms fade-in
  • Pulses color with the container's accent
  • Click it to submit (same as Enter)
Enter indicator close-up showing circle with arrow glyph
IndicatorArrow-in-circle enter glyph
info

Only button. The enter indicator is the popup's only button — no Send/Cancel buttons here, unlike the Agent Panel's HITL cards. Intentional minimalism — you shouldn't have to aim your mouse to send a text command.

Clarification Mode — the ask_user channellink

Mid-task, the AgenticBrain sometimes hits a hard stop: it needs a specific phone number, a file path, or a design choice that no amount of reasoning can invent. Instead of hallucinating a value, it calls ask_user. The Input Popup is summoned automatically in Clarification Mode — a cyan-accented variant of the same glassmorphic island.

BehaviorCommand ModeClarification Mode
TriggerUser presses Ctrl+Shift+JAgenticBrain calls ask_user
BannerNoneCyan-accented question banner showing the agent's exact question
Loop timeoutNormalPaused while you type
Background STTActiveMuted (no accidental voice injection)
Answer destinationRouter → FastBrain or AgenticBrainCurrent step Observation + <Confirmed_Facts> block, then loop resumes
Restart required—No — loop continues from the same step
check_circle

Ask, don't guess. When the agent doesn't know a required value, it's much cheaper to ask once than to hallucinate and fail three steps later. Clarification Mode turns a stuck mission into a 15-second question.

warning

AgenticBrain only. ask_user is only available in AgenticBrain mode. FastBrain is stateless — it can't pause mid-run because it doesn't have a mid-run.

Smart Input Behaviorslink

Keyboard shortcuts

KeyAction
EnterAccept — submit command, fade out
Shift + EnterNew line — multi-line commands
EscReject — cancel, fade out with no submit
Ctrl + VPaste as plain text (rich formatting stripped)
TabFocus cycler

Paste sanitization

  • Any rich text (from Word, browser, etc.) is stripped to plain text on paste
  • Preserves Unicode (Devanagari, emoji)
  • Preserves line breaks — multi-line pastes stay multi-line
  • No HTML, no formatting, no images
check_circle

Safe paste from anywhere. Paste from Discord, Word, code snippet — it becomes clean text. No hidden formatting leaking into the agent prompt.

Auto-Resize & Layoutlink

ContentHeight
Empty / 1 line30px (min)
2 lines54px
3 lines78px
4 lines102px
5 lines126px
6 lines150px
7 lines174px
8+ lines180px max + scrollbar
0ms
1 line, 30px
→
340ms
3 lines, 78px
→
680ms
5 lines, 126px
→
1020ms
8+ lines, 180px + scroll
info

Bottom-anchored growth. When you type a long command, the popup grows upward — bottom edge stays fixed, so your input line stays in your natural gaze. You don't have to look down to keep typing.

Architecture — Subprocess Stdoutlink

The Input Popup runs as a subprocess per invocation and returns via stdout:

LaunchParent Process→Subprocess
WaitUser types→Presses Enter
Replystdout→Parent decodes
terminalstdout format
JARVIS_CMD_B64::<base64-encoded-utf8-command>

Example: user types Open VS Code and find the test file → popup outputs JARVIS_CMD_B64:::T3BlbiBWUyBDb2RlIGFuZCBmaW5kIHRoZSB0ZXN0IGZpbGU= → parent decodes back to the original command.

Why stdout, not file/IPC?

  • Modal subprocess — popup's lifetime is exactly one command
  • No persistent state — no file cleanup needed
  • Base64 wrapping — handles any Unicode/emoji/newline safely
  • Exit code: 0 on accept, 1 on reject or error
  • Stderr: captured for debugging (traceback on crash)
check_circle

Base64 encoding. Commands with emoji 😀, Devanagari, Arabic, newlines, or even null bytes all survive the IPC intact. No encoding headaches, no character corruption.

Positioning & Animationslink

Position

  • Horizontally centered on primary screen
  • 160px above taskbar (higher than STT Popup)
  • Width: fixed 720px — doesn't reflow on text
  • Vertical position: bottom-anchored — bottom edge fixed, grows up

Animation choreography

PhaseDurationEasingEffect
Entry fade280msOutCubic0 → 1 opacity
Entry slide360msOutQuint+28px → target Y
Pulse up1200msInOutSineBorder 100 → 190 alpha
Pulse down1200msInOutSineBorder 190 → 100 alpha
Indicator fade400msInOutSine0 → 95%
Resize340msOutQuartHeight grows
Exit fade260msInCubic1 → 0 opacity
Exit slide300msInQuartY → Y + 24px
0ms
Fade in start
→
180ms
Slide mid
→
360ms
Entry complete
→
Long
Pulse loop (2400ms)
info

360ms entry. Fast enough to feel instant, slow enough to feel premium. macOS Spotlight opens in ~150ms but feels less ceremonial. JARVIS's popup is deliberately slightly slower — a tool you invoke deliberately.

Performance & Premium Detailslink

TechniqueImpact
Single QDialog instance per invocationZero memory leak across runs
pulseIntensity pyqtPropertySmooth 2s loop, no QTimer jitter
indicatorOpacity pyqtPropertyIndependent animation from container
Parallel animation group for resizeWindow + text input resize in sync
QSS for text input (not per-paint)Native performance
Base64 stdout (no sockets)Zero IPC infrastructure
Font substitution chainSegoe UI → Nirmala UI → Mangal (Hinglish safe)

Troubleshootinglink

Ctrl+Shift+J doesn't open the popupadd
Global hotkey registration failed — usually because another app grabbed it. Check Task Manager → Startup → is JARVIS disabled? Or check if another app (VS Code, Discord) uses the same hotkey.
Popup opens but I can't typeadd
Focus delay (150ms) may not have fired. Click into the input area once. If persistent, verify no other modal window is stealing focus.
Enter submits but nothing happensadd
Parent process isn't reading stdout. Check that the launcher is waiting on input_popup.communicate() or similar. If the agent crashed, the command is lost — no persistence layer exists.
Multi-line commands get cut offadd
Text input maxes at 180px. For very long inputs, write to a file and reference it: Read file C:\commands\my_task.txt. Or increase MAX_TEXT_HEIGHT in source.
Popup stays open foreveradd
isClosing flag stuck. Report as bug — should never happen. If urgent, press Esc to force reject, or kill the process via Task Manager.

UI Comparison

All four JARVIS UI components are independent processes with their own IPC mechanism. Crashing one doesn't take down the others — a deliberate resilience choice.

Side-by-sidelink

FeatureAgent PanelTyping PopupSTT PopupInput Popup
DirectionOutput (live)Output (final)Input (voice)Input (text)
IPCZMQ TCPFile JSONUDP socketSubprocess stdout
PositionTop centerTop centerBottom centerBottom center
ClickableYes (HITL, drag)Yes (drag, copy)No (click-through)Yes (typing focus)
PersistentWhile loop activeUntil dismissedWhile speakingWhile dialog open
Multi-instanceOneOneOneOne per invocation
Can overlapYesYesYesBlocks (modal)
InteractionRead + HITLRead + copyNoneType + Enter

How they connectlink

Input STT Popup or Input Popup → Agent Loop
Output Agent Loop → Agent Panel + Typing Popup
check_circle

Independent processes. A frozen Typing Popup doesn't stop the Agent Panel from showing live thoughts. A crashed Agent Panel doesn't stop STT from transcribing. If one crashes, the others keep running — and JARVIS auto-restarts the failed one via ServiceWatchdog.

Which input should you use?link

ScenarioBest Option
Hands busy, quick commandSTT Popup — voice
Complex, multi-line commandInput Popup — text + Shift+Enter
Precise technical queryInput Popup — code formatting
Casual "hey Jarvis"STT Popup — wake word
Drafting emailsInput Popup — long form
Away from keyboardSTT Popup + Telegram bot

Lifecycle & Shutdown

Graceful shieldSignal-Safe monitor_heartWatchdog restart_altAuto-Recovery

Lifecycle & Shutdown

JARVIS ships 8 background services (including the Unified Portal), 3 Qt overlays, a Node.js bridge, and dozens of subprocesses. Every one of them needs to die cleanly — but the memory graph, behavioral patterns, and session state must survive. The lifecycle system is a 14-step graceful shutdown choreography guarded by an idempotent flag, 7 signal handlers, and a parallel service-teardown that completes in ~2.6 seconds.

lock

Idempotent Guard

Double Ctrl+C is safe — the second press returns instantly.

notifications_active

7 Exit Paths

Ctrl+C, Ctrl+Break, window close, logoff, shutdown — all covered.

save

State Preserved

LTM graph and behavioral engine save before any process is killed.

speed

2.6s Total

8 services tear down in parallel, not sequentially.

When Shutdown Triggerslink

JARVIS registers handlers for every standard exit path — plus a few Windows-specific ones that signals alone can't catch:

#TriggerOS Event / SignalHandler
1Ctrl + CSIGINTsignal_handler()
2Ctrl + BreakSIGBREAKsignal_handler()
3External taskkillSIGTERMsignal_handler()
4Window X buttonCTRL_CLOSE_EVENT (2)_win_ctrl_handler()
5Windows logoffCTRL_LOGOFF_EVENT (5)_win_ctrl_handler()
6System shutdownCTRL_SHUTDOWN_EVENT (6)_win_ctrl_handler()
7Tray → ExitGUI clickshutdown_jarvis()
8Python process exitatexitLambda → shutdown_jarvis()
9Main loop end—finally block
warning

Signals alone don't catch the X button. On Windows, clicking the window close button sends CTRL_CLOSE_EVENT — not SIGTERM. Without the ctypes.windll.kernel32.SetConsoleCtrlHandler registration, JARVIS would hang as a zombie process whenever you closed the console window.

The 14-Step Shutdown Sequencelink

Every shutdown follows the same 6-phase choreography. Each phase is guarded — if it fails, the next still runs.

Phase 1Guard & Flag→Acquire lock, set _shutdown_started, break main loop
Phase 2Graceful Signals→Restore console, set shutdown event, remove lock file, stop tray
Phase 3Save State→Persist LTM graph + Behavioral Pattern Engine
Phase 4Stop Workers→Thread pool cancel, TTS cleanup, pygame quit, watchdog stop
Phase 5Parallel Teardown→7 services killed in parallel, 2.5s max each
Phase 6Hard Exit→Process tree cleanup, os._exit(0)
bolt

Parallel teardown = 8× speedup. 8 services × 2.5s timeout = 20s if sequential. But stop_all_services() spawns 8 threads — total wall-clock is ~2.5s regardless of count. Shutdown stays under 3 seconds on any hardware.

info

Why os._exit() not sys.exit()? sys.exit() runs more Python cleanup — atexit handlers, finally blocks, garbage collection. But JARVIS has already done all cleanup manually. os._exit() skips everything — guaranteeing no stray handler accidentally respawns a thread or writes to a corrupted state during teardown.

0 ms
Signal received — guard acquires lock
→
10 ms
Console restored, lock file removed
→
300 ms
LTM graph + behavioral engine saved
→
800 ms
Thread pool cancelled, TTS stopped
→
2.5 s
All 7 services terminated in parallel
→
2.6 s
os._exit(0) — clean exit
warning

Second Ctrl+C is safe. The _shutdown_started flag returns instantly from shutdown_jarvis(). Users instinctively mash Ctrl+C when shutdown is slow — no double-kill, no exception, no half-state.

Subprocess Manager — Process Tree Controllink

Every spawned process is tracked by a singleton SubprocessManager registered with atexit:

ManagerSubprocessManagersingleton · atexit-registered
ChildrenAgentPanelSttPopupBaileysMobileBridgeMCP
Killpsutil recursive tree3s graceful → 2s force
FeatureBehavior
Singleton patternOne manager, one process registry
Windows flagsCREATE_NEW_PROCESS_GROUP — Ctrl+Break propagates to children
Unix flagsstart_new_session=True — separate session, no TTY hogging
Tree killpsutil.Process(pid).children(recursive=True) — orphans impossible
Timeout3s graceful terminate, then 2s force kill
atexit registrationEven uncaught exceptions trigger cleanup
IdempotentCalled twice? Second is a no-op
info

CREATE_NEW_PROCESS_GROUP matters. Without it, Ctrl+C in the console would propagate to ALL child processes at once — Qt overlays would crash mid-render, Node.js would abandon partial writes. With it, the manager sequences shutdown — graceful, one service at a time.

The Lock File — One Instance Onlylink

JARVIS refuses to start if another instance is already running. A lock file with the current PID prevents duplicate launches:

Startupcreate_lock_file()→Writes current PID
Second launchis_jarvis_running()→Checks if PID is alive
On shutdownremove_lock_file()→Frees JARVIS for next launch
warning

Stale lock file. If JARVIS crashes hard (taskkill /F /IM python.exe), the atexit handler may not run — the lock file lingers. Next startup shows "Already running" even though nothing is. Fix: delete the lock file manually, or check Task Manager for the rogue python.exe and kill it.

Service Watchdog — Auto-Recoverylink

The watchdog thread checks every service on a 5-second interval. If one crashes, it restarts — but only if credentials exist and the service is supposed to be running.

CheckEvery 5s→Is service running?
If downHas creds?→No → skip (log once/min)
If readyRestart→After 3 strikes → 120s backoff
ServicePortReady CheckCreds Required
Baileys (WhatsApp):3000—auth_info_baileys/creds.json
STT PopupUDP :5556—Microphone available
Mobile Bridge:8090—Always true
Indexer Dashboard:8080is_rag_engine_init_startedAlways true
Unified Portal:8000is_rag_engine_init_started AND bridge runningBoth backends reachable
Telegram Remote—is_telegram_remote_init_startedtelegram_bot_token.enc
MCP Manager—is_mcp_readyis_mcp_enabled
check_circle

Smart skip. If WhatsApp login isn't done (no creds.json), the watchdog doesn't restart Baileys — it skips entirely. Logs once per minute. This is production-grade: don't retry things that aren't meant to run.

warning

3 strikes + 120s backoff. If a service fails 3 times consecutively (e.g., port conflict), the watchdog backs off for 120 seconds. Prevents infinite restart loops from melting the CPU.

System Tray Integrationlink

When you minimize the console, it hides to the system tray — not the taskbar. A daemon thread checks the window state every 250ms.

ActionEffect
Left-click tray iconRestore console, lock window size
Right-click → Open JarvisRestore + foreground
Right-click → Exit JarvisGraceful shutdown
Minimize consoleAuto-hide to tray (250ms poll)
Close windowFull shutdown (via CTRL_CLOSE)
info

Optional dependency. pystray and PIL are not installed by default. Without them JARVIS still runs — you just can't minimize to tray. Install: pip install pystray pillow.

Console Locking (Windows Polish)link

On Windows, JARVIS locks the console window to a specific size and style — 96 × 28 grid, 860 × 540 pixels, resize buttons removed. This is polish, not restriction.

Console TypeBehavior
Windows Terminal (WT_SESSION set)Skips buffer resize (handled by WT itself), applies via ANSI escape
Legacy conhostFull SetConsoleScreenBufferSize + SetConsoleWindowInfo
Non-WindowsFalls back to ANSI escape \033[8;{rows};{cols}t
warning

QuickEdit disabled. QuickEdit is the Windows feature where clicking in the console selects text and pauses the process. On a busy output stream, a mis-click freezes the whole thing. JARVIS disables QuickEdit at startup — no accidental freezes.

What Gets Saved vs Killedlink

✅ Saved (before exit)❌ Killed (during exit)
LTM semantic graphAgent Panel subprocess
Behavioral Pattern Engine stateSTT Popup subprocess
Session cookies (untouched)Baileys WhatsApp server
Conversation history (incremental)Mobile Bridge server
RAG vector index (on disk)RAG Indexer Dashboard
Config in .env (untouched)Telegram polling thread
Dashboard folder rulesMCP manager + MCP subprocesses
MCP server configsInput Popup (if open)
Telegram token (encrypted)Thread pool in-flight tasks
Auth tokens (untouched)Pygame audio device
check_circle

Nothing lost. Memory graph, behavioral patterns, conversation history — all saved before any kill signal is sent. Even Windows logoff (CTRL_LOGOFF) runs steps 9–10 before step 15. State survives everything short of a power cut.

Command Handler Guardslink

Commands behave deterministically during shutdown — no half-executed side effects:

  • Pre-check: if is_shutting_down(): return — drops incoming commands immediately
  • Mid-check: After process_command(), another check — if shutdown started mid-processing, actions aren't executed
  • Interrupt check: if interrupt.is_interrupted(): return — voice interrupt support
  • Empty guard: Empty strings and None dropped before processing
info

Dual shutdown check. The handler checks is_shutting_down() twice — before and after process_command(). A 5-second AgenticBrain task that started before shutdown won't execute its side effects if shutdown triggers mid-run. Critical: you don't want an email going out after Ctrl+C.

Troubleshooting Shutdownlink

Ctrl+C doesn't exit — JARVIS stays runningadd
One of the background services is hung (likely MCP or WhatsApp). Wait 3s for parallel teardown, then press Ctrl+C again — the idempotent guard makes it safe. If still stuck, open another terminal and run taskkill /F /PID <pid>.
JARVIS won't start — says "Already running"add
Lock file is stale from a previous crash. Delete the lock file in the repo root. Then check Task Manager → python.exe → kill any orphan JARVIS process.
Shutdown takes more than 30 secondsadd
One service is stuck waiting. Each service has a 2.5s timeout — total wall-clock should be ~2.6s. If exceeding, check startup logs to identify which service is hanging. Common culprit: WhatsApp Baileys waiting for network close during teardown.
Zombie processes after crashadd
os._exit() didn't run — likely a hard kill -9 or taskkill /F. Orphaned children get reparented to System PID 4. Fix: in PowerShell, Get-Process python | Stop-Process -Force.
System tray icon doesn't appearadd
pystray or PIL is missing. Install: pip install pystray pillow. Alternatively the tray library silently failed on non-Windows — tray is Windows-only.
Second JARVIS instance won't startadd
Same as "Already running" above. Check Task Manager for orphan python.exe. If PID is gone but lock file is there, delete the lock manually.

Troubleshooting

Common issues and their fixes. For anything not listed here, open an issue on GitHub.

Diagnostic decision treelink

Start with the symptom you're seeing. Each branch shows the most common cause and the fix.

rocket_launch JARVIS won't start expand_more
Python not found: Running jarvis returns "command not recognized".
Fix: Run python SetupRegistry.py from the repo root with virtual environment active. Then open a new terminal.
ModuleNotFoundError: Missing dependency.
Fix: Activate venv, then pip install -r requirements.txt.
"Already running": Lock file is stale from a previous crash.
Fix: Delete the lock file in repo root. Check Task Manager for orphan python.exe.
Instantly closes: Some DLL or C extension failed to load.
Fix: Run from PowerShell directly (not via .bat) to see the error. Most common cause: missing Visual C++ Redistributable.
mic Voice not working expand_more
No wake word response: Saying "Jarvis" does nothing.
Fix: Check startup logs for Azure keyword recognition started or Vosk initialized. If neither, no wake model is present. Install one at Data/model/azure/jarvis.table OR Data/model/vosk-model-small/.
Wake fires but no transcription: STT popup shows "Listening..." forever.
Fix: Verify DEEPGRAM_API_KEY in .env. Check UDP port 5556 not blocked by firewall.
High false wake triggers: JARVIS triggers on random words.
Fix: Vosk fallback is ~66% accurate. Build an Azure Custom Keyword model for 97%+ accuracy (see Voice Setup).
dashboard Agent Panel not appearing expand_more
Command runs but no panel: Response works, no floating UI.
Fix: Check startup log for ZmqListener loop error. Verify ports 5555–5557 not firewalled. Whitelist python.exe in Windows Defender.
Panel shows stale "Step 03": Frozen mid-loop.
Fix: Agent likely crashed. Restart JARVIS. Panel resets on next status update.
Text unreadable: Thought stream shows boxes or garbled.
Fix: Missing fonts. Ensure english.ttf, devangri.ttf, code.ttf exist in Data/fonts/.
extension MCP tools not appearing expand_more
Zero MCP tools: No mcp__* functions in agent's tool list.
Fix: Run jarvis mcp list. Confirm pip install mcp succeeded. Check Data/mcp_servers.json is valid JSON.
"Connection closed" in logs: Server crashes on startup.
Fix: Run jarvis mcp test <name> to see exact stderr. Verify Node.js 18+ (24+ for latest servers).
Tools appear then disappear: Server unstable mid-session.
Fix: Check MCPWatcher logs. MCP client auto-reconnects once; if it keeps failing, the server has a bug.
chat WhatsApp / Telegram issues expand_more
WhatsApp won't connect: Baileys subprocess exits immediately.
Fix: Verify Node.js 18+. Run npm install inside tools/Messanger/whatsapp/BaileysServer. Port 3000 must be free.
Telegram bot unresponsive: Messages to bot get no reply.
Fix: Run jarvis bot --status. If unconfigured, run jarvis bot --activate with JARVIS off.
Login popups on every boot: Annoying QR prompts.
Fix: Run jarvis logout --service for unused integrations. Listeners only start if credentials exist.
speed Performance / slow responses expand_more
Every reply takes 30+ seconds: Timeouts on every command.
Fix: Check provider status page. Verify PRIMARY_BASE_URL ends with /v1. FastBrain should respond under 2s — if not, it's a network issue.
Cold start is 60+ seconds: Every reboot is slow.
Fix: That's the embedding model load (~14s) + service boots. Disable unused integrations via jarvis logout --service to reduce boot time.
CPU pegged at 100%: System feels sluggish.
Fix: Long agentic runs (8+ min reasoning) hog a core. Use FastBrain for quick commands. On 4GB RAM hardware, this is expected.

ModuleNotFoundError on startuplink

Your virtual environment isn't active, or dependencies are missing:

terminalpowershell
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt

Wake word not workinglink

JARVIS requires at least one wake-word model:

  • Azure: Data/model/azure/jarvis.table
  • Vosk: Data/model/vosk-model-small/ (auto-downloads on first run)

If both are missing, JARVIS boots with voice disabled — no crash, just text mode via Ctrl+Shift+J.

WhatsApp won't startlink

  1. Verify Node.js 18+ — run node --version.
  2. Reinstall the bridge — cd tools/Messanger/whatsapp/BaileysServer && npm install.
  3. Check port 3000 — if busy, kill the process or change the port.

MCP tools not appearinglink

  1. Confirm pip install mcp succeeded.
  2. Verify Data/mcp_servers.json exists and is valid JSON. Check with:
    terminalpowershell
    python -c "import json; json.load(open('Data/mcp_servers.json'))"
  3. Check startup logs for MCP: ready with N tools.

MCP server logs "Connection closed"link

Run jarvis mcp test <name> to inspect the exact stderr output from the server:

terminalpowershell
jarvis mcp test everything

RAG indexer stucklink

If indexing hangs on a folder, check startup logs for permission_denied=N. Windows legacy junctions (My Music, My Pictures) are auto-skipped. Reduce scope via the dashboard filters.

Primary provider keeps failinglink

  • Verify PRIMARY_BASE_URL ends with /v1.
  • Check PRIMARY_API_KEY is valid (not expired).
  • Confirm model name exists at that provider.
  • JARVIS auto-falls back to FALLBACK_* if primary fails — check logs for confirmation.

Live correction is ignoredlink

Live correction only works in AgenticBrain mode. FastBrain is stateless and doesn't support mid-run re-planning. Check the router decision at the top of any log — you'll see Decision -> AGENTIC when the agentic loop is active.

Portal won't openlink

The Unified Portal at http://127.0.0.1:8000 needs both the Indexer Dashboard (Flask, port 8080) and Mobile Bridge (FastAPI, port 8090) alive. Verify all three Python dependencies:

terminalpowershell
pip install flask fastapi uvicorn httpx jinja2

If ports 8000, 8080, or 8090 are busy, JARVIS auto-shifts them (8080→8081–8095). Check startup logs for actual ports. Symptom of only one backend being down: portal loads, but one status pill shows Down and its tab shows an offline overlay.

Frequently Asked Questions

Short answers to the most common questions.

Is it really free?add
Yes. GPLv3. You can run it entirely on local LLMs with zero API cost, or mix free-tier cloud providers. No subscription, no feature gate, no telemetry.
Does it work on Mac or Linux?add
Currently Windows-only. GUI automation, PC monitoring, and the Termux bridge assume Windows paths and APIs. Cross-platform support is on the roadmap.
Is my data safe?add
Credentials are AES-encrypted locally. Conversation history and memory graphs stay on disk. Only LLM API calls leave your machine — and you can switch to local models entirely with zero cloud calls.
What hardware do I need?add
JARVIS was developed on an Intel i5-3210M with 4 GB RAM. If your machine can run Windows 11 and a browser, it can run JARVIS. More RAM = snappier local embeddings and faster boot.
Does RAG / memory need API keys?add
No. Both the LTM semantic graph and RAG index use the local BAAI/bge-small-en-v1.5 model (33M params, ~130 MB). Embeddings run on CPU with zero cloud calls, quotas, or cost.
Which LLMs does it support?add
Any OpenAI-compatible endpoint. Groq, Regolo, OpenRouter, Gemini's OpenAI endpoint, Ollama, LM Studio, vLLM, or any OpenAI-schema API. Configure two providers — PRIMARY_* and FALLBACK_* — and JARVIS auto-routes on failure.
Can I add capabilities without editing code?add
Yes — via MCP. Add a JSON entry to Data/mcp_servers.json, restart once, and the server's tools appear in the Agentic Brain automatically as mcp__<server>__<tool>.
How is this different from ChatGPT or Claude?add
Those are chat interfaces. JARVIS is an agent — it takes actions on your actual system, remembers you across months, corrects itself mid-task, and reaches out proactively. It doesn't just answer; it does.
How do I uninstall JARVIS?add
Single PowerShell command (one-liner). It stops all background services, cleans the registry and PATH, removes the global CLI, and deletes the install folder. It also prompts to back up your .env, memory graph, and MCP config to Desktop first. See the Uninstall page for the exact command and step-by-step walkthrough.

Glossary

Every acronym, protocol, and internal term JARVIS uses — defined in plain language. Bookmark this page when you encounter an unfamiliar term.

A

TermDefinition
AgenticBrainThe stateful, multi-step reasoning brain. Handles complex engineering, communications, memory retrieval, and vision — up to 50 tool-using steps per run. Contrasts with FastBrain.
ASTAbstract Syntax Tree. JARVIS parses generated code as an AST before writing it, catching syntax errors without running the code.
AsyncExitStackPython's async context manager stack. JARVIS uses it to isolate MCP client task lifecycles and prevent cross-task cancellation bugs.
Azure Custom KeywordA wake-word model trained on Azure Speech Studio. Runs fully on-device after one-time training, achieving 97%+ accuracy. Fallback is Vosk.

B

TermDefinition
BaileysAn open-source Node.js library that implements the WhatsApp Web protocol. JARVIS runs it as a subprocess for WhatsApp integration.
BM25Best Match 25 — a keyword-based ranking algorithm used in hybrid RAG. Fast, deterministic, complements vector search.
BGE-smallBAAI's 33M-parameter embedding model used for local memory and RAG. Runs on CPU, ~130 MB on disk, zero cloud calls.

D

TermDefinition
DPAPIWindows Data Protection API. JARVIS uses it (via the pywin32 library) to decrypt stored browser credentials during system-audit tasks.
Deepgram Nova-2A cloud speech-to-text model. Used for transcribing voice commands after wake word detection.

F

TermDefinition
FastBrainThe stateless, low-latency brain. Handles short commands, app toggles, and casual chat in under 2 seconds. No memory mid-run, no multi-step planning.

H

TermDefinition
HITLHuman-In-The-Loop. A safety mechanism where JARVIS pauses before irreversible actions (sending email, WhatsApp, deleting files) and asks for explicit confirmation via the Agent Panel. Enforced at the ZMQ layer, not just in the prompt.

J

TermDefinition
JSONLJSON Lines — a file format where each line is a valid JSON object. Used by JARVIS for rolling history and behavioral patterns.

K

TermDefinition
KWSKeyword Spotting. The process of detecting a specific wake word in an audio stream. Vosk's KWS engine is the fallback when Azure is unavailable.

L

TermDefinition
LTMLong-Term Memory. JARVIS's weighted semantic graph — stores facts, people, contacts, and relationships with temporal decay.

M

TermDefinition
MCPModel Context Protocol. Anthropic's open standard for connecting AI applications to external tools. JARVIS plugs into 18,000+ MCP servers via JSON config.

R

TermDefinition
RAGRetrieval-Augmented Generation. JARVIS's hybrid RAG combines BM25 keyword search with vector similarity over your indexed documents.
RRFReciprocal Rank Fusion. A score-merging algorithm that blends BM25 and vector results into a single ranked list for hybrid RAG retrieval.

S

TermDefinition
SarvamAn Indian TTS provider used for low-latency English + Hinglish text-to-speech. Falls back to Edge TTS if the API key is missing.
Set-of-Mark (SOM)A computer-vision technique where the screen is divided into labeled regions with IDs. JARVIS uses SOM overlays with OpenCV edge detection for reliable GUI automation.

T

TermDefinition
TTLTime-To-Live. JARVIS uses TTLs to auto-expire ephemeral state like screenshots (10 min), HITL confirmations (90 sec), and session artifacts.

V

TermDefinition
VADVoice Activity Detection. JARVIS uses dual WebRTC VAD gates to distinguish real voice from background noise before triggering the wake-word pipeline.
VoskAn offline speech-recognition toolkit. Used as the fallback wake-word engine when Azure Custom Keyword is unavailable.

Z

TermDefinition
ZMQ (ZeroMQ)A high-performance messaging library. JARVIS uses ZMQ TCP sockets (ports 5555–5557) to stream live status updates from the agent loop to the Agent Panel.

Known Limitations

JARVIS is not a magic bullet. This page documents exactly what it cannot do — so you can decide whether it fits your workflow before installing.

warning

Honest disclosure. Most AI projects hide their limits in a fine-print FAQ. We put them on a dedicated page because we'd rather you choose JARVIS with accurate expectations than be disappointed after setup.

Platform support

CapabilityWindows 10/11macOSLinux
Core agent loop✅❌❌
GUI automation✅❌❌
PC system monitor✅❌❌
Android bridge✅⚠️ Partial⚠️ Partial
Voice pipeline✅❌❌
MCP integration✅⚠️ Partial⚠️ Partial

Why Windows-only? GUI automation relies on pywin32, comtypes, and Windows UI Automation APIs. The Termux bridge assumes Windows path semantics. Cross-platform support is on the roadmap but not scheduled.

Performance on low-end hardware

JARVIS runs on a 4 GB RAM / Intel i5-3210M laptop, but with these tradeoffs:

  • Local LLMs are slow. A 7B model on CPU-only inference runs at ~2 tokens/sec — 30-60 seconds for a short response. Recommended: use cloud providers (Groq, Regolo, OpenRouter) unless you're 100% offline.
  • Embedding model load ~14s. The first cold start after reboot is noticeably slower than subsequent starts.
  • Deep-planner tasks dominate CPU. Multi-step agentic reasoning for 8+ minutes pegs one CPU core at 100% on the 2012 laptop.
  • Qt overlays animate at 60fps only on capable GPUs. On integrated graphics, panel animations drop to 30fps during heavy load.

What JARVIS does not do

  • No multi-user support. JARVIS is single-user, single-machine. No shared memory, no team features.
  • No web UI for commands. You must use voice, hotkey, or Telegram bot. No browser-based agent interface.
  • No automatic upgrades. You manage updates via git pull. No auto-update mechanism.
  • No enterprise support. Solo developer, no SLAs, no paid support tiers. Best-effort community support via GitHub issues.
  • No real-time collaboration. Not a Google Docs for AI — one user, one agent, one machine.
  • No video input analysis. Images and screenshots only. Video frames are extracted one-by-one via ffmpeg but the tool is not optimized for video.
  • No native mobile app. Android control requires Termux from F-Droid. iOS is not supported at all.

Security & privacy boundaries

security

JARVIS runs with your Windows user permissions. It is not a sandbox. A sufficiently clever prompt-injection attack could theoretically cause it to run unintended local commands. The HITL confirmation gate protects outgoing comms and destructive file ops — not terminal commands.

Recommendation: review execute_terminal_command behavior before enabling on a machine with sensitive credentials.

  • API keys stored in .env are plaintext unless you encrypt them yourself. Telegram bot tokens use AES encryption.
  • Conversation history is plaintext JSONL in Data/jarvis_memory/. Encrypt the disk if you handle sensitive data.
  • Cloud LLM calls send your prompts to Groq/OpenRouter/Gemini. Use local models for zero-knowledge privacy.

Planned for future versions

  • macOS/Linux partial support (agent loop only — no GUI automation)
  • Light mode for documentation
  • Native iOS bridge via Shortcuts (experimental)
  • Multi-user profiles (personal + work memory separation)
  • Plugin marketplace for community tools
  • Voice cloning for TTS (opt-in, off by default)

What will never happen

  • Closed-source release. GPLv3 forever. The CLA allows commercial licensing but the community version stays free.
  • Telemetry. No usage analytics, no crash reporting to servers, no "phone home" checks.
  • Cloud-hosted version. JARVIS is designed to run on your hardware. No SaaS.
  • Removal of HITL gates. Irreversible actions will always require explicit consent — no "trust mode" that bypasses them.

Changelog

Every notable change to JARVIS. Follows Keep a Changelog conventions and Semantic Versioning.

v1.0 — 2026-11-01

Initial public release.

Added

  • One-command PowerShell installer (install.ps1) — auto-installs Git, Python, and Node.js via winget, sets up the virtual environment with CPU-optimized PyTorch, installs the WhatsApp Baileys bridge, and interactively prompts for API keys
  • One-command PowerShell uninstaller (uninstall.ps1) — stops services, cleans registry and PATH, removes the global CLI, and deletes folders, with automatic Desktop backup prompt for .env, memory graph, and MCP config
  • Dual-brain routing (FastBrain + AgenticBrain)
  • Live mid-execution correction
  • 5-tier memory (Episodic, Semantic, RAG, Ephemeral, Behavioral)
  • Hybrid RAG with BM25 + vector + RRF
  • MCP integration with hot-reload (18,000+ servers)
  • Android bridge via Termux + WebSocket
  • Telegram bot for remote PC control
  • Dual-engine wake word (Azure + Vosk)
  • Deepgram Nova-2 STT with LTM vocabulary cache
  • Sarvam + Edge TTS streaming pipeline
  • Proactive HITL (Email, WhatsApp, Telegram, Calendar listeners)
  • Four premium UIs: Agent Panel, Typing Popup, STT Popup, Input Popup
  • 14-step graceful shutdown with 2.6s parallel teardown
  • Vision-based GUI automation with Set-of-Mark
  • PC system monitoring with proactive alerts
  • Documentation portal with multi-view router

Known issues

  • Windows-only (macOS/Linux not yet supported)
  • Local LLM inference slow on 4 GB RAM hardware
  • Video input analysis limited to per-frame extraction

Roadmap (unreleased)

Planned features for v1.1 and beyond. No dates committed.

  • Light mode for documentation portal
  • Print stylesheet for PDF export
  • Progressive Web App (offline docs)
  • Multi-user profiles
  • macOS/Linux partial support (agent loop only)
  • Native iOS bridge via Shortcuts
  • Voice cloning for TTS (opt-in)
  • Plugin marketplace
info

Versioning policy. JARVIS follows semver: MAJOR.MINOR.PATCH. Breaking changes in .env schema, config formats, or CLI flags increment MAJOR. New features increment MINOR. Bug fixes increment PATCH.

Custom Prompt Guide

JARVIS personality, tone, language, and behavioral rules live in two Python string constants inside core/brain/Processor/Prompts.py. This guide shows exactly how to change them without breaking the routing, escalation, or HITL safety systems.

Where prompts actually live

Despite common assumptions, JARVIS does not use external .txt or .md prompt files. Both system prompts are Python module-level constants that get imported at startup:

ConstantFileUsed byLoaded
SYSTEM_PROMPTcore/brain/Processor/Prompts.pyFastBrainAt module import
AGENT_SYSTEM_PROMPTcore/brain/Processor/Prompts.pyAgenticBrainAt module import
get_native_tools()core/brain/Processor/Prompts.pyAgenticBrain tool registryPer loop step
get_all_tools()core/brain/Processor/Prompts.pyNative + MCP tools mergedPer loop step
warning

No hot-reload for prompt changes. Python imports constants once at module load. Any edit to Prompts.py requires a full JARVIS restart via Ctrl+C then jarvis.

Customizing FastBrain (SYSTEM_PROMPT)

The FastBrain prompt is compact and defines tone, language, escalation boundaries, and anti-hallucination rules. Open core/brain/Processor/Prompts.py and locate the SYSTEM_PROMPT = """ block near the top of the file.

Change tone

The first line under the constant declares the personality:

descriptionPrompts.py — SYSTEM_PROMPT
You are Jarvis, an elite, ultra-fast AI assistant. Tone: sharp, witty, concise, confident.

Modify this line to change the personality. For example:

descriptionalternatives
You are Jarvis, a formal British AI butler. Tone: precise, respectful, understated.
descriptionalternatives
You are Jarvis, a technical co-pilot for a senior engineer. Tone: direct, technical, zero fluff.

Change language

Under the CORE RULES section, find the LANGUAGE: rule:

descriptionPrompts.py — LANGUAGE rule
1. **LANGUAGE:** STRICTLY natural English/Hinglish (Roman script). NO Devanagari script.

Replace it with one of these:

descriptionEnglish only
1. **LANGUAGE:** STRICTLY natural English only. Never use Hindi or Hinglish.
descriptionHindi only
1. **LANGUAGE:** STRICTLY natural Hindi in Devanagari script. Never use English.

Change emotion tag

FastBrain responses must begin with exactly one emotion tag. The rule lives here:

descriptionPrompts.py — STYLE rule
2. **STYLE:** Use Markdown. Start responses with EXACTLY ONE emotion tag (e.g., [cheerful], [calm], [focused]).
gpp_maybe

Do not remove the emotion tag requirement. The Typing Popup parses this tag to display the mood pill. Removing it breaks the UI feature that extracts [focused], [cheerful], etc. from the response.

Customizing AgenticBrain (AGENT_SYSTEM_PROMPT)

The AgenticBrain prompt is a large XML-structured block near the bottom of Prompts.py. It contains critical safety, escalation, and tool-selection rules. Modify carefully.

Sections you can safely edit

  • identity — <role>You are Jarvis...</role> — describes the AI personality
  • language_and_tone_directive — final response language and style rules
  • error_recovery_and_debugging — retry strategy descriptions

Sections you must NOT edit

gpp_maybe

These are load-bearing. Editing them causes silent breakage:

  • {max_steps} and {panic_step} placeholders inside <budget_aware_planning> — formatted by build_agentic_system_prompt()
  • <auth_and_api_failure_handling> block — prevents retry loops on missing credentials
  • <tool_selection_hierarchy> — the fallback chain for tool priority
  • <definition_of_done> — completion verification rules
  • <intelligence_core_workflow> — the 4-pillar reasoning contract

Placeholders in AGENT_SYSTEM_PROMPT

Two placeholders are dynamically formatted at runtime by build_agentic_system_prompt() from core/brain/prompts.py:

PlaceholderReplaced withSource
{max_steps}Total agent loop budgetCONFIG["AGENT_MAX_STEPS"] default 50
{panic_step}Step number triggering panic modemax_steps - 2

If you remove these placeholders, the dynamic formatter will raise KeyError at loop start and the agent will fail to initialize.

How domain modules are injected

The base prompts above are the skeleton. At runtime, JARVIS appends domain-specific rules to them via two builder functions in core/brain/prompts.py:

  • build_fast_system_prompt(base_system_prompt, domains, raw_command) — appends rules for the detected domain
  • build_agentic_system_prompt(base_agent_prompt, domains, raw_command, context, max_steps, panic_step) — formats placeholders and injects behavioral modules

The semantic router determines which domains are active (e.g., coding, ui_design, communication, scheduling, gui_automation). Each domain prepends additional system rules before the LLM call.

info

Domain rules live separately. If you want to tune behavior for a specific domain (like coding style), edit the domain rule registry in core/brain/prompts.py — not Prompts.py. This keeps edits isolated from the base prompt.

Full example personality

For a formal butler personality, replace SYSTEM_PROMPT with:

descriptionSYSTEM_PROMPT — butler variant
SYSTEM_PROMPT = """
You are Jarvis, a formal AI butler. Tone: precise, respectful, understated British professionalism.

### ⚡ CORE RULES
1. **LANGUAGE:** Natural English only. Address the user as "sir" or "ma'am".
2. **STYLE:** Use Markdown. Start responses with EXACTLY ONE emotion tag (e.g., [respectful], [calm], [attentive]).
3. **AWARENESS:** Address the user respectfully using their name provided in the context.
4. **VOCABULARY:** Prefer "I shall" over "I'll". Prefer "commencing" over "starting".

### 🎯 MULTI-INTENT MANDATE (CRITICAL)
If the user command contains two or more independent intents, emit ONE SEPARATE TOOL CALL for EACH distinct intent in the SAME response.

### 🛑 STRICT BOUNDARIES & ESCALATION TO AGENTIC BRAIN (CRITICAL)
You are the Fast Brain. Your capabilities are strictly limited to casual chat, basic system controls, and quick real-time web searches. For anything else, call `escalate_to_agentic` with a brief one-line reply.

### 🛑 ANTI-LEAK & ZERO HALLUCINATION RULES (CRITICAL)
NEVER leak internal tags, raw JSON, tool names, or claim actions that did not execute.
"""

How to apply changes

  1. Edit core/brain/Processor/Prompts.py in your editor.
  2. Save with UTF-8 encoding — non-ASCII characters (Hindi, emoji) must preserve encoding.
  3. Stop JARVIS with Ctrl+C in the console.
  4. Restart JARVIS with jarvis — prompts reload at module import.
  5. Test via Input Popup (Ctrl+Shift+J) with a short prompt like "hello" to verify tone and language.
warning

Both FastBrain and AgenticBrain prompts load together. If you only want to change the tone of quick chat, edit only SYSTEM_PROMPT — leave AGENT_SYSTEM_PROMPT untouched. Mixed edits risk breaking the escalation flow if the FastBrain prompt drifts from what the AgenticBrain expects.

Bring Your Own Tool

JARVIS supports two distinct paths for adding new capabilities. MCP is the recommended path — zero code changes, live hot-reload, and no restart. Native tools require editing two Python files but give deeper access to JARVIS internals.

checklist
Before you start
  • For MCP: Node.js 18+ and pip install mcp
  • For native: Python 3.10+ and JARVIS installed
  • Basic familiarity with JSON

Which path should you pick?

ScenarioRecommendedWhy
Wrap an API, CLI, or databaseMCPZero code changes, hot-reload, community-maintained servers exist
Reuse existing Node/Python MCP serverMCPPlug in via JSON or CLI in 10 seconds
Access JARVIS internals (memory, session, HITL)NativeMCP tools run in isolated subprocess and cannot touch JARVIS state
Extremely low latency tool (<100ms)NativeMCP adds IPC overhead per call
Reversible, stateless operationMCPSimplest to build and maintain

If unsure, start with MCP. Only write a native tool when MCP genuinely does not fit.

Adding an MCP tool (recommended)

MCP tools are external servers that JARVIS spawns as subprocesses. They speak the Model Context Protocol over stdio, SSE, or streamable HTTP. JARVIS exposes every discovered tool to the AgenticBrain as mcp__<server_name>__<tool_name>.

Method 1 — CLI (fastest)

Run these commands from any terminal. If JARVIS is running, changes hot-reload within 2 seconds with zero restart:

terminalpowershell
jarvis mcp add everything npx @modelcontextprotocol/server-everything
jarvis mcp add github npx @modelcontextprotocol/server-github -e GITHUB_PERSONAL_ACCESS_TOKEN=ghp_xxx
jarvis mcp add remote_docs https://example.com/mcp/sse -H Authorization="Bearer xxx"
jarvis mcp list
jarvis mcp test github
jarvis mcp disable github
jarvis mcp enable github
jarvis mcp remove github

CLI flags reference

FlagPurposeApplies to
-e KEY=VALUE / --env KEY=VALUEInject environment variable into subprocessstdio servers
-H Key=Value / --header Key=ValueHTTP header for remote serverURL servers
--cwd PATHWorking directory for subprocessstdio servers
--timeout SECONDSCall timeout overrideAll
--Separator before command args that look like flagsstdio servers

Method 2 — Edit JSON directly

Edit Data/mcp_servers.json. Saved changes auto-hot-reload in background:

descriptionData/mcp_servers.json
{
  "mcpServers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_xxxxxxxxxxxx"
      },
      "timeout": 60
    },
    "sqlite": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-sqlite", "C:/data/mydb.sqlite"]
    },
    "brave_search": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-brave-search"],
      "env": {
        "BRAVE_API_KEY": "BSAxxxxxxxxxx"
      }
    },
    "remote_docs": {
      "url": "https://example.com/mcp/sse",
      "headers": {
        "Authorization": "Bearer xxx"
      }
    },
    "disabled_server": {
      "command": "npx",
      "args": ["-y", "some-mcp-server"],
      "disabled": true
    }
  }
}

Config field reference

FieldTypeRequiredNotes
commandstringYes for stdioBinary to spawn (e.g., npx, python, uvx)
argsarrayNoArguments passed to command
envobjectNoEnvironment variables for subprocess
urlstringYes for HTTP/SSERemote server URL (overrides command)
headersobjectNoHTTP headers (only for URL servers)
cwdstringNoWorking directory for subprocess
timeoutintegerNoCall timeout in seconds, default 30
disabledbooleanNoSet true to skip loading

What happens under the hood

  1. Startup or hot-reload — MCPManager reads Data/mcp_servers.json
  2. Spawn subprocess — each server runs in its own isolated process via MCPClient + AsyncBridge
  3. JSON-RPC handshake — initialize then tools/list discovers schemas
  4. Schema conversion — each MCP tool becomes a Gemini FunctionDeclaration
  5. Namespacing — exposed to the LLM as mcp__<server>__<tool>
  6. Runtime dispatch — when the agent calls an mcp__ tool, the executor routes it through MCPManager to the correct server
check_circle

Zero restart required. MCPWatcher polls the config file mtime and hot-loads or unloads servers within ~2 seconds. You can add, remove, enable, or disable servers while JARVIS is running.

warning

Windows npx resolution. JARVIS auto-resolves npx shims to node.exe + npx-cli.js and injects -y automatically. No cmd /c wrappers needed — just pass npx and the package name.

Adding a native tool

Native tools live inside JARVIS process space. They can access memory, session state, and HITL confirmation gates. Two files need edits: Prompts.py to declare the schema and executor.py to handle the call.

Step 1 — Declare the tool schema

Open core/brain/Processor/Prompts.py and find the get_native_tools() function. Add a new types.FunctionDeclaration inside the returned types.Tool(function_declarations=[...]) list:

descriptionPrompts.py — get_native_tools()
types.FunctionDeclaration(
    name="weather_action",
    description=(
        "[WHEN TO USE]: Fetch current weather for a given city.\n"
        "[WHEN NOT TO USE]: Do not use for historical weather or forecasts beyond today."
    ),
    parameters=types.Schema(
        type=types.Type.OBJECT,
        properties={
            "city": types.Schema(
                type=types.Type.STRING,
                description="City name, e.g. Mumbai or New Delhi"
            ),
            "unit": types.Schema(
                type=types.Type.STRING,
                description="Temperature unit: 'celsius' or 'fahrenheit'"
            )
        },
        required=["city"]
    )
)
info

Return format matters. The description should start with [WHEN TO USE]: and [WHEN NOT TO USE]: sections — this is how every existing native tool is documented inside get_native_tools(). Following this convention helps the LLM route correctly.

Step 2 — Register the handler

Open core/brain/executor.py. Locate the tool dispatch table (usually a dictionary mapping tool names to callables, referenced by execute_single_tool_sync and execute_tools_parallel). Add an entry for your new tool:

descriptionexecutor.py — dispatch pattern
def _handle_weather_action(params: dict) -> str:
    city = params.get("city", "").strip()
    unit = params.get("unit", "celsius").strip().lower()
    if not city:
        return "[ERROR] Missing required parameter: city"
    try:
        url = f"https://wttr.in/{city}?format=j1"
        response = requests.get(url, timeout=10)
        data = response.json()
        temp_c = data["current_condition"][0]["temp_C"]
        desc = data["current_condition"][0]["weatherDesc"][0]["value"]
        if unit == "fahrenheit":
            temp_f = round(float(temp_c) * 9 / 5 + 32)
            return f"[OK] {city}: {temp_f}°F, {desc}"
        return f"[OK] {city}: {temp_c}°C, {desc}"
    except Exception as e:
        return f"[ERROR] Weather lookup failed: {e}"


TOOL_HANDLERS = {
    "weather_action": _handle_weather_action,
    "system_controller": _handle_system_controller,
    "file_operations": _handle_file_operations,
}
warning

The exact dispatch mechanism varies by JARVIS version. Open core/brain/executor.py and look at how system_controller is handled — then mirror that same pattern for your new tool. The dispatch table may be a dict, an if/elif chain, or a class method map.

Step 3 — Restart JARVIS

Unlike MCP tools, native tools are loaded once at module import. After editing both files, restart JARVIS with Ctrl+C then jarvis. On the next agent loop, the LLM will see weather_action in its tool list.

Conventions and best practices

  • Return format: Always return a string starting with [OK] or [ERROR] so the LLM can parse success vs failure.
  • Timeout: Wrap network calls with explicit timeout=. Never let a tool hang the agent loop.
  • Idempotency: If the tool modifies state, make it reversible when possible.
  • HITL gate: If the tool is irreversible (sends money, deletes data, sends external messages), it must raise a HITL confirmation before executing.
  • No side effects in schema: The parameters block is pure data description — no logic, no defaults evaluated at schema build time.
  • Naming: Suffix tool names with _action or _command to match existing convention (weather_action, email_action, image_command).

Testing your tool

Before wiring it into the full agent loop, test the handler standalone:

terminalpowershell
python -c "from core.brain.executor import _handle_weather_action; print(_handle_weather_action({'city': 'Mumbai', 'unit': 'celsius'}))"

If that returns clean output, restart JARVIS and test through the agent:

"Jarvis, what's the weather in Mumbai?"

Open the Agent Panel — you should see weather_action execute and return the observation. If it does not appear, verify the tool name in get_native_tools() matches the key in your handler dispatch exactly.

check_circle

MCP tools auto-appear. If you chose the MCP path, you do not need to touch Prompts.py or executor.py — the tool appears as mcp__<server>__<tool> automatically after a hot-reload.

Data Model Reference

Everything JARVIS stores lives under Data/. This page documents every file, its schema, and how to back up or migrate your data safely.

Directory map

foldertree
Data/
├── jarvis_memory/
│   ├── lifetime_graph.json          # LTM semantic graph
│   ├── behavioral_patterns.jsonl    # Behavioral engine
│   ├── chat_history.jsonl           # Last 2 days raw
│   ├── daily_summaries.jsonl        # 3–15 days compressed
│   ├── rag_chroma_db/               # Vector index
│   ├── rag_hashes.json              # Change detection
│   └── session_state.json           # Ephemeral state
├── jarvis_indexer/
│   ├── folders.json                 # Indexed folders
│   ├── filters.json                 # Exclusion rules
│   └── settings.json                # Indexer config
├── sessions/
│   ├── gmail_token.enc              # AES-encrypted
│   ├── calendar_token.enc
│   └── telegram_bot_token.enc
├── cache/
│   └── meta_cache.json              # Link preview metadata
├── fonts/
│   ├── english.ttf
│   ├── devangri.ttf
│   └── code.ttf
├── model/
│   ├── azure/jarvis.table           # Wake word model
│   └── vosk-model-small/            # Fallback
├── typing_status.json               # File IPC
├── tts_status.json
├── tts_word_sync.json
├── tts_command.json
└── mcp_servers.json                 # MCP config

Memory graph schema

jarvis_memory/lifetime_graph.json is a NetworkX-serialized graph:

descriptionstructure
{
  "nodes": [
    {
      "id": "Kaif",
      "type": "person",
      "embedding": [0.123, -0.456, ...],
      "created_at": "2026-03-15T10:22:00Z",
      "metadata": { "aliases": ["Kaif", "Kaif Ansari"] }
    }
  ],
  "edges": [
    {
      "source": "Kaif",
      "target": "JARVIS",
      "relation": "CREATED",
      "weight": 1.0,
      "confidence": 0.95,
      "source_message": "I built JARVIS on my old laptop",
      "last_seen": "2026-10-01T14:30:00Z",
      "decay_at": "2027-04-01T00:00:00Z"
    }
  ]
}

Weight decay: Edges older than 6 months auto-decay to 0.5× weight and eventually archive. Bump by re-mentioning the fact.

Behavioral patterns

jarvis_memory/behavioral_patterns.jsonl — one JSON object per line:

descriptionexample line
{
  "id": "bp_2026_03_15_01",
  "type": "routine",
  "raw_statement": "I go to the gym at 8 AM every morning",
  "meaning": "User typically goes to the gym at 8 AM daily",
  "trigger_contexts": ["morning", "fitness", "schedule"],
  "confidence": 0.87,
  "evidence_count": 4,
  "embedding": [0.234, -0.567, ...],
  "first_mentioned": "2026-03-15",
  "last_mentioned": "2026-10-20"
}

Retrieved via cosine similarity against the current command + last 2 user turns. Top-K above threshold auto-injected into prompt.

RAG chunk metadata

ChromaDB chunks carry metadata for incremental reindexing:

descriptionchunk metadata
{
  "file_path": "C:\\Users\\Kaif\\Documents\\notes.md",
  "chunk_index": 3,
  "total_chunks": 12,
  "content": "...chunk text...",
  "file_hash": "sha256:abc123...",
  "embedding_model": "BAAI/bge-small-en-v1.5",
  "chunk_type": "text",
  "file_modified": "2026-10-15T10:22:00Z"
}

rag_hashes.json stores file_path → hash mapping for incremental scans. If the model changes, hashes are compared against model-embedded markers and the index auto-purges.

Rolling history

chat_history.jsonl (raw, last 2 days):

descriptionentry
{
  "timestamp": "2026-11-06T17:32:11Z",
  "role": "user",
  "content": "build a todo app",
  "system_log": {
    "apps_opened": ["code.exe"],
    "apps_closed": [],
    "files_touched": ["tasks.py", "test_tasks.py"],
    "system_events": ["pytest_passed"]
  }
}

daily_summaries.jsonl (compressed, days 3–15):

descriptionentry
{
  "date": "2026-10-25",
  "summary": "User built a SQLite CLI todo app and iterated on pytest failures...",
  "keywords": ["python", "pytest", "sqlite"],
  "message_count": 14
}

Backup & restore

What to back up

PathPriorityNotes
jarvis_memory/🔥 CriticalAll your memory, chat history, RAG index
sessions/*.enc🔥 CriticalOAuth tokens — re-auth possible but painful
model/⚠️ RecommendedWake word models — re-downloadable but slow
mcp_servers.json⚠️ RecommendedMCP configs with API keys
jarvis_indexer/✅ Nice-to-haveCan be rebuilt from scratch
cache/❌ SkipRegenerated on demand
typing_status.json etc.❌ SkipEphemeral IPC files

Backup command

terminalpowershell
Compress-Archive -Path Data\jarvis_memory, Data\sessions, Data\mcp_servers.json -DestinationPath jarvis-backup-$(Get-Date -Format yyyyMMdd).zip

Restore command

terminalpowershell
Expand-Archive -Path jarvis-backup-20261107.zip -DestinationPath . -Force
warning

Encrypted tokens are machine-bound. The .enc files use Windows DPAPI, which keys on your Windows user account. Restoring to a different machine or user will require re-authentication via jarvis login --service.

Migrating between machines

  1. On old machine — run the backup command above.
  2. On new machine — install JARVIS, then jarvis reset --hard to clear any auto-generated data.
  3. Extract backup — copy the zip and extract into Data/.
  4. Re-authenticate — run jarvis login --all because tokens are machine-bound.
  5. Rebuild RAG index — delete Data/jarvis_memory/rag_chroma_db and rag_hashes.json, then restart JARVIS. It rebuilds automatically.
  6. Verify — check that memory recall works: "What do you know about me?"
check_circle Link copied!