Jarvis AI Voice Assistant Project

1. Project Objective

Build a fully local AI voice assistant that can:

• Listen through a microphone
• Convert speech to text using Whisper
• Process requests using a local LLM (Gemma4 via Ollama)
• Respond using text-to-speech
• Run continuously in a loop

Goal :  to create a privacy-focused voice assistant that runs entirely on local hardware without relying on cloud AI services.

2. Environment Setup

Create Project Directory
mkdir ~/jarvis
cd ~/jarvis
Create Python Virtual Environment
python3 -m venv venv
source venv/bin/activate

Verify virtual environment:

which python3
which pip

Expected output should point to:

~/jarvis/venv/bin/

3. Python Dependencies

Install required packages:

pip install faster-whisper
pip install ollama
pip install pyttsx3
pip install sounddevice
pip install scipy

Or install all together:

pip install faster-whisper ollama pyttsx3 sounddevice scipy

4. Ollama Setup

Verify Ollama installation:

ollama --version

List available models:

ollama list

Installed models:

gemma4:latest
gemma4:e4b
gemma3:12b
llama3.1:8b

Test Gemma4 manually:

ollama run gemma4:latest

Example:

>>> What is cybersecurity?

Model successfully generated responses.

5. Faster-Whisper Validation

from faster_whisper import WhisperModel

model = WhisperModel(
"base",
device="cpu",
compute_type="int8"
)

segments, info = model.transcribe(
"test.wav",
vad_filter=True,
beam_size=5
)

print("Language:", info.language)

text = ""

for segment in segments:
print(segment.text)
text += segment.text

print("\nFULL:", text)

6. Text-to-Speech Validation

import pyttsx3

engine = pyttsx3.init()
engine.say("Jarvis is online and working")
engine.runAndWait()

7. Initial AI Integration Test

Created file:

nano jarvis_test.py

Contents:

import ollama
import pyttsx3

engine = pyttsx3.init()

user_input = "What is cybersecurity?"

response = ollama.chat(
model="gemma4:latest",
messages=[
{"role": "user", "content": user_input}
]
)

reply = response["message"]["content"]

print("Jarvis:", reply)

engine.say(reply)
engine.runAndWait()

Execute:

python3 jarvis_test.py

8. Real-Time Jarvis Assistant

Created:

nano jarvis_realtime.py

import sounddevice as sd
from scipy.io.wavfile import write
from faster_whisper import WhisperModel
import ollama
import pyttsx3
import time

samplerate = 16000
duration = 5

model = WhisperModel(
"base",
device="cpu",
compute_type="int8"
)

engine = pyttsx3.init()

def record_audio():
print("🎤 Listening...")

recording = sd.rec(
int(duration * samplerate),
samplerate=samplerate,
channels=1,
dtype="int16"
)

sd.wait()

return recording

try:
while True:

audio = record_audio()

write(
"temp.wav",
samplerate,
audio
)

segments, info = model.transcribe(
"temp.wav"
)

text = " ".join(
[seg.text for seg in segments]
).strip()

if not text:
continue

print("\n🧑 You said:", text)

response = ollama.chat(
model="gemma4:latest",
messages=[
{
"role": "system",
"content": (
"You are Jarvis, a concise AI assistant. "
"Always reply in 1 or 2 short sentences. "
"Avoid long explanations."
)
},
{
"role": "user",
"content": text
}
]
)

reply = response["message"]["content"]

print("🤖 Jarvis:", reply)

engine.say(reply)
engine.runAndWait()

time.sleep(0.5)

except KeyboardInterrupt:
print("\n👋 Jarvis stopped.")

9. Running Jarvis

Activate environment:

cd ~/jarvis
source venv/bin/activate

Start assistant:

python3 jarvis_realtime.py

Example output:

🎤 Listening...

🧑 You said: Hello Jarvis

🤖 Jarvis: Hello! How can I help you today?

10. Final Status Checklist

✅ Python Virtual Environment — Complete

✅ Faster-Whisper Installation — Complete

✅ Whisper Model Download — Complete

✅ Speech-to-Text (STT) Functionality — Complete

✅ Ollama Installation — Complete

✅ Gemma4 Model Setup — Complete

✅ Text-to-Speech (TTS) Functionality — Complete

✅ AI Response Generation — Complete

✅ Real-Time Voice Assistant Integration — Complete

Conclusion

This project successfully built a fully local AI voice assistant capable of listening, understanding speech, processing requests with a local LLM, and responding through voice output. By integrating Faster-Whisper, Ollama (Gemma4), and pyttsx3, a complete end-to-end conversational system was achieved.

While Version 1 is fully functional, there is still plenty of room for improvement and new features to explore. This serves as a solid foundation for future enhancements and experimentation.