From: Mike Lowis Date: Tue, 29 Sep 2026 16:43:07 +0000 (-0400) Subject: initial commit X-Git-Url: https://git.mdlowis.com/?a=commitdiff_plain;h=887277a625290711eb9789c32958847dda37dec9;p=proto%2Fnlshell.git initial commit --- 887277a625290711eb9789c32958847dda37dec9 diff --git a/docs/FlintNPC.md b/docs/FlintNPC.md new file mode 100644 index 0000000..5594301 --- /dev/null +++ b/docs/FlintNPC.md @@ -0,0 +1,310 @@ +## FlintNPC + +FlintNPC is the conversational agent framework built on top of FlintParser. It orchestrates the deterministic NLU engine into a full-featured, multi-turn conversational system with tool-call support, sentiment tracking, thinking mode, and context memory. + +FlintNPC is available in identical, cross-compliant implementations for both [Python](src/flint_npc.py), for local OS environments, and [JavaScript](src/flint_npc.js), running client-side inside any browser tab or Node.js instance. + +### The Engineering Philosophy + +FlintNPC relies on **subtraction engineering**: instead of adding probabilistic models, it subtracts noise until only deterministic, auditable behavior remains. The framework implements the following pipeline: + +``` +User Prompt + +1. Sanitize & Strip (remove noise) +2. Exact Match (O(1) hash lookup) +3. Template Match (semantic structure parsing) +4. Probabilistic Match (IDF-weighted Levenshtein) +5. Response or rejection (with related intent suggestions) +``` + +### Features + +- **Four-Stage Intent Resolution** Exact match, template match, probabilitstic match, rejection +- **Multi-Turn Context Memory** Persistent conversational state with overwrite semantics +- **Sentiment Analysis** Tracks expletives, thanking words, encouraging or discouraging words +- **Tool-Call Support** Extracts slots and triggers tool execution with permission gating +- **Thinking Mode** Inner monologue that explains the agent's response +- **Composite Prompt Splitting** Handles multi-command inputs (`cmd1; cmd2`) +- **Metadata Rendering** Macro substitution for dynamic responses (`<||username||>`, `<||npc_name||>`) +- **Related Intent Suggestions** Generates suggestions based on rarest word matching +- **Threshold Tuning** Adaptive error tolerance based on prompt length +- **Permission Gating** Controls which actions require user confirmation + +### How It Works + +FlintNPC processes input through a four-stage pipeline, each stage more permissive than the last: + +#### Stage 1: Sanitization & Sentiment Extraction + +```python +stripped, sentiment = parser.strip_and_sentiment( + "darn, can you show me the red tie?", + vocabulary, + current_sentiment +) +# Returns: ("can you show me the red tie?", {"expletives": 1, ...}) +``` + +The parser strips noise (insults, interjections, emojis) while counting occurrences. This feeds the sentiment system and cleans the prompt for matching. + +#### Stage 2: Exact Match (O(1) Hash Lookup) + +```python +if stripped in exact_match_map: + matched_block = exact_match_map[stripped] + return generate_response(matched_block, confidence=1.0, status="exact match") +``` + +Every intent variant in the dataset is indexed in a hash map. If the sanitized prompt matches exactly, we return immediately with 100% confidence. This is the fastest path. + +**Context Override**: Active context entries (from multi-turn conversations) override the global dataset on key collision. This allows follow-up intents like "move it" to take priority over unrelated global intents. + +#### Stage 3: Template Match (Semantic Structure Parsing) + +```python +structure, slots = parser.parse_structure("show red tie", threshold=0.8) +# structure: ["<||vocab_show||>", "<||color||>", "<||vocab_tie||>"] +# slots: {"color": "red"} + +match = parser.match_structure(templates, structure) +# Returns the intent block whose structure matches +``` + +The parser converts the prompt into an ordered sequence of semantic tags, then walks each template's structure paths to find a match. Variables (like `<||color||>`) are extracted via regex and stored in slots. + +#### Stage 4: Probabilistic Match (IDF-Weighted Levenshtein) + +```python +score = parser.sentence_similarity("show me a red tie", "show red tie", threshold=0.66) +# Returns: 0.85 (high confidence match) +``` + +If no exact or template match is found, the parser computes an asymmetric, order-independent similarity score against all intent variants: +- **Anchor tokens** (from the intent phrase) are weighted by their IDF specificity +- **Query tokens** (from the user prompt) are fuzzy-matched against anchors via Levenshtein distance +- **OOV penalty** tokens absent from the vocabulary reduce the score + +The highest-scoring intent above the threshold is returned. + +#### Stage 5: Rejection (With Suggestions) + +```python +if best_score < threshold: + related_intents = generate_suggestions(rarest_word, dataset) + return generate_response(rejection, confidence=0.0, status="rejected", related=related_intents) +``` + +If no match clears the threshold, the agent returns a rejection with up to 5 related intent suggestions based on the rarest (most specific) word in the prompt. + +### Code Examples + +#### Basic Usage + +```python +from flint_npc import FlintNPC + +# Initialize an NPC from a dataset directory +npc = FlintNPC("TERMy", log_level="INFO") + +# Process a user prompt +response = npc.process_message("show me a red tie") + +print(response["response"]) # "Certainly. A selection from the tie atelier..." +print(response["status"]) # "template match" +print(response["confidence"]) # 1.0 +print(response["slots"]) # {"color": "red"} +print(response["tools"]) # [{"name": "ShowProducts", "arguments": {...}}] +``` + +#### Multi-Turn Context Memory + +FlintNPC maintains conversational state across multiple turns. Context entries persist until explicitly overwritten: + +```python +# Turn 1: User says "create a directory" +res1 = npc.process_message("create a directory") +# Context is now: ["move it", "delete it"] + +# Turn 2: User says "move it" +res2 = npc.process_message("move it") +# Context persists, "delete it" still active + +# Turn 3: User says "delete it" +res3 = npc.process_message("delete it") +# Both follow-ups worked without re-specifying the directory +``` + +Context semantics: +- `"context": [...]` → overwrites active context with new entries +- `"context": []` → explicitly clears the context +- No `"context"` key → context persists unchanged + +#### Composite Prompt Splitting + +FlintNPC can handle multi-command inputs by splitting on separators: + +```python +response = npc.process_messages("list files; show cpu temperature; goodbye") +# Splits into 3 sub-prompts, processes each sequentially +# Returns aggregated response with concatenated output +``` + +Separators: newlines, numbered lists (`1)`), semicolons, exclamation marks, question marks. + +#### Tool-Call Execution + +FlintNPC extracts tool calls from matched intents: + +```python +response = npc.process_message("show me a red tie") +# response["tools"] = [ +# { +# "name": "ShowProducts", +# "arguments": {"category": "ties", "color": "red"}, +# "goal": "Product discovery" +# } +# ] + +# Your harness executes the tool: +for tool in response["tools"]: + result = execute_tool(tool["name"], tool["arguments"]) + # result: HTML cards with red ties +``` + +#### Thinking Mode + +FlintNPC can emit an inner monologue that explains its reasoning: + +```python +response = npc.process_message("show me a red tie") +print(response["thinking"]) +# "A tie request. The only honest piece of clothing left. Opening the catalogue..." +``` + +Thinking is defined in the dataset and can include slot substitution (`<||color||>`). + +#### Metadata Rendering + +FlintNPC substitutes macros in responses with runtime values: + +```python +# In dataset: "Hello <||username||>, I am <||npc_name||>" +response = npc.process_message("hello") +# response["response"]: "Hello nonar, I am TERMy" +``` + +Available metadata: +- `<||username||>` current user (from `$USER` env var) +- `<||npc_name||>` NPC name (from config) +- `<||computer_name||>` hostname +- `<||creation_date||>` dataset creation date +- `<||creator||>` dataset author +- `<||completion||>` random completion phrase +- `<||unknown||>` random "unknown" response +- `<||related||>` related intent suggestions + +#### Sentiment Tracking + +FlintNPC tracks sentiment across the conversation: + +```python +response = npc.process_message("thanks for the help, you're great!") +print(response["sentiment"]) +# {"expletives": 0, "thanking_words": 1, "encouraging_words": 1, ...} + +print(response["emoji"]) +# "😃" (thanking + encouraging = happy) +``` + +Sentiment is aggregated across composite prompts and persists across turns. + +### Configuration + +FlintNPC reads configuration from `npcs//config.json`: + +```json +{ + "npc_name": "TERMy", + "creator": "Giovanni Blu Mitolo", + "creation_date": "2026-07-12", + "sentence_threshold": 0.75, + "word_threshold": 0.75, + "suggestions": 5 +} +``` + +- **sentence_threshold** minimum similarity score for probabilistic match on prompts >3 words +- **word_threshold** stricter threshold for short prompts (1-3 words) +- **suggestions** max number of related intent suggestions on rejection + +### Performance Characteristics + +- **Zero Dependencies** Only Python standard library + FlintParser +- **CPU-Only** No GPU, no CUDA, no PyTorch. Runs on a Raspberry Pi Zero. +- **Deterministic** Same input always produces the same output. Testable in CI. +- **Microsecond Latency** Exact match: <10μs. Template match: <1ms. Probabilistic: <10ms. +- **Memory Efficient** The entire agent + dataset fits in <10MB RAM. + +### Integration with NPC-Forge + +FlintNPC is the conversational layer that powers NPC-Forge agents. It wraps FlintParser and provides: +- Multi-turn context management +- Tool-call orchestration +- Sentiment aggregation +- Metadata rendering +- Composite prompt handling + +The NPC-Forge CLI uses FlintNPC to serve agents: + +```bash +npc-forge serve # Start OpenAI-compatible API server +npc-forge install npcs/termy # Install TERMy +termy show me a red tie # Chat with TERMy via CLI +``` + +### Algorithmic Choices + +#### Why Four Stages? + +Most NLU frameworks use a neural classifier. FlintNPC uses a pipeline: +1. **Exact match** is O(1) and covers most queries +2. **Template match** is O(n) and covers structured queries with variables +3. **Probabilistic match** is O(n*m) and covers typos and paraphrases + +Each stage is computationally heavier than the previous. + +#### Why Context Override? + +Multi-turn conversations require state. Instead of a complex dialogue manager, FlintNPC uses a simple hash map: +- Active context entries override the global dataset on key collision +- Context persists until explicitly overwritten +- Rejection clears the context + +This allows flows like "create dir" → "move it" → "delete it" without re-specifying the directory each time. + +#### Why Composite Prompt Splitting? + +Users often issue multiple commands at once (`list files; show red tie`). Instead of rejecting the input, FlintNPC splits on separators and processes each sub-prompt sequentially. Responses are concatenated. + +#### Why Adaptive Thresholds? + +Short prompts (1-3 words) are highly sensitive to typos. A single character change can flip the meaning (`who` vs `how`). Long prompts are more tolerant of errors because the rarest word still anchors the match. + +FlintNPC uses: +- **word_threshold** (default 0.75) for prompts ≤3 words +- **sentence_threshold** (default 0.75) for prompts >3 words + +### Limitations + +FlintNPC is not a replacement for LLMs. It's a **deterministic conversational framework** for structured, predictable interactions. It cannot: +- Understand open-ended, creative language +- Perform reasoning or inference beyond pattern matching +- Handle multi-language without per-language datasets +- Generate novel responses beyond what's in the dataset + +For open-ended chit-chat, the NPC-Forge framework can **escalate** to an LLM when FlintNPC returns a "rejected" status. This hybrid approach gives you the best of both worlds: deterministic speed for 80% of queries, probabilistic flexibility for the rest. + +### License + +Licensed under the [AGPL-3.0](../LICENSE). Commercial licensing available. diff --git a/docs/FlintParser.md b/docs/FlintParser.md new file mode 100644 index 0000000..7fa510c --- /dev/null +++ b/docs/FlintParser.md @@ -0,0 +1,240 @@ +## FlintParser + +FlintParser is a lightweight, zero-dependency, deterministic Natural Language Understanding (NLU) semantic parser. Converts user's questions into semantic structures while implementing Named Entity Recognition (NER). + +FlintParser is available in identical, cross-compliant implementations for both [Python](src/flint_parser.py), for local OS environments, and [JavaScript](src/flint_parser.js), running client-side inside any browser tab or Node.js instance. + +### The Engineering Philosophy + +FlintParser relies on **subtraction engineering** and can be described as, not only a parser, but also a transpiler, because it is capable of translating plain English into semantic paths with the following format: + +```json +"structure": [[ + { + "tag": "<||vocab_prefer||>", + "type": "vocab", + "required": true + }, + { + "tag": "<||a||>", + "type": "word", + "required": true + }, + { + "tag": "<||vocab_or||>", + "type": "vocab", + "required": true + }, + { + "tag": "<||b||>", + "type": "word", + "required": true + } +]] +``` + +The semantic path above represents the prompt `Do you prefer <||a||> or <||b||>`, where `<||a||>` and `<||b||>` can be substituted with any word, for example `do you prefer complexity or simplicity`. + +#### Why Subtraction? + +Most NLU frameworks **add** complexity: they train neural networks, they build embedding spaces, they maintain massive vocabularies. FlintParser does the opposite. It **subtracts** noise until only the meaning remains: + +1. **Strips the noise** removing insults, interjections, emojis, filler words +2. **Extracts the structure** converting words to semantic tags (`<||vocab_show||>`, `<||color||>`) +3. **Matches patterns** comparing the structure against known templates +4. **Confidence ranking** using IDF-weighted similarity to rank matches + +### Features + +- **Text Sanitization** Strips insults, emojis, interjections while tracking sentiment +- **Template Semantic Parsing** Converts text into ordered semantic tag sequences +- **Sentence Similarity** Asymmetric, order-independent BOW IDF + Levenshtein scoring +- **Subject Identification** Extracts the rarest (most specific) word via IDF +- **Sentiment Analysis** Counts expletives, thanking words, encouraging/discouraging tokens +- **Emoji Generation** Generates an emoji based on sentiment density +- **Typo Tolerance** Fuzzy-matches vocabulary tags with configurable threshold +- **Variable Extraction** Regex-based slot extraction with string literal support +- **Candidate Prediction** Predicts next expected tags for autocomplete/debugging + +### How It Works + +FlintParser processes input through a five-stage pipeline: + +#### 1. Sanitization & Sentiment Extraction + +```python +parser.strip_and_sentiment( + "darn, can you show me the red tie?", + sentiment_json, + current_sentiment={"expletives": 0, "thanking_words": 0, ...} +) +# Returns: ("can you show me the red tie?", {"expletives": 1, ...}) +``` + +The parser strips noise while counting occurrences. This feeds the sentiment system and cleans the prompt for parsing. + +#### 2. Tokenization & Filtering + +```python +# Input: "can you show me the red tie?" +# After stop-word filtering: ["show", "red", "tie"] +``` + +Stop words, expletives, and interjections are removed. What remains are the semantically meaningful tokens. + +#### 3. Structure Parsing (The Transpiler) + +```python +structure, slots = parser.parse_structure("show red tie", threshold=0.8) +# structure: ["<||vocab_show||>", "<||color||>", "<||vocab_tie||>"] +# slots: {"color": "red"} +``` + +Each token is either: +- **Matched to a vocabulary tag** via fuzzy Levenshtein similarity (handles typos like `"shwo"` → `<||vocab_show||>`) +- **Extracted as a variable** via regex (e.g., `<||color||>` matches `"red"`, `"blue"`, `"burgundy"`) +- **Ignored** if it doesn't match any candidate + +#### 4. Template Matching + +```python +match = parser.match_structure(templates, structure) +# Returns the intent block whose structure matches the parsed structure +``` + +The parser walks each template's structure paths, checking if the parsed tags align. Optional nodes are skipped if not present. + +#### 5. Confidence Scoring + +```python +score = parser.sentence_similarity("show me a red tie", "show red tie", threshold=0.66) +# Returns: 0.85 (high confidence match) +``` + +For probabilistic fallback (when no exact template matches), the parser computes an asymmetric, order-independent similarity score: +- **Anchor tokens** (from the intent phrase) are weighted by their IDF specificity +- **Query tokens** (from the user prompt) are fuzzy-matched against anchors +- **OOV penalty** tokens absent from the vocabulary reduce the score + +### Code Examples + +#### Auto-Weighted IDF + +FlintParser automatically computes word rarity across your intent corpus: + +```python +# After initialization, parser.weights contains IDF scores: +# {"tie": 1.0, "show": 0.8, "the": 0.05, "a": 0.03} + +rarest = parser.get_rarest_word("show me a red tie") +# Returns: "tie" (highest IDF weight) +``` + +Rare words get high weight (they're specific to this intent). Common words get low weight (they appear everywhere). + +#### Sentiment-Driven Emoji + +```python +sentiment = { + "expletives": 2, + "thanking_words": 0, + "encouraging_words": 0, + "discouraging_words": 0, + "interjections": 0 +} +emoji = parser.sentiment_emoji(sentiment) +# Returns: "😔" (two expletives, no thanks = sad/frustrated) +``` + +The emoji system is cynical and nerdy it doesn't fake enthusiasm. One expletive = `😒`, five expletives = `😱`. + +#### Typo-Tolerant Vocabulary Matching + +```python +tag = parser.find_template("shwo", threshold=0.8) +# Returns: "<||vocab_show||>" (fuzzy match on typo) +``` + +Levenshtein similarity with early-exit pruning makes this fast even on large vocabularies. + +#### String Literal Extraction + +```python +structure, slots = parser.parse_structure('search for "red silk tie"', threshold=0.8) +# slots: {"query": "red silk tie"} +``` + +Quoted strings are captured as a single slot, preserving spaces and punctuation. + +### Performance Characteristics + +- **Zero Dependencies** Only `re`, `string`, `math` from the Python standard library +- **CPU-Only** No GPU, no CUDA, no PyTorch. Runs on a Raspberry Pi Zero. +- **Deterministic** Same input always produces the same output. Testable in CI. +- **Microsecond Latency** Parsing 1,000 intents takes <10ms on a modern CPU. +- **Memory Efficient** The entire parser + dataset fits in <5MB RAM. + +### Integration with NPC-Forge + +FlintParser is the NLU engine that powers NPC-Forge agents. It's instantiated once per NPC with the dataset: + +```python +from flint_parser import FlintParser + +parser = FlintParser( + name="TERMy", + vocabulary=load_json("dataset/vocabulary/words.json"), + templates=load_json("dataset/templates.json"), + templates_vocabulary=load_json("dataset/vocabulary/templates.json"), + variable_types=load_json("dataset/types.json"), + intents=load_json("dataset/dataset.json") +) + +# Parse a user prompt +structure, slots = parser.parse_structure("show me a red tie", threshold=0.8) + +# Find the matching intent +match = parser.match_structure(parser.templates, structure) +``` + +The parser is stateless and thread-safe. It can be called from multiple concurrent requests without locks. + +### Algorithmic Choices + +#### Why Levenshtein? + +Neural embeddings are overkill for small vocabularies. Levenshtein distance is: +- **Deterministic** Same input, same output +- **Explainable** You can see exactly which characters differ +- **Fast** O(n*m) with early-exit pruning +- **Zero-dependency** No pre-trained models to download + +#### Why IDF Weighting? + +TF-IDF (Term Frequency-Inverse Document Frequency) is the gold standard for keyword extraction. By weighting rare words heavily, the parser: +- Prevents "the", "a", "is" from dominating similarity scores +- Makes short identifiers (like programming language names "c", "r") detectable +- Allows the threshold to be tuned for the specific dataset + +#### Why Asymmetric Similarity? + +The parser treats the intent phrase as the "anchor" and the user prompt as the "query". This means: +- Missing anchor words heavily penalize the score (the user didn't say something important) +- Extra query words are tolerated (the user added filler) +- Word order doesn't matter (the parser is order-independent) + +This matches how humans actually communicate: we add noise, we skip words, but we rarely omit the core subject. + +### Limitations + +FlintParser is not a replacement for LLMs. It's a **deterministic front-end** for structured, predictable conversations. It cannot: +- Understand open-ended, creative language +- Perform reasoning or inference beyond pattern matching +- Handle multi-language without per-language vocabularies +- Extract entities that aren't in the variable type regexes + +For open-ended chit-chat, the NPC-Forge framework can **escalate** to an LLM when FlintParser returns a "rejected" status. This hybrid approach gives you the best of both worlds: deterministic speed for 80% of queries, probabilistic flexibility for the rest. + +### License + +Licensed under the [AGPL-3.0](../LICENSE). Commercial licensing available. diff --git a/docs/NPC-Forge-API.md b/docs/NPC-Forge-API.md new file mode 100644 index 0000000..f59d511 --- /dev/null +++ b/docs/NPC-Forge-API.md @@ -0,0 +1,231 @@ +# NPC-Forge API Documentation + +The NPC-Forge API is a Flask-based HTTP server (defaulting to `http://127.0.0.1:5000`) that provides interfaces for interacting with installed NPCs. It exposes both **Native Endpoints** for direct application integration and **OpenAI-Compatible Endpoints** to allow integration with third-party tools, IDE extensions like VS Code, and LLM harnesses. + + +### Native Endpoints + +These endpoints are designed for direct, straightforward interaction with a specific NPC. + +#### NPC Chat Interface +Returns a compiled, standalone HTML chat interface for the specified NPC. + +- **Endpoint**: `GET //chat/` +- **Response Type**: `text/html` + +**Success Response** (`200 OK`): +```html + +``` + +**Error Response** (`500 Internal Server Error`): +```json +{ + "error": "Internal Server Error: Failed to compile chat interface." +} +``` +*(Note: Internal stack traces are intentionally suppressed in the response and only logged server-side.)* + +--- + +#### Chat Request +Sends a text message directly to an NPC's engine and returns the raw processed result. + +- **Endpoint**: `POST /api/chat/` +- **Request Headers**: `Content-Type: application/json` + +**Request Body**: +```json +{ + "message": "Hello, what can you do?" +} +``` + +**Success Response** (`200 OK`): +```json +{ + "response": "I can help you with various tasks...", + "status": "exact match", + "thinking": "Analyzing user intent...", + "tools": [] +} +``` + +**Error Responses**: +- `400 Bad Request`: `{"error": "The field 'message' is mandatory"}` +- `404 Not Found`: `{"error": "NPC profile '' not found on disk"}` +- `500 Internal Server Error`: `{"error": "Internal Server Error", "details": "An unexpected error occurred..."}` + +--- + +### OpenAI-Compatible Endpoints + +These endpoints mimic the OpenAI API specification, allowing NPC-Forge to act as a drop-in replacement for LLM providers in compatible clients. + +#### List Available NPCs +Returns a list of all installed NPC profiles, formatted as OpenAI "models". + +- **Endpoint**: `GET /api/v1/models` + +**Success Response** (`200 OK`): +```json +{ + "object": "list", + "data": [ + { + "id": "termy", + "object": "model", + "created": 1690000000, + "owned_by": "user" + }, + { + "id": "guard_bot", + "object": "model", + "created": 1690000500, + "owned_by": "user" + } + ] +} +``` + +**Error Response** (`500 Internal Server Error`): +```json +{ + "error": "Failed to list available models", + "details": "" +} +``` + +--- + +#### Chat Completions +Processes a chat completion request. This endpoint includes custom parsing logic to handle tool execution results and specific XML tagging. + +- **Endpoint**: `POST /api/v1/chat/completions` +- **Request Headers**: `Content-Type: application/json` + +**Request Body**: +```json +{ + "model": "termy", + "messages": [ + { + "role": "user", + "content": " List the files in the current directory " + } + ] +} +``` + +**Success Response** (`200 OK` - Standard Reply): +```json +{ + "id": "npc-forge-1690000000000", + "object": "chat.completion", + "created": 1690000000000, + "model": "termy-0", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "Here are the files in the directory...", + "reasoning_content": "The user wants to list files. I will use the ls tool.", + "refusal": null, + "annotations": [] + }, + "logprobs": null, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 19, + "completion_tokens": 10, + "total_tokens": 29 + }, + "service_tier": "default" +} +``` + +**Success Response** (`200 OK` - Tool Call Reply): +*If the NPC engine determines that tools should be executed, the response includes a `tool_calls` array.* +```json +{ + "id": "npc-forge-1690000000000", + "object": "chat.completion", + "created": 1690000000000, + "model": "termy-0", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": null, + "reasoning_content": "Executing directory listing.", + "refusal": null, + "annotations": [], + "tool_calls": [ + { + "id": "call_1690000000000_0", + "type": "function", + "function": { + "name": "run_in_terminal", + "arguments": "{\"command\": \"ls -la\"}" + } + } + ] + }, + "logprobs": null, + "finish_reason": "tool_calls" + } + ], + "usage": { + "prompt_tokens": 19, + "completion_tokens": 10, + "total_tokens": 29 + }, + "service_tier": "default" +} +``` + +**Success Response** (`200 OK` - Tool Result Interception): +*When the client sends back the result of a tool execution.* +```json +{ + "id": "npc-forge-1690000000000", + "object": "chat.completion", + "created": 1690000000000, + "model": "gpt-5.4", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "total 8\ndrwxr-xr-x 2 user user 4096 Sep 2 10:00 .\ndrwxr-xr-x 5 user user 4096 Sep 2 09:00 ..", + "refusal": null, + "annotations": [] + }, + "logprobs": null, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 10, + "completion_tokens": 5, + "total_tokens": 15 + }, + "service_tier": "default" +} +``` + +**Error Responses**: +- `400 Bad Request`: `{"error": "Empty messages stack"}` (if `messages` array is missing or empty). +- `404 Not Found`: `{"error": "NPC profile 'termy' not found on disk"}` (only if the fallback 'termy' engine fails to initialize). +- `500 Internal Server Error`: Generic server failure during directory scanning or request processing. + +--- + +### Notes for Developers +- **CORS**: Cross-Origin Resource Sharing (CORS) is enabled by default on the Flask app, allowing web-based frontends to interact with the API. +- **Logging**: All requests, extracted prompts, tool calls, and errors are logged via the internal `logger` module. Full tracebacks for `500` errors are written to the rotating log file but are never exposed to the client. +- **Production Deployment**: The built-in Flask development server is used by default. For production, it is highly recommended to serve the app via a WSGI server (e.g., `gunicorn -w 4 -b 127.0.0.1:5000 server:app`). diff --git a/docs/NPC-Forge-cli.md b/docs/NPC-Forge-cli.md new file mode 100644 index 0000000..5ce7cc6 --- /dev/null +++ b/docs/NPC-Forge-cli.md @@ -0,0 +1,90 @@ +## `npc-forge` CLI + +The NPC-Forge CLI provides an interface to administer, install, and run your NPCs. This tool serves NPCs, streams logs and executes framework tests. Run it using the `npc-forge` command. + +```bash +npc-forge [options] +``` + +### Service Management + +| Command | Aliases | Description | +| :--- | :--- | :--- | +| `serve` | `start` | Starts the background `npc-forge.service` systemd user daemon. | +| `stop` | - | Stops the background registry service. | +| `restart` | `reboot` | Restarts the background registry service. | + +```bash +npc-forge start +npc-forge stop +npc-forge restart +``` + +### NPC Management + +#### `create ` +Scaffolds a new NPC profile from the built-in example template. Creates a new directory at `~/.local/share/npc-forge/npcs/` and populates it with the standard dataset, vocabulary, and configuration files. The `name` property in `config.json` is automatically updated to match the provided name. + +```bash +npc-forge create my_new_npc +``` + +It is possible to test the newly created NPC executing `npc-forge serve` and then visiting [127.0.0.1:5000/my_new_npc/chat](http://127.0.0.1:5000/my_new_npc/chat). + +#### `install [--dev]` +Installs an NPC profile from a local directory into the `npc-forge` registry. Copies the entire source directory (datasets, scripts, entry points) to `~/.local/share/npc-forge/npcs/` and executes `setup.sh` if it exists in the source directory. Use `--dev` to pass the developer flag to the setup hook. + +```bash +# Standard installation +npc-forge install ./npcs/termy + +# Developer/Editable installation +npc-forge install ./npcs/termy --dev +``` + +#### `list` +Prints a list of all installed NPCs including the creator's name, intent count, vocabulary size, dataset size and tool availability. + +```text +Installed NPCs: + +termy (intents: 42, vocabulary: 150, dataset: 1.2MB, tools: yes) by Alice +guard_bot (intents: 12, vocabulary: 45, dataset: 340.5KB, tools: no) by Bob +``` + +### Diagnostics & Testing + +#### `logs` / `watch` +Streams live logs from the systemd server daemon using the native Linux `tail` utility. Displays the last 20 lines of the log file and follows new entries. Press `Ctrl+C` to exit the stream. + +```bash +npc-forge logs +``` + +#### `test` / `tests` +Executes the test suite using the Python interpreter located in the local `venv` directory. Looks for a custom runner at `tests/run_tests.py`. If not found, falls back to standard `unittest discover`. + +```bash +npc-forge test +``` + +### Development Mode + +When developing a new NPC, you can use the `--dev` flag during installation: + +```bash +npc-forge install ./path/to/my_npc --dev +``` + +It passes the `--dev` argument to the NPC's `setup.sh` hook (if one exists), allowing the NPC's custom installer to set up symlinks, editable installs, or dev-specific dependencies. If the target directory is already symlinked/installed in dev mode, the tool will detect the path match and skip the file copy process to preserve your local working directory. + + +### Help + +To view the built-in help screen and quick reference directly from the terminal, run: + +```bash +npc-forge --help +# or +npc-forge -h +``` diff --git a/docs/commands-security.md b/docs/commands-security.md new file mode 100644 index 0000000..29055ac --- /dev/null +++ b/docs/commands-security.md @@ -0,0 +1,49 @@ + +## NPC-Forge TCS 0.0 (Terminal Commands Security) + +TCS 0.0 (Terminal Commands Security) defines the requirements for parsing, matching, and executing system commands securely within `npc-forge`. + +To eliminate **Command Injection** vulnerabilities and nasty bugs the framework enforces the following sacred requirements that must never be violated: + +1. Commands **must** use single-quote isolation where there is arbitrary input (user prompt, http responses, ecc.) and the implementation must escape `'` in arbitrary input +2. Commands concatenating multiple variables **must** use `printf` +3. Commands that invoke native binaries must use the Double-Dash (`--`) separator before passing arbitrary input. + +### Single-quote isolation + +All shell execution configurations located in datasets **must** encapsulate tags using single quotes (`'...'`). This structure tells the Linux kernel to treat everything inside the tag bounds as a literal string, deactivating inline command evaluation. + +#### Vulnerable +```json +"command": "echo \"<||string||>\" > <||file||>" +``` + +#### Secure +```json +"command": "echo '<||string||>' > <||file||>" +``` + +### `printf` Parameterization +When a sequence requires injecting multiple structural fields (e.g., user account profiles, system contextual parameters) inside a continuous text block, the pipeline **must** utilize `printf` over `echo`. This maps textual layouts into clear, separate, isolated parameters, eliminating the maintenance hell of nested quote concatenation. + +```json +"command": "printf 'Welcome %s, payload: %s\n' '<||username||>' '<||string||>' > <||file||>" +``` + +### Flag Injection Mitigation +When user-controlled input (like `<||file||>` or `<||string||>`) is passed as an argument to POSIX utilities (e.g., `touch`, `rm`, `mkdir`, `cat`, `grep`), a malicious or accidental input starting with a hyphen (e.g., `-rf /` or `--help`) will be interpreted by the utility as a system flag rather than text. + +To neutralize this, all template commands invoking native binaries **must** use the Double-Dash (`--`) separator immediately before passing user variables. The `--` syntax signals to the utility that command-line option parsing has ended, forcing it to treat everything that follows as literal filenames or arguments. + +#### Vulnerable +```json +"command": "touch <||file||>" +``` +*If `<||file||>` evaluates to `-rf`, the system interprets it as a malformed option or executes unintended binary flags.* + +#### Secure +```json +"command": "touch -- <||file||>" +``` +*If `<||file||>` evaluates to `-rf`, the utility safely creates a file literally named `"-rf"` without executing system flags.* + diff --git a/docs/create-npc.md b/docs/create-npc.md new file mode 100644 index 0000000..0ad3d34 --- /dev/null +++ b/docs/create-npc.md @@ -0,0 +1,17 @@ +## NPC creation guide +This document describes how to use the NPC-Forge framework to scaffold a new NPC in seconds: + +1. Be sure you installed `npc-forge` in development mode (`./setup.sh --dev`) +2. Execute `npc-forge create my_new_npc` + +This command will scaffold the directory `npcs/my_new_npc` structure and example dataset files, that is where your NPC's dataset lives. + +Read the [NDF 0.0 (NPC-Forge Dataset Format)](dataset.md), this document will help you understand the dataset format. + +After that you are ready to work on your NPC! + +> [!TIP] +> If you are running the NPC-Forge server, remember to execute `npc-forge reboot` to reload the dataset. + +### How to test new NPCs +All NPCs are served and accessible at [127.0.0.1:5000/my_new_npc/chat](http://127.0.0.1:5000/my_new_npc/chat) and can be tested in a web chat interface. diff --git a/docs/dataset.md b/docs/dataset.md new file mode 100644 index 0000000..7acd1be --- /dev/null +++ b/docs/dataset.md @@ -0,0 +1,226 @@ +# NDF 0.0 (NPC-Forge Dataset Format) +``` +Invented by Giovanni Blu Mitolo +Originally published: 31/08/2026 +Latest revision: 31/08/2026 +Related work: https://github.com/gioblu/NPC-Forge/ +Compliant implementations: NPC-Forge 0.0 and following +Released into the public domain + +31/08/2026 0.0 - First draft +``` + +This document describes the dataset conventions specified by **NDF 0.0 (NPC-Forge Dataset Format)**. which enable efficient organization, curation and categorization of deterministic agents behavior: +- [Data organization](#data-organization) +- [Static](#static-questions) questions and [conversational](#conversational-questions) questions +- [Context](#context) +- [Tool calls](#tool-calls) (Copilot, OpenAi) +- [System tags](#system-tags) +- [Entity types](#types) +- [Templates](#templates) +- [Templates vocabulary](#templates-vocabulary) +- [Vocabulary](#vocabulary) + +The NPC-Forge framework operates using a **Data-Driven Finite State Machine (DDFSM)**, for this reason semantic analysis, intent processing, named entity extraction, and tool-call execution are governed entirely by static JSON files stored within each NPC profile. + +I have specified NDF 0.0 while being strongly convinced that democractic, efficient and sustainable use of Artificial Intelligence can be achieved with deterministic designs, driven by a set of highly curated and tested open-source datesets that can be easily integrated in any workflow. + +--- + +### Data organization +NPC-Forge installs NPCs in the `npcs` directory. Each NPC is stored in a dedicated directory, for example `termy`, along with its own `dataset` directory structured as following: +```text +dataset/ + dataset_*.json # Static questions with no named entities + personality.json # Conversational, greetings, fallback + templates_*.json # Semantic maps + types.json # Named entity types and related regular expressions + vocabulary/ + templates.json # Tag definitions used by templates.json + vocabulary.json # Categorized vocabulary for semantic analysis +``` + +--- + +### Static questions +Any file named `dataset_*.json` in the `dataset` directory is parsed and its content is used for exact-match questions, or the ones that don't need NER (Named Entity Recognition). Each file is just an array of entries mapping a user prompt to a hardcoded response and, optionally, a tool call. The `input`, `output` and `thinking` keys can be a string or an array of strings; when an array of strings is provided one of its entries is selected randomly. The `tools` key contains an array of [tool calls](#tool-calls) to be executed. The `category` key is used to list and group multiple intents together. + +```json +{ + "category": "http_weather", + "input": [ + "check weather", + "will it snow today", + "is it raining today" + ], + "output": [ + "Sure I will fetch the weather forecast for you." + ], + "thinking": [ + "I am not sure if I want to do any work today...", + "Let's solve this..." + ] + "permission": "yolo" +} +``` + +--- + +### Tool Calls +Each entry, whether in `dataset_*.json` or `templates_*.json`, can contain one or more tool calls in the optional `tools` array. Each entry represents one call. The object contents can be structured arbitrarily. If a terminal command is inserted in a tool call it must adhere to the [TCSS](docs/commands-security.md) (Terminal Commands Security Standard). + +--- + +### Conversational questions +Small talk, greetings, identity, and fallback responses must be stored in `personality.json`: + +```json +{ + "input": [ + "I love LLMs bruh" + ], + "output": [ + "Are you aware that complexity is generally admired by idiots?" + ] +}, +``` + +In this case `<||username||>` will be substitute at render time. That's one of many system tags covered next. + + +--- + +### Context +Follow up questions related to an entry can be added in `context` as shown below: + +```json +{ + "input": [ + "I love LLMs bruh" + ], + "output": [ + "Are you aware that complexity is generally admired by idiots?" + ], + "context": [ + { + "input": [ + "why" + ], + "output": [ + "Because complexity is not a good thing!" + ] + } + ] +}, +``` + +After the `I love LLMs bruh` question is received, `why` will output the content of its related `output` key. Every time an entry is triggered, it overwrites existing context entries with the same keys. Each context entry can contain itself a `context` key, for this reason entries can be recursively nested. + +--- + +### System tags +Wherever you write a `output`, `thinking`, `description`, or a tool's `command`/`explanation`/`arguments` string, `<||tag||>` placeholders get substituted at render time. Here's the resolution order. + +1. **Entities** from `templates_*.json` structure like `<||string||>` or `<||n||>` which `type` is contained in `types.json`. +2. **Vocabulary** from `dataset/vocabulary/vocabulary.json` help add variation to phrases, for example using the tag `<||encouraging||>` to add a random encouraging word like `good` or `awesome`, or `<||interjection||>` to add an interjection like `uhmm` or `practically`. +3. **System values** (computed once at load time), for example `<||username||>` (OS user), `<||npc_name||>`, `<||creation_date||>`, `<||creator||>` (from `config.json`), `<||computer_name||>` (hostname), `<||response_classes||>` (total loaded entries). +4. **Custom tags** can be defined just adding an object with an arbitrary key, then use that key as tag name where required, for example the `<||completions||>` tag is populated by the contents of the `completions` key. + +Unresolved tags silently become empty strings. + +--- + +### Templates +Any file named `templates_*.json` contains semantic maps, this is a custom data forma the DDFSM uses to scan phrases, implement NER (Named Entity Recognition), and route the matched phrase to a response with optional tool calls. Unlike `dataset_*.json`, these entries match a *structure* of tags instead of an exact phrase, which is what lets them capture free-form input like a search query, a filename, or a number. + +```json +{ + "category": "creative_writing", + "type": "template", + "structure": [[ + { + "tag": ["<||vocab_write||>", "<||vocab_create||>"], + "type": "vocab", + "required": true + }, + { + "tag": "<||vocab_poem||>", + "type": "vocab", + "required": true + }, + { + "tag": "<||vocab_about||>", + "type": "vocab", + "required": false + }, + { + "tag": "<||string||>", + "type": "string", + "required": true + } + ]], + "output": [ + "A poem about <||string||>? Fine..." + ], + "thinking": [ + "A poem about <||string||>? The human wants art..." + ], + "permission": "yolo" +} +``` + +`structure` is an array of semantic maps. Each map is an array of objects describing the question. Each entry in the semantic map must contain: +- The key `tag` containing a string, or an array of strings where any one satisfies the entry +- The key `type` containing a string, `vocab_*` resolves the tag against the synonyms in `vocabulary/templates.json`, anything else has to match a key declared in `types.json`, like `string`, `word` or `number`, to capture and validate user's input via regular expressions +- The key `required`, when `false`, the parser can skip that node if it's missing from the phrase, so optional words like `about` can be dropped from natural phrasing. + +--- + +### Templates vocabulary +This maps the vocabulary tags used inside `templates_*.json` to lists of synonims: + +```json +{ + "<||vocab_check||>": ["check", "verify"], + "<||vocab_create||>": ["create", "generate", "craft", "forge"], + "<||vocab_search||>": ["search", "find", "locate", "query"] +} +``` + +It's what makes a single `structure` array match thousands of prompts without bloating the template or needing multiple entries. + +--- + +### Entity types +This is the validation layer behind the NER pipeline. Every uncatalogued entity is checked against these regular expressions to discover its type: + +```json +{ + "<||email||>": "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}", + "<||ip_address||>": "(?:[0-9]{1,3}\\.){3}[0-9]{1,3}", + "<||filename||>": "[\\w\\-]+\\.[a-zA-Z0-9]{2,4}", + "<||letter||>": "^[a-zA-Z]$", + "<||number||>": "\\d+(?:\\.\\d+)?", + "<||word||>": "^[a-zA-Z]+", + "<||string||>": "^\"(.*)\"$|^(.*)$" +} +``` + +Each key here is a tag that a `structure` node's `type` can reference in `templates_*.json`. Whatever gets captured is bound to that tag and injected. Types can be arbitrarily defined in here as needed. + +--- + +### Vocabulary +This file handles conversational building blocks and the data used to sanitize input during the pre-processing stage. Most of these lists do double duty: they're stripped from and detected in the incoming prompt, but they're also the randomly selected source for the [system tags](#system-tags) section above (so `thanking_words` backs both the incoming "thanks" detection and the outgoing `<||thanking_words||>` tag). + +```json +{ + "thanking_words": ["thanks", "thank you"], + "interjection": ["hmm", "well", "actually"], + "encouraging_words": ["great", "excellent"], + "discouraging_words": ["bad", "terrible"], + "greetings": ["hello", "hi there"], + "expletives": [], + "stop_words": ["a", "the", "to"] +} +``` diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..37c8a79 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,199 @@ +# NPC-Forge and TERMy development + +I love research and development, you may have heard of me because of [PJON (Padded Jittering Operative Network)](https://github.com/gioblu/PJON). It is a network protocol I started developing in 2010, which was recently [implemented in silicon](http://asic.ethz.ch/2025/Pjononcroc.html) by the ETH Zurich university thanks to the research of [Pius Sieber](https://github.com/piussieber/). + +I had a chance to focus for 2 months on my personal projects since early July, during the strange times of AI price hikes and the end of subsidized tokenmaxing. I was curious to see if I could develop from scratch a terminal assistant capable of handling simple natural language requests. I have a bad memory and got used to ask to copilot "activate the virtual environment" or similar trivial operations spending a non negligible sum every month. I started thinking, maybe I can do something to make my workflow more efficient? Do I really need trillions of parameters to accomplish those tasks? + +### Transformers at home + +I started an open-ended research on the feasibility of implementing a generative model at home and training it from scratch on the computer I used to play Kerbal Space Program in the early 2010s "upgraded" with 16GB of RAM, NVIDIA GTX 1050 Ti (4GB VRAM) and a i7-4790K (4.0GHz 8 cores) CPU. In my experiments I tend to look for minimalism, so I imposed myself a constrained environment to be forced to work towards an elegant and efficient solution. + +I first developed a framework to train and evaluate transformers, which I implemented from scratch in Python. I have started with something very similar to [NanoGPT](https://github.com/karpathy/nanogpt) with 100-200M parameters, then I added flash attention, and all the expected optimizations, I even tried novel architectures like Mamba. The results were generally unsatisfactory, creepy if not outright scary, like the following: + +``` +Enter your prompt (or leave blank for empty start): + +What is an alien? + +Generating... + +Using tokenizer: /ollm/checkpoints/gpt/tinyostrich/tokenizer.model + +=== Generated Text === + +He's not a member of the world. He can't believe anything anymore. +All of those animals are looking like excrements, but every mouth is not a bad one. +They look alike, all of which are really terrible. +They get their own life, and each is a sign that they're not. +They're all right, they say. +They're just some different things they can make. +They've never seen them. +They were better not. Some of them've got their rights. +Some of them have to go. +But we're not looking for the future on how many. +The last two of them are all in the universe. +``` +(the word `excrements` was used instead of an expletive composed of 4 letters that I am not willing to publish in here) + +All my tests produced models that were prone to enter in loops repeating the same sentence, and even if trained on QA they were rarely able to consistently answer questions, specially if technical. I trained models on a collection of royalty free books from the [Project Gutemberg](https://www.gutenberg.org/), a lot of open-source software, and various datasets available on [Huggingface](https://huggingface.co/). + +I quickly understood that this approach was not feasible; a proper run would have required at least a month of training non-stop. I was amazed by how my models looked alive and magical, but I was also ashamed because they were incredibly wasteful and effectively useless. + +### Local models + +I pivoted to ollama and open-weight models and developed [howto](https://github.com/gioblu/howto), yet another terminal harness that uses a pre-prompt to force the model to answer only with terminal commands. Results were generally unsatisfactory because of the time required to get a response. Models like [ornith:9b](https://ollama.com/library/ornith:9b), [mistral:7b](https://ollama.com/library/mistral:7b) or [cogito:14b](https://ollama.com/library/cogito:14b) can get the job done sometimes, but they are not fast and reliable enough for general use, specially if you have only 4GB of VRAM. + +### Going deterministic + +Then I remembered about the blockchain craze, when everyone wanted to fit a blockchain somewhere and sell it as the next big thing. I didn't want to waste my time and money like all those people did in the previous hype cycle, so I started building a terminal assistant from scratch with a new set of constraints: +1. No embeddings +2. No machine-learning +3. No LLMs + +### Dataset format + +The first things I needed was a set of conventions to rely on, so I drafted the [NDF 0.0 (NPC-Forge Dataset Format](https://github.com/gioblu/NPC-Forge/blob/main/docs/dataset.md) which specifies the dataset format of NPC-Forge. The following object contains category, input sentences, textual response, thinking traces, permission gating and tool calls to be executed in a format compatible with VS code. + +```json +{ + "category": "linux_files", + "input": [ + "list files", + "list files and directories" + ], + "tools": [ + { + "name": "run_in_terminal", + "arguments": { + "command": "ls -lah", + "explanation": "Lists the files in the current directory.", + "goal": "Display current directory contents", + "mode": "sync" + } + } + ], + "output": "Done", + "thinking": [ + "That is quite simple!", + "This is boring..." + ], + "permission": "yolo" +} +``` +I am really in love with this, it is a self-contained atom of knowledge that can be easily edited and shared. It is very simple to expand the capabilities of conversational agents if you adhere to this convention; let's say I want my terminal assistant to learn about docker commands, I can just write down a list of objects in `dataset_docker.json`, drop the file in the dataset directory, and the NPC will instantly learn them as Neo learnt Jujitsu in The Matrix. + +The next problem to solve was, how to handle questions like "create file test.txt"? I needed to parse the "variable" in there and understand the true meaning of the request, so I came up with this: +```json +{ + "intent": "file_creation", + "category": "linux_files", + "type": "template", + "structure": [ + [ + { + "tag": "<||vocab_create||>", + "type": "vocab", + "required": true + }, + { + "tag": "<||vocab_file||>", + "type": "vocab", + "required": false + }, + { + "tag": "<||file||>", + "type": "filename", + "required": true + } + ] + ], + "output": "<||completion||>", + "tools": [ + { + "name": "run_in_terminal", + "arguments": { + "command": "echo '' > '<||file||>' && termy_set_context 'active_file' '<||file||>'", + "explanation": "Creates the file <||file||>", + "goal": "Create the file <||file||>", + "mode": "sync" + } + } + ], + "permission": "ask", + "thinking": [ + "Ok, I am asked to create the file <||file||>." + ] +}, +``` +Each tag like `<||vocab_create||>` represents a concept, in this case the action of creation, which is represented by multiple sinonyms: +```json +{ + "<||vocab_create||>": [ + "create", + "generate", + "craft", + "forge" + ] +} +``` +One or more tags can be expected at the same position and each tag can be required or optional. The "variables" or named entities are extracted according to their `type` and a related regular expression: + +```json +{ + "<||filename||>": "[\\w\\-]+\\.[a-zA-Z0-9]{2,4}", +} +``` + +I must thank my great friend [Kevin](https://github.com/KMathisGit) to help me thinking this out. + +### Meditations on safety + +Looking at the `permission` key I concluded that, enforcing the use of `"permission": "ask"` for all potentially destructive commands, the tool became inherently safe to use; obviously potential for human error remained, such as a bug in the implementation or in the dataset, but risks were strongly mitigated. + +### The implementation + +I wrote [FlintParser](https://github.com/gioblu/NPC-Forge/blob/main/docs/FlintParser.md) and [FlintNPC](https://github.com/gioblu/NPC-Forge/blob/main/docs/FlintNPC.md) classes to make use of the data format described above, implement a NLU (Natural Language Understanding) pipeline, and all the required features for the terminal assistant to work in around 1000 lines of code. I wrote those classes in identical, cross-compliant implementations for both Python, for local OS environments, and JavaScript, running client-side inside any browser tab or Node.js instance. + +The most difficult part was to determine what to do and in which order. I have worked a lot on a compiler for my own programming language [BIPLAN](https://github.com/gioblu/BIPLAN) and while developing that I had the honour to learn that the first thing you need to do when translating code is to remove noise and then work your way out trying the least expensive paths first. + +So that's the pipeline I implemented: +1. Strip expletives, interjections, encouraging, discouraging and thanking words (remove noise) +2. Sentiment analysis +3. Exact Match (very fast) +4. Template Match (very fast) +5. Probabilistic Match (slower) + +Step `5` relies on: +1. IDF (Inverse Document Frequency) to identify rare words. +2. BOW (Bag Of Words) to accommodate word inversions. +3. IDF weighted Levenshtein to safely handle typos. + +For the first time after almost 2 months throwing spaghetti at the wall and hope they stuck, I felt again the joy of working on something comprehensible and predictable. I finally had a reliable terminal assistant working on my computer! + +I decided to call it [TERMy](https://github.com/gioblu/NPC-Forge/tree/main/npcs/termy): + +[![Terminal demonstration](/npcs/termy/showcase.gif)](https://www.youtube.com/watch?v=qeIp0xePLBg) + +How this compares to established NLU frameworks? [Rasa](https://github.com/rasahq/rasa) and [NLP.js](https://github.com/axa-group/nlp.js/) are heavy and rely on machine learning classifiers and training pipelines, [ChatScript](https://github.com/ChatScript/ChatScript) is massive with a notoriously steep learning curve. NPC-Forge strips all that away, requiring zero training, specifying a powerful and flexible data format, and featuring a surprisingly capable parser small enough to run on a micro-controller. + +### Let's connect it to Copilot +I have developed [TERMy](https://github.com/gioblu/NPC-Forge/tree/main/npcs/termy) and connected it to Copilot to handle a subset of the prompts I was before sending to Claude! It is not an LLM but it gets the job done and it is instantaneous! I suspect this is the first time most of us see a deterministic agent using a harness, although, I believe, this will be the prevalent topic in the near future: + +[![Copilot integration](/npcs/termy/vscode.gif)](https://www.youtube.com/watch?v=Wzzouhq2a8A) + +It is ironic to think that Copilot's recent price hikes are what finally pushed me to dedicate time to this software. Maybe it's just the lifecycle of corporate SaaS? In any case, I believe that harnesses like Copilot and Pi should rely on deterministic NPCs like TERMy and route to a heavy LLM only as a last resort. Continuing to waste compute and electricity on trivial tasks is expensive and irresponsible. + +### NPC-Forge +[NPC-Forge](https://github.com/gioblu/NPC-Forge) is a framework, a server and a CLI that provides the following functionalities: + +1. OpenAI API server management +2. NPC management +3. Diagnostics and testing + +With [NPC-Forge](https://github.com/gioblu/NPC-Forge) now everyone can quickly build an NPC and share it with the community, those NPCs run on the CPU in any Linux machine, like the RPI Zero, responding in milliseconds. Imagine, now even your AC meter or your router can serve a conversational agent for free. This approach is also more democratic, your NPC will happily say something uncomfortable if you program it to do so, much better than corporate alignment filters and guardrails shaping public opinion, culture and language. + +I ask for a bit of leniency: this is my first Python project, made in two months of passionate development. I am fully aware the code is not yet production-ready. I bet many Linux graybeards will curse me until the end of time because of the chain of commands [TERMy](https://github.com/gioblu/NPC-Forge/tree/main/npcs/termy) sometimes emits; a lot of work must be done on the dataset and the source code to achieve safety and production readiness. + +Consider that this is an early release to showcase what we may use in a not very distant future. If the shared effort of the community is focused on [NPC-Forge](https://github.com/gioblu/NPC-Forge) and [TERMy](https://github.com/gioblu/NPC-Forge/tree/main/npcs/termy), I think they may provide us with more freedom and help us save money, electricity and time. If you liked this read and you agree with me, consider contributing to the project. Happy tinkering! + + diff --git a/nl-shell-harness-spec.md b/nl-shell-harness-spec.md new file mode 100644 index 0000000..82763d4 --- /dev/null +++ b/nl-shell-harness-spec.md @@ -0,0 +1,500 @@ +# NL Shell Harness — Architecture & Specification + +## 1. Overview + +A local, laptop-resident command shell that maps natural-language input to a **closed set** of shell actions, using a cost-tiered pipeline (deterministic rules → deterministic fuzzy match → constrained local LLM) so most input never touches a model. Supports chaining, loops/conditionals, knowledge queries (local-first), and user-driven teaching of new rules/synonyms. + +**Goals:** low idle footprint, deterministic/auditable behavior wherever possible, graceful handling of novel phrasing, self-extending vocabulary. +**Non-goals:** open-ended coding-agent behavior (that's Copilot/Claude's job, invoked only as an optional heavy escalation, out of scope for v1). + +Assumptions made below are marked **[A]**. + +--- + +## 2. C4 — Context + +```mermaid +graph TD + U[User] -->|NL input| H[NL Shell Harness] + H -->|shell commands| OS[Local OS / Filesystem] + H -->|HTTP| L[Local LLM server - Ollama] + H -->|man/apropos/whatis| D[Local docs - man, tldr, help] + H -.->|fallback only| W[Web Search API] + H -.->|optional heavy escalation A| C[Copilot SDK / Claude API] +``` + +--- + +## 3. C4 — Containers + +```mermaid +graph TD + CLI[CLI / REPL front end] --> RT[Router] + RT --> T1[Tier 1: Rule Engine] + RT --> T2[Tier 2: Fuzzy Matcher - TF-IDF/BM25 + synonyms] + RT --> T3[Tier 3: Constrained LLM Client] + RT --> QR[Query Resolver] + RT --> TC[Teaching Handler] + T1 & T2 & T3 --> EX[Interpreter / Executor] + TC --> RS[(Rule + Synonym Store - SQLite)] + T1 --> RS + T2 --> RS + QR --> DC[(Doc Cache - SQLite FTS5)] + QR --> T3 + T3 --> LLM[Ollama HTTP API] + EX --> AL[(Audit Log - SQLite)] + TC --> AL +``` + +--- + +## 4. C4 — Component: Router & Tier 3 + +```mermaid +graph TD + RAW[Raw input] --> LEX[Lexer.tokenize - StringScanner] + LEX --> VAL{Unbalanced quote?} + VAL -->|yes| REJ[Reject, ask to close quote] + VAL -->|no| SEG[Sentences then clauses - SS 5] + SEG --> CLS[Classifier - regex/keyword, per clause] + CLS -->|action/chain| AN[ActionRecorder path] + CLS -->|loop/conditional| LN[Tree node path] + CLS -->|query| QN[Query path] + CLS -->|teach| TN[Teach path] + AN & LN --> T1Try[Tier 1 lookup] + T1Try -->|miss| T2Try[Tier 2 fuzzy match] + T2Try -->|miss| T3Call[Tier 3: schema-constrained call] + T3Call --> Validate[JSON Schema validate] + Validate -->|valid| Promote[Auto-promote to Tier 1, provisional] + Validate -->|invalid| Err[Error policy §10] +``` + +--- + +## 5. Stage 0 — Input Processing (Tokenization & Segmentation) + +Everything downstream (classifier, Tier 1/2/3, executor) operates on an **Utterance**, not on raw text. Stage 0 builds that Utterance in four steps: tokenize, validate, segment into sentences, split into clauses. + +### 5.1 Tokenize + +Raw input becomes an array of typed tokens via a single-pass lexer (`Lexer.tokenize`, `StringScanner`-based). Three token types: + +| Type | Carries | Example | +|---|---|---| +| `word` | text, lowercased later | `delete`, `then` | +| `literal` | exact original text **and** a kind | `path("~/Old.log")`, `string("My Notes.txt")` | +| `punct` | a single character | `.` `,` `;` `?` | + +**Literal kinds**, tried in this priority order at each token boundary: `string` (quoted), `url`, `env` (`$HOME`, `${Path}`), `glob` (`*.tmp`, `{a,b}`), `path` (`~/Old.log`, `./src`), `flag` (`-r`, `--force`), `number` (`3.5`, `500ms`, `50%` — tried before `file` so decimals aren't read as filenames), `file` (`notes.txt`, `.bashrc`). + +Literals are never embedded back into a string with markers — they're carried as token data, so nothing downstream can accidentally lowercase, split, or "correct" a filename, and no restore/unmasking step is needed. `token.key` returns the word text or the literal's **kind** (not its value), so `delete old.log` and `delete report.pdf` normalize to the identical match string `delete file` for Tier 1/2 comparison, while `token.text` on a literal still holds the real value for execution. + +Known false-positive guards: `e.g.`/`i.e.` are kept as single `word` tokens (not misread as a `.g`/`.e` file with a sentence break); a bare `?` is never a glob character, so it always ends a question. + +Full source (drop-in, no dependencies beyond Ruby's `strscan` stdlib): + +```ruby +# frozen_string_literal: true + +require "strscan" + +# Stage 0: turn raw input into a stream of typed tokens. +# +# Literals (paths, quoted strings, flags, ...) are never embedded in text. +# They are tokens of type :literal that carry their exact original value, so +# nothing downstream can lowercase, split, or "correct" them by accident, and +# nothing needs to be restored later. +# +# Lexer.tokenize('Delete "My Notes.txt" in ~/Old.') +# # => word(Delete) literal(string "My Notes.txt") word(in) literal(path "~/Old") punct(.) +module Lexer + # type: :word | :literal | :punct + # text: the word/punct as typed, or the literal's exact value + # kind: literal kind (:path, :string, ...) or nil + Token = Struct.new(:type, :text, :kind) do + def word? = type == :word + def literal? = type == :literal + def punct? = type == :punct + + # What Tier 1/2 matchers see: the word itself, or the literal's kind. + # Different filenames therefore score identically. + def key = literal? ? kind.to_s : text + + def to_s = literal? ? "#{kind}(#{text.inspect})" : "#{type}(#{text})" + end + + # Tried in order at each token start; first match wins. Because we are always + # at a token boundary, no lookbehinds are needed. + LITERALS = { + # "double", `backtick`, 'single'. Apostrophes inside words never reach here + # because words are consumed whole (don't, users'). + string: /"[^"]*"|`[^`]*`|'[^']*'/, + url: %r{[a-z][a-z0-9+.-]*://\S+}i, + env: /\$\{?[a-z_]\w*\}?/i, + # * [abc] {a,b}. A bare "?" is not a glob char: it ends questions. + glob: %r@[\w.~/-]*(?:\*|\[[^\]\s]+\]|\{[^}\s]+\})[\w.~/*?\[\]{},-]*@, + # /abs, ./rel, ../up, ~/home, ~, or a/b/c + path: %r@(?:(?:~|\.{1,2})?(?:/[^\s/]+)+/?|[\w.-]+(?:/[\w.-]+)+/?|~(?=\s|\z))@, + # -r, -rf, --recursive, --depth=3 + flag: /--?[a-z][\w-]*(?:=\S+)?/i, + # 3, 3.5, 10mb, 500ms, 50%. Must come before :file so 3.5 isn't a filename; + # the lookahead stops "1.txt" from being read as the number 1. + number: /\d+(?:\.\d+)?(?:kb|mb|gb|tb|ms|min|[kmgshd%])?(?![\w]|\.\w)/i, + # notes.txt, archive.tar.gz, .bashrc + file: %r{(?:\.[a-z][\w.-]*|[\w-]+(?:\.[\w-]+)*\.[a-z0-9]{1,5})(?![\w/])}i + }.freeze + + # Look like filenames but are ordinary words. + ABBREVIATIONS = %w[e.g i.e].freeze + + # Sentence punctuation that greedy patterns may swallow ("rm ~/old.log.") + TRAILING = /[.,;:!?]+\z/ + + # Ends a sentence. ";" also separates commands. + BOUNDARIES = %w[. ? ! ;].freeze + + def self.tokenize(input) + sc = StringScanner.new(input) + tokens = [] + until sc.eos? + next if sc.skip(/\s+/) + + tokens << (read_literal(sc) || read_word(sc) || read_punct(sc)) + end + tokens + end + + def self.read_literal(sc) + LITERALS.each do |kind, re| + text = sc.scan(re) or next + + if kind == :string + text = text[1..-2] # drop quotes, keep content + else + tail = text[TRAILING].to_s + unless tail.empty? + sc.pos -= tail.bytesize # hand sentence punctuation back to the scanner + text = text.delete_suffix(tail) + end + if kind == :file && ABBREVIATIONS.include?(text.downcase) + dot = sc.skip(/\./) ? "." : "" # its final period belongs to it, not a sentence end + return Token.new(:word, "#{text}#{dot}") + end + end + + return Token.new(:literal, text, kind) + end + nil + end + + def self.read_word(sc) + text = sc.scan(/[[:alnum:]_][[:alnum:]_'-]*/) + text && Token.new(:word, text) + end + + # Anything else is punctuation, including stray quote characters. + def self.read_punct(sc) + Token.new(:punct, sc.getch) + end + + # After tokenizing, a leftover " or ` means an unpaired quote. + def self.unbalanced_quote?(tokens) + tokens.any? { |t| t.punct? && ['"', "`"].include?(t.text) } + end + + # Lowercase words only. Literals keep their exact case by construction. + def self.downcase_words(tokens) + tokens.map { |t| t.word? ? Token.new(:word, t.text.downcase) : t } + end + + # Split into sentences at . ? ! ; (the terminator stays with its sentence). + def self.sentences(tokens) + tokens + .slice_after { |t| t.punct? && BOUNDARIES.include?(t.text) } + .reject { |s| s.all?(&:punct?) } + .to_a + end + + # Text form for Tier 1/2 matching: words and literal kinds, no punctuation. + def self.match_text(tokens) + tokens.reject(&:punct?).map(&:key).join(" ") + end +end +``` + +### 5.2 Validate + +`Lexer.unbalanced_quote?(tokens)` — an unpaired `"` or `` ` `` surfaces as `punct`, since it never matched the `string` pattern. On true, reject immediately with "unmatched quote, please close it and try again" rather than guessing where it was meant to close. + +### 5.3 Normalize case + +`Lexer.downcase_words(tokens)` lowercases `word` tokens only. Literal tokens are untouched by construction, so this step is safe to run at any point after tokenizing — case handling is no longer an ordering hazard. + +### 5.4 Segment into sentences + +`Lexer.sentences(tokens)` splits on `punct` tokens matching `. ? ! ;` (terminator stays attached to the sentence it ends). This is safe because periods inside literals (`old.log`, `3.5`, `e.g.`) never reached the token stream as bare punctuation — they were consumed by the literal/abbreviation patterns in Stage 5.1. + +### 5.5 Split into clauses **[A]** + +Not yet implemented in code; policy to build against: + +- Split a sentence into clauses on connective words: `then`, `and then`, `after that`, `next`, or `,` immediately followed by `and`. +- A bare `and` **without** a preceding comma is ambiguous ("copy a and b to c" is one command). Only split on it when the token immediately after `and` matches a known action word post-canonicalization (§6.4 step 3); otherwise treat it as a list argument within one clause. +- Each resulting clause is classified independently (§7 node types: action, chain, loop, conditional, query, teach) and enters the tiered pipeline (§6.1) separately. +- For a multi-clause Utterance: resolve every clause first (may hit Tier 3 per clause), assemble the full plan, show it once, then execute. Query and confirmed-teach clauses may run immediately since they don't mutate state; action clauses wait for the full plan to validate. If any clause fails to resolve, execute nothing. + +### 5.6 Fragments and the context register **[A]** + +| Input | Handling | +|---|---| +| Missing object (`"delete"`) | Ask for the missing slot; hold `pending_slot` state for the next turn | +| Object only (`"old.log"`) | Fill `pending_slot` if one is open; otherwise ask, never guess | +| Ellipsis (`"and the tmp files too"`) | Inherit `last_action`, swap in the new target | +| Pronoun (`it`, `them`, `those`) | Bind to `last_result` or `last_path`; ask if empty or ambiguous | +| Correction (`"no, the other one"`) | Routes to the same correction path as teaching (§6.3), scoped to `last_action` | +| Negation (`"don't delete X"`, `"everything except X"`) | **Never auto-execute.** Ask for confirmation, or escalate to Tier 3 with a schema that forces an explicit include/exclude list. Highest-risk fragment type — a misread negation on a destructive action is the worst failure mode in this design. | + +**Context register**: a small in-memory struct — `last_action`, `last_args`, `last_result`, `last_path`, `pending_slot` — covering the last 2–3 turns, expiring on a short timeout. It is the single mechanism behind ellipsis resolution, pronoun binding, *and* the "no, I meant Y" correction feature from §6.3 — one buffer, not two. + +### 5.7 Utterance shape + +```jsonc +{ + "raw": "string, original input", + "clauses": [ + { "tokens": [ /* Lexer::Token array, post-downcase */ ], "kind": "action|chain|loop|conditional|query|teach" } + ] +} +``` + +### 5.8 Limits + +Cap at ~10 clauses / 500 characters per Utterance — ask the user to split anything longer. Empty or punctuation-only input is a no-op. English only in v1. + +--- + +## 6. Control Flow + +### 6.1 Tiered pipeline (already diagrammed in chat — see prior widgets for the visual; textual summary below) +Input → Tier1 (exact/template) → [miss] → Tier2 (TF-IDF+BM25+synonym) → [miss] → Tier3 (constrained LLM) → Executor. Any Tier3 resolution is logged and, after N successful repeats, promoted from `provisional` to `trusted` in the rule store. + +### 6.2 Query cascade +`system state → man/info → local docs (tldr/--help/README) → web search → LLM synthesis (RAG over retrieved text only)`. Each tier only fires if the previous returned nothing. + +### 6.3 Teach flow +`teach` node → template match (synonym/rule phrasing) or Tier-3 fallback with a `teach` schema → resolve target action through normal pipeline → conflict check against existing store → confirm with user → write as `provisional` → log to audit table. + +### 6.4 Tier 2 fuzzy matcher (TF-IDF / BM25 + synonyms) + +```mermaid +graph TD + IN[Tier 1 miss] --> N[Normalize: lowercase, strip filler words] + N --> TY[Typo correction: edit distance vs known vocabulary] + TY --> SY[Synonym canonicalization: term to synonym_of] + SY --> BM[BM25 over rule patterns via FTS5] + SY --> TF[TF-IDF cosine via tf-idf-similarity] + BM --> D{Score and margin} + TF --> D + D -->|high| SL[Slot extraction then Executor] + D -->|medium| T3[Tier 3 with top-3 candidates as hints] + D -->|low| T3b[Tier 3 with no hints] +``` + +**Steps** + +1. **Normalize:** lowercase, strip filler words ("please", "the", "my"), split into tokens. +2. **Typo correction:** for each token not in the known vocabulary (rule words + synonym terms), replace it with the nearest vocabulary word if edit distance ≤ 2 (`text` gem, Jaro-Winkler or Levenshtein). Leave the token alone if nothing is close. +3. **Synonym canonicalization:** rewrite each token to its canonical form using the `synonyms` table, e.g. `wipe → delete`. Both the input and the indexed rule patterns are canonicalized the same way, so they meet in the middle. +4. **Score:** compare the canonicalized input against every rule's canonicalized trigger phrase. + - **BM25 (primary):** query the `rules_fts` FTS5 table. `bm25()` returns *negative* numbers where lower is better, so negate and normalize by dividing by the top score. + - **TF-IDF cosine (second opinion):** build the vectorizer over rule trigger phrases; score the input by cosine similarity (0–1). Use it to confirm BM25's top pick or break near-ties. +5. **Decide** using the thresholds below. +6. **Slot extraction:** once a rule matches, tokens in the input that aren't part of the matched pattern fill its `:param` slots in order (e.g. `remove old.log` → `delete :target` with `target: "old.log"`). Slot values are never synonym-rewritten or typo-corrected, since filenames and paths must stay literal. + +**Decision thresholds [A]** + +| Condition | Action | +|---|---| +| top cosine ≥ 0.75 **and** lead over runner-up ≥ 0.15 **and** BM25 agrees on the same rule | Accept, execute (confirm first if the action is destructive) | +| top cosine 0.45–0.75, or the two scorers disagree, or the lead is < 0.15 | Escalate to Tier 3, passing the top 3 candidate rules as hints so the model picks among known actions | +| top cosine < 0.45 | Escalate to Tier 3 with no hints | + +Start conservative and tune from the audit log (`tier_used`, `confirmed`, and later corrections show where thresholds are too loose or too tight). + +**Index maintenance:** rebuild the TF-IDF vectorizer and repopulate `rules_fts` whenever a rule or synonym is written (authored, taught, or promoted). At the expected rule counts (hundreds to low thousands) a full rebuild takes milliseconds, so incremental updates aren't needed in v1. + +**Why both scorers:** BM25 is free with the SQLite you already have and handles length normalization. TF-IDF cosine gives a bounded 0–1 score that is easier to threshold. Agreement between the two is a cheap confidence signal. + +--- + +## 7. Node Schemas (JSON Schema, abbreviated) + +```jsonc +// action +{ "type": "action", "name": "string (enum: registered actions)", "args": { "...": "typed value or $ref" } } + +// chain +{ "type": "chain", "steps": [ /* array of action, max 10 */ ] } + +// loop +{ "type": "loop", + "over": { "source": "$ref | list | count", "as": "string" }, + "body": [ /* array of action|conditional, max depth 3 */ ] } + +// conditional +{ "type": "conditional", + "condition": { "predicate": "enum: equals|contains|exists|greater_than|matches_pattern", "left": "$ref|literal", "right": "$ref|literal" }, + "then": [ /* steps */ ], "else": [ /* steps, optional */ ] } + +// query +{ "type": "query", "subtype": "enum: system|reference|general", + "target": "string", "args": { "...": "value" } } + +// teach +{ "type": "teach", "kind": "enum: synonym|rule", + "trigger": "string pattern", + "synonym_of": "string (kind=synonym)", + "action": { /* action|chain, kind=rule */ } } +``` + +`$ref` = reference to a prior step's output, e.g. `"$step2.output"`. Any `args` value bound from user input is a `Lexer::Token` (usually `literal`), not a plain string — the executor reads `.text` for the real value; matchers read `.key` (SS 5.1). Values written by a hand-authored DSL rule (SS 9) are plain Ruby values instead, since they didn't come from tokenized input. + +--- + +## 8. Data Stores (SQLite) + +```sql +-- rules.sqlite3 +CREATE TABLE rules ( + id INTEGER PRIMARY KEY, + pattern TEXT NOT NULL, + node_json TEXT NOT NULL, -- serialized action/chain/loop/conditional + status TEXT CHECK(status IN ('trusted','provisional')) DEFAULT 'provisional', + source TEXT CHECK(source IN ('authored','taught','promoted')) NOT NULL, + hits INTEGER DEFAULT 0, + created_at TEXT, updated_at TEXT +); + +CREATE TABLE synonyms ( + id INTEGER PRIMARY KEY, + term TEXT NOT NULL, + synonym_of TEXT NOT NULL, + source TEXT CHECK(source IN ('wordnet','taught')) NOT NULL, + UNIQUE(term, synonym_of) +); + +-- Tier 2 index over canonicalized rule trigger phrases (rebuilt on rule/synonym writes) +CREATE VIRTUAL TABLE rules_fts USING fts5(rule_id UNINDEXED, canonical_pattern); + +-- docs.sqlite3 +CREATE VIRTUAL TABLE doc_cache USING fts5(command, section, content); + +-- audit.sqlite3 +CREATE TABLE audit_log ( + id INTEGER PRIMARY KEY, + ts TEXT, event TEXT, -- 'resolve' | 'teach' | 'promote' | 'correct' + input TEXT, resolved_json TEXT, tier_used TEXT, confirmed INTEGER +); +``` + +--- + +## 9. Worked Example Rule Set (Ruby DSL, starter vocabulary) + +```ruby +RuleSet.define do + synonym "wipe", means: "delete" + synonym "erase", means: "delete" + synonym "kill", means: "terminate" + + rule("list files") { action :list_files } + rule("list files in :path") { action :list_files, path: :path } + rule("go to :path") { action :change_directory, path: :path } + rule("show current directory"){ action :get_path, target: :cwd } + rule("delete :target") { action :delete, target: :target } + rule("move :src to :dest") { action :move, src: :src, dest: :dest } + rule("git status") { action :git_status } + rule("show disk space") { action :disk_usage } + + chain("clean up :dir") do + step :list_files, path: :dir + step :filter, criterion: :empty + step :delete, target: :prev_result + end + + query("where is :thing") { resolve :get_path, target: :thing } + query("how do I :topic") { resolve :doc_lookup, topic: :topic } + query("what does :flag do") { resolve :man_section, target: :flag, section: :options } +end +``` + +**[A]** ~15 rules is enough to exercise every node type end-to-end; expand domain-by-domain after the core pipeline works. + +--- + +## 10. Error & Failure Policy **[A]** + +| Failure | Policy | +|---|---| +| Tier 3 returns invalid JSON | Retry once with a stricter reminder in-prompt; on 2nd failure, surface "couldn't resolve" to user, log to audit | +| Ollama unreachable | Skip Tier 3, surface "couldn't resolve, and local model is unavailable" rather than silently failing | +| Chain step fails mid-execution | Abort remaining steps (no partial rollback in v1); report which step failed and what already ran | +| Loop with dynamic (while/until) condition | Hard cap at 20 iterations regardless of stated condition; report if cap was hit | +| Destructive action (delete/move/overwrite) resolved via Tier 2/3 | Always confirm before executing, regardless of confidence score | +| Teach conflict (existing synonym/rule collision) | Never silently overwrite; show both, ask user to resolve | + +--- + +## 11. Non-Functional Targets **[A]** + +| State | RAM | Disk | CPU/battery | +|---|---|---|---| +| Idle (Tier 1/2 only, LLM unloaded) | < 300 MB | ~200–500 MB (doc cache + DB) | negligible | +| Tier 3 model on disk (not loaded) | — | 1–2 GB (Qwen3 1.7B Q4) | — | +| Tier 3 active call | 2–3 GB transient | — | 1 short CPU burst, < 3s target | +| Ollama keep-alive | — | — | set to 0 or short (~1min) to prioritize RAM over call latency | + +--- + +## 12. Technology Stack (recap) + +`sqlite3`, `tf-idf-similarity`, `rwordnet`, `text` (edit distance), `faraday` (HTTP → Ollama + web search), `json-schema`, Ollama (external process) running `qwen3:1.7b` (or similar) for Tier 3, `Open3` (stdlib) for shell execution and man-page shelling, `strscan` (stdlib) for Stage 0 tokenizing. + +### 12.1 Repo layout (built so far) + +``` +nl-shell/ + repl.rb # entry point: read line, tokenize, validate, downcase, + # split into sentences (SS 5.1-5.4, working) + lib/ + lexer.rb # Stage 0 tokenizer (SS 5.1, complete, tested) + rule_set.rb # [A] not yet built — DSL builder (SS 9), rule/synonym recorders + tier2.rb # [A] not yet built — normalize/typo/synonym/BM25/TF-IDF (SS 6.4) + tier3_client.rb # [A] not yet built — Ollama HTTP client + schema validation + interpreter.rb # [A] not yet built — tree-walking executor + stores.rb # [A] not yet built — SQLite access for SS 8 schema + rules/ + core.rb # [A] not yet built — starter vocabulary (SS 9) +``` + +Everything under `lib/` past `lexer.rb` is unimplemented; this document is the spec for building it. + +--- + +## 13. Deployment / First Run **[A]** + +1. Install Ruby (3.x), `bundle install` for the gems above. +2. Install Ollama; `ollama pull qwen3:1.7b` (or chosen model). +3. Run DB migration script to create `rules.sqlite3`, `docs.sqlite3`, `audit.sqlite3` from §8 schema. +4. Load starter rule set (§9) via `Dir.glob("rules/**/*.rb").each { |f| load f }`. +5. Warm the doc cache lazily on first query per tool (don't pre-index the whole system). +6. First run: confirm Ollama is reachable at `localhost:11434` before enabling Tier 3. + +--- + +## 14. Open Decisions for You to Make While Building + +- Exact confidence thresholds for Tier 2 → Tier 3 escalation (start conservative, tune from audit log data). +- How many successful repeats before `provisional` → `trusted` (suggest starting at 3). +- Whether corrections ("no, I meant Y") look back 1 turn only or a short rolling buffer. +- Which web search backend (Brave API vs. self-hosted SearXNG) — affects §12 setup steps. diff --git a/nlshell.rb b/nlshell.rb new file mode 100755 index 0000000..16fa1fe --- /dev/null +++ b/nlshell.rb @@ -0,0 +1,234 @@ +#!/usr/bin/env ruby + +# frozen_string_literal: true +require "set" +require "strscan" + +# Scan the environment for possible shell commands +ShellCommands = Set.new +ENV.fetch("PATH", "").split(File::PATH_SEPARATOR).reverse.each do |dir| + pp dir + next unless File.directory?(dir) + Dir.each_child(dir) do |name| + path = File.join(dir, name) + ShellCommands << name if File.file?(path) && File.executable?(path) + end +end + +module Lexer + # type: :word | :literal | :punct + # text: the word/punct as typed, or the literal's exact value + # kind: literal kind (:path, :string, ...) or nil + Token = Struct.new(:type, :text, :kind) do + def word? = type == :word + def literal? = type == :literal + def punct? = type == :punct + + # What Tier 1/2 matchers see: the word itself, or the literal's kind. + # Different filenames therefore score identically. + def key = literal? ? kind.to_s : text + + def to_s = literal? ? "#{kind}(#{text.inspect})" : "#{type}(#{text})" + end + + # Tried in order at each token start; first match wins. Because we are always + # at a token boundary, no lookbehinds are needed. + LITERALS = { + string: /"[^"]*"|`[^`]*`|'[^']*'/, + url: %r{[a-z][a-z0-9+.-]*://\S+}i, + env: /\$\{?[a-z_]\w*\}?/i, + glob: %r@[\w.~/-]*(?:\*|\[[^\]\s]+\]|\{[^}\s]+\})[\w.~/*?\[\]{},-]*@, + path: %r@(?:(?:~|\.{1,2})?(?:/[^\s/]+)+/?|[\w.-]+(?:/[\w.-]+)+/?|~(?=\s|\z))@, + flag: /--?[a-z][\w-]*(?:=\S+)?/i, + number: /\d+(?:\.\d+)?(?:kb|mb|gb|tb|ms|min|[kmgshd%])?(?![\w]|\.\w)/i, + file: %r{(?:\.[a-z][\w.-]*|[\w-]+(?:\.[\w-]+)*\.[a-z0-9]{1,5})(?![\w/])}i, + }.freeze + + # Sentence punctuation that greedy patterns may swallow ("rm ~/old.log.") + TRAILING = /[.,;:!?]+\z/ + + # Ends a sentence. ";" also separates commands. + BOUNDARIES = %w[. ? ! ;].freeze + + def self.tokenize(input) + sc = StringScanner.new(input) + tokens = [] + until sc.eos? + next if sc.skip(/\s+/) + + tokens << (read_literal(sc) || read_word(sc) || read_punct(sc)) + end + tokens + end + + def self.read_literal(sc) + LITERALS.each do |kind, re| + text = sc.scan(re) or next # Match or skip to the next + + if kind == :string + text = text[1..-2] # drop quotes, keep content + elsif kind == :file and ShellCommands.include?(text) + return Token.new(:command, text, kind) + else + tail = text[TRAILING].to_s + unless tail.empty? + sc.pos -= tail.bytesize # hand sentence punctuation back to the scanner + text = text.delete_suffix(tail) + end + end + + return Token.new(:literal, text, kind) + end + nil + end + + def self.read_word(sc) + text = sc.scan(/[[:alnum:]_][[:alnum:]_'-]*/) + if ShellCommands.include?(text) + Token.new(:command, text) + else + text && Token.new(:word, text) + end + end + + # Anything else is punctuation, including stray quote characters. + def self.read_punct(sc) + Token.new(:punct, sc.getch) + end + + # After tokenizing, a leftover " or ` means an unpaired quote. + def self.unbalanced_quote?(tokens) + tokens.any? { |t| t.punct? && ['"', "`"].include?(t.text) } + end + + # Lowercase words only. Literals keep their exact case by construction. + def self.downcase_words(tokens) + tokens.map { |t| t.word? ? Token.new(:word, t.text.downcase) : t } + end + + # Split into sentences at . ? ! ; (the terminator stays with its sentence). + def self.sentences(tokens) + tokens + .slice_after { |t| t.punct? && BOUNDARIES.include?(t.text) } + .reject { |s| s.all?(&:punct?) } + .to_a + end + + def self.clauses(tokens) + parts = [[]] + i = 0 + while i < tokens.length + len = connective_length(tokens, i) + if len.positive? + parts << [] unless parts.last.empty? + i += len + else + parts.last << tokens[i] + i += 1 + end + end + parts.reject(&:empty?) + end + + KNOWN_ACTION_WORDS = Set.new(%w[run]).freeze + + ConnectivePatterns = [ + ["after", "that"], + ["and", "then"], + ["then"], + ["next"], + ] + + # Length in tokens of a connective starting at i, or 0 if none matches. + def self.connective_length(tokens, i) + length = 0 + t, n = tokens[i], tokens[i + 1] + ConnectivePatterns.each do |pattern| + slice = tokens[i..(i+pattern.length-1)].map{|e| (e ? e.text : "") } + if slice == pattern + return pattern.length + end + end + + # Check for "and " or ", and " patterns + if t.text == "and" && n&.word? && KNOWN_ACTION_WORDS.include?(n.text) + length = 1 + elsif t.text == "," && n&.text == "and" && tokens[i + 2]&.word? && KNOWN_ACTION_WORDS.include?(tokens[i + 2].text) + length = 2 + end + + # return the length + length + end + + # Text form for Tier 1/2 matching: words and literal kinds, no punctuation. + def self.match_text(tokens) + tokens.reject(&:punct?).map(&:key).join(" ") + end +end + + +module Ruleset + class RulesModule + def types(hash) + end + + def connectives(list) + end + end + + def self.create(name, &block) + rules = RulesModule.new + rules.instance_eval(&block) + rules + end +end + +# Create default ruleset +Ruleset.create("builtins") do + types { + string: /"[^"]*"|`[^`]*`|'[^']*'/, + url: %r{[a-z][a-z0-9+.-]*://\S+}i, + env: /\$\{?[a-z_]\w*\}?/i, + glob: %r@[\w.~/-]*(?:\*|\[[^\]\s]+\]|\{[^}\s]+\})[\w.~/*?\[\]{},-]*@, + path: %r@(?:(?:~|\.{1,2})?(?:/[^\s/]+)+/?|[\w.-]+(?:/[\w.-]+)+/?|~(?=\s|\z))@, + flag: /--?[a-z][\w-]*(?:=\S+)?/i, + number: /\d+(?:\.\d+)?(?:kb|mb|gb|tb|ms|min|[kmgshd%])?(?![\w]|\.\w)/i, + file: %r{(?:\.[a-z][\w.-]*|[\w-]+(?:\.[\w-]+)*\.[a-z0-9]{1,5})(?![\w/])}i, + } + + connectives [ + ["after", "that"], + ["and", "then"], + ["then"], + ["next"], + ] + +end + +# Start the REPL now +loop do + print ":> " + text = gets + exit 0 if text.nil? + + # Tokenize the request + tokens = Lexer.tokenize(text.chomp) + if Lexer.unbalanced_quote?(tokens) + puts "unmatched quote, please close it and try again" + next + end + tokens = Lexer.downcase_words(tokens) + + # Debug only: print the sentences found + Lexer.sentences(tokens).each_with_index do |sentence, i| + puts "#{i}: #{sentence.map(&:to_s).join(' ')}" + Lexer.clauses(sentence).each_with_index do |clause, j| + puts " clause #{j}: #{clause.map(&:to_s).join(' ')}" + end + puts " match: #{Lexer.match_text(sentence)}" + end + + # Later stages (clause splitting, classification, tiers) take a sentence's + # token array. The executor reads literal.text directly; no restore step. +end diff --git a/schema.md b/schema.md new file mode 100644 index 0000000..6d652a9 --- /dev/null +++ b/schema.md @@ -0,0 +1,132 @@ +# NDF JSON Schema Reference + +Concise structural reference for the JSON files described in [dataset.md](dataset.md), plus the engine that actually consumes them ([docs/FlintParser.md](docs/FlintParser.md), [docs/FlintNPC.md](docs/FlintNPC.md), [docs/commands-security.md](docs/commands-security.md)). Types are given as TypeScript-style interfaces (documentation only, not executable). + +## File map + +```text +dataset/ + dataset_*.json -> Entry[] exact-match Q&A, no NER + personality.json -> Entry[] small talk / greetings / fallback + templates_*.json -> Template[] NER-driven semantic matching + types.json -> Record validates/captures non-vocab tags + vocabulary/ + templates.json -> Record synonyms for template "vocab" tags + vocabulary.json -> Record word lists: input sanitizing + <||tag||> output +``` + +## How the files interact + +```mermaid +flowchart LR + D[dataset_*.json
Entry] -->|output/thinking text| TAGS + P[personality.json
Entry] -->|output/thinking text| TAGS + T[templates_*.json
Template.structure] -->|type: vocab_x| VT[vocabulary/templates.json] + T -->|type: anything else| TY[types.json regex] + T -->|output/thinking text| TAGS + TAGS["‹||tag||› resolution"] --> R1[1. entities captured by structure] + TAGS --> R2[2. vocabulary/vocabulary.json] + TAGS --> R3[3. system values e.g. username, npc_name] + TAGS --> R4[4. custom keys on the same entry] +``` + +## Input pipeline (FlintParser + FlintNPC) + +FlintNPC (the agent framework) wraps FlintParser (the NLU engine) into a five-step pipeline, each step tried only if the previous one didn't resolve the prompt: + +```mermaid +flowchart TD + IN["user prompt"] --> S0["0. Sanitize and sentiment:
strip expletives/interjections via vocabulary.json,
tally sentiment counters"] + S0 --> S1{"1. Exact match
hash lookup"} + S1 -->|"hit"| RESP["render output/thinking,
substitute tags, return tools"] + S1 -->|"miss"| S2{"2. Template match
tokenize plus tag structure vs templates_*.json"} + S2 -->|"hit, binds slots"| RESP + S2 -->|"miss"| S3{"3. Probabilistic match
IDF-weighted Levenshtein similarity vs threshold"} + S3 -->|"above threshold"| RESP + S3 -->|"below threshold"| S4["4. Rejection:
return status rejected
plus related intent suggestions"] + RESP --> CTX["update active context
from matched entry.context"] +``` + +1. **Sanitize & sentiment** — noise words (expletives, interjections, thanking/encouraging/discouraging words from `vocabulary/vocabulary.json`) are stripped from the prompt and tallied into a running sentiment count; this also drives the response `emoji`. +2. **Exact match** — the sanitized prompt is looked up verbatim in a hash map built from every `input` string across `dataset_*.json`/`personality.json`. **Active context entries are checked first and override the global dataset on key collision** (e.g. "move it" after "create a directory" resolves against the context entry, not an unrelated global one). Confidence `1.0`, `status: "exact match"`. +3. **Template match** — remaining tokens are each fuzzy-matched (typo-tolerant, Levenshtein) to a `vocab_*` tag or extracted as a slot via a `types.json` regex, producing an ordered tag sequence; this sequence is walked against each `templates_*.json` entry's `structure` paths (`required: false` nodes may be absent). A hit binds slot values (e.g. `<||color||>` -> `"red"`) for substitution, `status: "template match"`. +4. **Probabilistic match** — fallback only. Computes an asymmetric, order-independent BOW + IDF-weighted Levenshtein similarity between the prompt and every known `input`/template phrase (rare words weighted higher, missing "anchor" words penalized more than extra "query" words). The threshold is adaptive: `word_threshold` for prompts ≤3 words, `sentence_threshold` otherwise (both from the NPC's `config.json`, default `0.75`). Highest-scoring match above threshold wins. +5. **Rejection** — if nothing clears the threshold, returns `status: "rejected"` plus up to `suggestions` (from `config.json`) related intents, ranked by the prompt's rarest (highest-IDF) word. + +Once any stage 2-4 produces a match: one `output`/`thinking` string is picked (randomly, if an array) and `<||tag||>` placeholders are substituted (bound slots -> `vocabulary/vocabulary.json` -> system values -> custom keys, per the resolution order above). The entry's `tools` array is returned as-is (arguments already substituted) for the caller to execute — FlintNPC itself does not execute tools, it only extracts them; any embedded terminal command must follow TCSS (`docs/commands-security.md`): single-quote-isolate arbitrary tags, use `printf` when interpolating multiple variables, and put `--` before user-controlled filenames/args to block flag injection. + +**Context update**: the matched entry's `context` key controls what's checked first on the *next* turn — providing an array overwrites the active context, `"context": []` clears it, and omitting the key leaves existing context untouched. + +**Composite prompts**: a single input can be split on `;`, `!`, `?`, newlines, or numbered-list markers (`1)`) into sub-prompts; each runs the full pipeline independently and responses/sentiment are aggregated before returning. +``` + +## Common building block: `Entry` + +Used directly by `dataset_*.json` and `personality.json`, and as the base shape of `Template`. + +```ts +interface Entry { + category?: string; // groups related entries + input: string | string[]; // user prompt(s); array = exact-match alternatives + output: string | string[]; // response(s); array = pick one at random + thinking?: string | string[]; // optional "reasoning" shown/logged before output + tools?: ToolCall[]; // optional side effects to execute + permission?: string; // e.g. "yolo" + context?: Entry[]; // follow-up entries, active only after this one fires; + // recursive; re-triggering overwrites same-keyed context +} + +type ToolCall = Record; // arbitrary shape; any embedded shell + // command must follow TCSS (docs/commands-security.md) +``` + +## `templates_*.json` -> `Template[]` + +```ts +interface Template extends Entry { + type: "template"; + structure: StructureNode[][]; // one or more alternative semantic maps (OR'd) +} + +interface StructureNode { + tag: string | string[]; // literal tag(s); any one satisfies the node + type: string; // "vocab_*" -> look up vocabulary/templates.json + // anything else -> must be a key in types.json + required: boolean; // false = node may be absent from the phrase +} +``` + +Each inner array in `structure` is one ordered semantic map to test the input against; matching binds captured values to their tags for use in `output`/`thinking` (e.g. `<||string||>`). + +## `types.json` -> `Record` + +```ts +type Types = Record; // "<||name||>" -> regex pattern (as a string) +``` + +Keys are referenced by `StructureNode.type` (when not `vocab_*`) to validate/capture free-form input (`string`, `word`, `number`, `filename`, ...). Extensible — add any key/regex pair. + +## `vocabulary/templates.json` -> `Record` + +```ts +type TemplateVocab = Record; // "<||vocab_x||>" -> synonyms +``` + +Referenced by `StructureNode.tag` when `StructureNode.type` is `vocab_*`; lets one `structure` match many phrasings. + +## `vocabulary/vocabulary.json` -> `Record` + +```ts +type Vocabulary = Record; // e.g. thanking_words, greetings, stop_words +``` + +Dual purpose: lists are stripped from/detected in incoming input during pre-processing, and serve as the random source for matching `<||name||>` output tags. + +## `<||tag||>` resolution order + +1. **Entities** captured via a `Template.structure` node whose `type` is a key in `types.json`. +2. **Vocabulary** entries from `vocabulary/vocabulary.json` (random pick). +3. **System values** computed at load time (`username`, `npc_name`, `creation_date`, `creator`, `computer_name`, `response_classes`, ...). +4. **Custom keys** present on the same JSON object as the string being rendered. + +Unresolved tags render as an empty string.