# NOOX

## 1. Project Introduction

NOOX is an open-source intelligent hardware platform designed for scenarios involving "external hardware + desktop automation + Large Language Models (LLMs)". It emphasizes plug-and-play convenience and portability, aiming to achieve intelligent control and automated operations of host computers through a single ESP32-S3-based device. NOOX combines hardware control, desktop automation, and AI autonomous planning capabilities, providing users with a powerful intelligent desktop assistant and development tool.

## 2. Features

The NOOX platform offers the following core features:

*   **USB Composite Device**: The device connects to the host via a USB Type-C interface, simultaneously emulating USB HID (Keyboard/Mouse) and USB CDC (Virtual Serial Port) for bidirectional communication and control with the host.
*   **AI Integration and Autonomous Planning**:
    *   Supports multiple LLM providers (OpenRouter / DeepSeek / OpenAI-compatible APIs).
    *   In advanced mode, the AI can perform multi-step planning based on context and command output, automatically invoke tools, and iterate execution until the user's goal is achieved.
    *   Supports dialogue history management to maintain multi-turn conversation context.
*   **Web Control Console**: Built-in HTTP and WebSocket servers provide an intuitive Web UI for device configuration (LLM provider, API key, WiFi settings) and real-time AI conversations.
*   **Host Agent Auto-Bootstrapping**: The device uses USB HID to emulate keyboard operations, automatically opening PowerShell on the host and using `Invoke-WebRequest` to download and execute the cross-platform host agent program from the device's built-in web server.
*   **Cross-Platform Shell Interaction**: The host agent program communicates with the device via USB CDC using JSON, enabling cross-platform Shell command execution and feedback (supporting powershell/pwsh/cmd/bash/sh, etc.).
*   **LLM Tool Capabilities**: Tools callable by the LLM include:
    *   `run_command`: Execute Shell commands on the host.
    *   `hid_keyboard_type`: Type text via HID keyboard.
    *   `hid_keyboard_press`: Simulate key combinations or special keys.
    *   `hid_keyboard_macro`: Execute a series of keyboard operation macros.
    *   `gpio_set`: Control GPIO/LED pins on the ESP32-S3 device.
*   **Multi-Channel Interaction**: In addition to the Web interface, the device offers local UI interaction via an OLED screen and physical buttons, displaying status information and enabling menu navigation.
*   **Dynamic Configuration Management**: Supports runtime modification of LLM provider, API key, and WiFi configurations via the Web interface, with persistent storage.

## 3. Usage

The NOOX platform offers multiple interaction methods, allowing users to flexibly interact with the device and AI.

### 3.1 Web Interface Interaction

Access the Web console by browsing to the device's IP address (viewable on the OLED screen).

1.  **Chat Mode**:
    *   Enter natural language messages in the input box on the main interface and click "Send" or press Enter.
    *   The AI will respond in natural language and supports Markdown rendering for clear, formatted information.
    *   Switch to this mode using the "Chat" button in the top bar.
2.  **Advanced Mode**:
    *   Switch to this mode using the "Advanced" button in the top bar.
    *   In this mode, the AI can autonomously plan and invoke tools (e.g., `run_command`, `hid_keyboard_type`, `gpio_set`) to complete complex tasks.
    *   Users can observe the AI's tool invocation process and results, and continue inputting instructions....