# DataNet SDK for TouchDesigner

> Alpha status: this package is currently a build guide and reference implementation. The repo does not yet ship a ready-to-install `.tox`, `.toe`, or packaged TouchDesigner module.

TouchDesigner doesn't use a traditional importable library — integration is done through Python DATs (Text DATs with Python scripts), custom Components (`.tox` files), and DAT Execute CHOPs. Here's what's needed and how to build it.

---

## How TouchDesigner Integration Works

TouchDesigner runs **Python 3.11** (as of TD 2023+). It has access to the full Python standard library including `asyncio`, `threading`, `json`, and `urllib`. However, **third-party pip packages** are not available by default — you either use the built-in `websocket` module (Python 3.11+ has `websockets` support via `http.client`) or ship a bundled package.

The recommended approach: use Python's built-in `http.client` for the auth REST call and `websocket` (via the `websockets` pip package added to TD's Python environment) for the WebSocket connection.

---

## What Needs to Be Built

### 1. Python Script DAT — `DataNet.py`

A standalone Python script designed to be pasted into a **Text DAT** in TouchDesigner. It:

- Uses `threading.Thread` + a background event loop for the WebSocket (TD's main thread must never block)
- Uses `http.client` or `urllib.request` for the auth POST (avoids needing `aiohttp`)
- Uses `websocket-client` (sync) or `websockets` (async) for the WebSocket connection
- Exposes a simple callable API usable from any other Python DAT in the same project:
  ```python
  dn = op('DataNet').module.datanet   # access the module's global
  dn.subscribe("sensors/temp", lambda ch, data: op('ReceivedDAT').appendRow([data]))
  dn.publish("sensors/temp", {"value": 42.0})
  ```

### 2. TouchDesigner Component — `DataNet.tox`

A prebuilt `.tox` Component containing:
- The Python script DAT wired up and ready
- A **Table DAT** output that auto-appends incoming messages as rows (channel, data JSON, timestamp)
- A **CHOP** output that maps numeric fields from incoming messages to CHOP channels in realtime
- **Custom Parameters** on the Component COMP for:
  - `API Key` (string par)
  - `Channel` (string par)
  - `API URL` (string par, default `https://api.datanet.art`)
  - `WS URL` (string par, default `wss://ws.datanet.art`)
  - `Auto Connect` (toggle, default ON)
  - `Status` (read-only string par — shows "Connected", "Disconnected", "Error: ...")
- A **DAT Execute DAT** that fires callbacks on message receive → writes to Table DAT + converts to CHOP channels
- A **Script CHOP** that reads the Table DAT and outputs numeric fields as CHOP channels

### 3. Python Package Setup Script — `setup_packages.py`

A one-time setup script to install required packages into TouchDesigner's Python environment:

```python
# Run this in a TouchDesigner Script DAT or the Textport
import subprocess, sys

# Get TD's Python executable
td_python = sys.executable

# Install websocket-client (sync, simpler for TD's threading model)
subprocess.run([td_python, "-m", "pip", "install", "websocket-client"], check=True)
# OR for async approach:
# subprocess.run([td_python, "-m", "pip", "install", "websockets"], check=True)

print("Done. Restart TouchDesigner.")
```

---

## Architecture Decisions for TouchDesigner

### Threading Model

TouchDesigner is **single-threaded on the main cook thread**. Your Python callbacks from WebSocket messages **cannot** write to operators directly from a background thread — you'll get a crash or corrupt state.

**Solution:** Use a thread-safe queue:
```python
import queue, threading

_message_queue = queue.Queue()

def _ws_thread():
    # background thread: receive messages, put into queue
    while True:
        msg = ws.recv()
        _message_queue.put(msg)

def poll():
    # called from a DAT Execute or Script CHOP's cook method (main thread)
    while not _message_queue.empty():
        msg = _message_queue.get_nowait()
        # safe to write to ops here
        op('messages').appendRow([msg['ch'], str(msg['d']), msg['ts']])
```

Wire a **DAT Execute DAT** to a **Timer CHOP** (or use the project's `onFrameStart` callback) to call `poll()` every frame.

### CHOP Output

For realtime CHOP data (e.g., sensor values mapped to channels for LFO-style data):

```python
# In a Script CHOP's cook() method
def cook(scriptOp):
    scriptOp.clear()
    last = datanet.getLastMessage("sensors/env")
    if last:
        scriptOp.appendChan("temperature")[0] = last["data"].get("temperature", 0)
        scriptOp.appendChan("humidity")[0] = last["data"].get("humidity", 0)
```

### SSL/TLS

TouchDesigner's Python includes the system SSL certificates, so `wss://` connections work out of the box with `websocket-client`. No special cert configuration needed.

---

## File Structure to Build

```
packages/sdk-touchdesigner/
├── DataNet.py              # Standalone Python module (paste into Text DAT)
├── DataNet.tox             # Prebuilt TouchDesigner component (binary, built in TD)
├── setup_packages.py       # One-time package installer
├── examples/
│   ├── basic_subscribe/
│   │   └── basic_subscribe.toe    # Minimal TD project showing subscribe → Table DAT
│   ├── sensor_dashboard/
│   │   └── sensor_dashboard.toe   # Full dashboard with CHOP channels + rendering
│   └── publish_from_chop/
│       └── publish_from_chop.toe  # Reads a CHOP and publishes values to DataNet
└── README.md               # This file
```

> **Note:** `.tox` and `.toe` files are binary TouchDesigner formats and must be created inside TouchDesigner itself — they cannot be generated by a script. The Python module (`DataNet.py`) and setup scripts can be written as plain text and are the primary deliverables.

---

## `DataNet.py` — The Core Module

```python
"""
DataNet.py — SJS DataNet SDK for TouchDesigner
Paste this into a Text DAT named 'DataNet' in your project.
Access via: op('DataNet').module.connect(api_key)
"""
import json, threading, queue, time, urllib.request, urllib.error

# Optional: websocket-client (pip install websocket-client)
try:
    import websocket
    HAS_WS = True
except ImportError:
    HAS_WS = False
    print("DataNet: websocket-client not installed. Run setup_packages.py first.")

API_URL = "https://api.datanet.art"
WS_HOST = "ws.datanet.art"
WS_PORT = 443
WS_PATH = "/ws"

_message_queue = queue.Queue()
_ws = None
_jwt = None
_subscriptions = {}  # channel -> [callbacks]
_event_callbacks = {"connect": [], "disconnect": [], "error": []}
_connected = False
_heartbeat_thread = None
_ws_thread = None
_stop_event = threading.Event()


def connect(api_key, api_url=API_URL, ws_url=None):
    """Fetch JWT and open WebSocket. Call from main thread (e.g., project start)."""
    global _jwt, _ws, _ws_thread, _heartbeat_thread

    # Fetch JWT
    try:
        req = urllib.request.Request(
            f"{api_url}/auth/token",
            data=json.dumps({"apiKey": api_key}).encode(),
            headers={"Content-Type": "application/json"},
            method="POST"
        )
        with urllib.request.urlopen(req, timeout=10) as resp:
            _jwt = json.loads(resp.read())["token"]
    except Exception as e:
        _fire_event("error", e)
        raise

    # Open WebSocket in background thread
    _stop_event.clear()
    _ws_thread = threading.Thread(target=_run_ws, daemon=True)
    _ws_thread.start()


def _run_ws():
    global _ws, _connected

    ws_url = f"wss://{WS_HOST}:{WS_PORT}{WS_PATH}"

    def on_open(ws):
        global _connected
        _connected = True
        # Re-subscribe all channels
        for ch in _subscriptions:
            ws.send(json.dumps({"op": "sub", "ch": ch}))
        _message_queue.put(("__event__", "connect", None))

    def on_message(ws, message):
        try:
            env = json.loads(message)
            if env.get("op") == "pub" and "ch" in env:
                _message_queue.put(("__msg__", env))
        except Exception:
            pass

    def on_error(ws, error):
        _message_queue.put(("__event__", "error", error))

    def on_close(ws, code, msg):
        global _connected
        _connected = False
        _message_queue.put(("__event__", "disconnect", None))

    _ws = websocket.WebSocketApp(
        ws_url,
        header={"Sec-WebSocket-Protocol": f"bearer, {_jwt}"},
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close,
    )
    _ws.run_forever(ping_interval=30, ping_timeout=10)


def subscribe(channel, callback):
    """Register a callback for a channel. callback(channel, data, from_, timestamp)"""
    if channel not in _subscriptions:
        _subscriptions[channel] = []
        if _ws and _connected:
            _ws.send(json.dumps({"op": "sub", "ch": channel}))
    _subscriptions[channel].append(callback)


def unsubscribe(channel, callback=None):
    if callback is None:
        _subscriptions.pop(channel, None)
        if _ws and _connected:
            _ws.send(json.dumps({"op": "unsub", "ch": channel}))
    elif channel in _subscriptions:
        _subscriptions[channel] = [c for c in _subscriptions[channel] if c != callback]


def publish(channel, data):
    if _ws and _connected:
        _ws.send(json.dumps({"op": "pub", "ch": channel, "d": data}))


def disconnect():
    global _connected
    _connected = False
    _stop_event.set()
    if _ws:
        _ws.close()


def on(event, callback):
    if event in _event_callbacks:
        _event_callbacks[event].append(callback)


def _fire_event(event, *args):
    _message_queue.put(("__event__", event, args))


def poll():
    """
    Call this every frame from a DAT Execute or Script CHOP.
    Drains the message queue and dispatches callbacks safely on the main thread.
    """
    dispatched = 0
    while not _message_queue.empty() and dispatched < 50:
        item = _message_queue.get_nowait()
        dispatched += 1
        if item[0] == "__msg__":
            env = item[1]
            ch = env.get("ch", "")
            data = env.get("d")
            from_ = env.get("from", "")
            ts = env.get("ts", int(time.time() * 1000))
            for cb in _subscriptions.get(ch, []):
                try:
                    cb(ch, data, from_, ts)
                except Exception as e:
                    print(f"DataNet: callback error on {ch}: {e}")
        elif item[0] == "__event__":
            _, event, args = item
            for cb in _event_callbacks.get(event, []):
                try:
                    cb(*(args or []))
                except Exception as e:
                    print(f"DataNet: event callback error ({event}): {e}")
```

---

## Wiring It Up in a TD Project

### Step 1: Install the package
Run `setup_packages.py` in the Textport once.

### Step 2: Create the Text DAT
- Add a **Text DAT**, name it `DataNet`
- Paste `DataNet.py` into it

### Step 3: Create a startup Script DAT
```python
# In project1/callbacks → onStart
import op
dn = op('DataNet').module
dn.on('connect', lambda: print("DataNet connected!"))
dn.subscribe("sensors/temperature", lambda ch, data, frm, ts:
    op('messages').appendRow([ch, str(data), ts]))
dn.connect("ak_your_key_here")
```

### Step 4: Poll every frame
- Add a **DAT Execute DAT**, set event to "Frame Start"
```python
def onFrameStart(dat):
    op('DataNet').module.poll()
```

### Step 5: Map to CHOP
- Add a **Script CHOP**
```python
def cook(scriptOp):
    scriptOp.clear()
    mod = op('DataNet').module
    last_temp = mod._subscriptions.get("sensors/temperature")
    # Or maintain a dict of last values in DataNet.py
    scriptOp.appendChan("temperature")[0] = 0  # populate from your last_value dict
```

---

## What Still Needs to Be Built

| Item | Status | Notes |
|---|---|---|
| `DataNet.py` core module | Done above | Paste into Text DAT |
| `setup_packages.py` | Done above | Run once in Textport |
| `DataNet.tox` component | **Needs TD** | Build interactively in TouchDesigner |
| `.toe` example projects | **Needs TD** | Binary format, built in TD |
| CHOP channel mapper | Partial | Needs `last_values` dict in `DataNet.py` |
| Numeric field auto-extraction | Not started | Parse JSON numbers → CHOP channels automatically |

The Python code above is the hard part. The `.tox`/`.toe` files are best created by opening TouchDesigner, wiring the nodes, and hitting **Save Component**.

---

## Tips

- **Never call `time.sleep()` on the main thread** in TouchDesigner — use `poll()` + the frame loop instead.
- **`websocket-client`** (sync) is simpler for TD than `websockets` (async) because TD's threading model doesn't play well with `asyncio` event loops.
- **Numeric data maps naturally to CHOPs** — structure your DataNet payloads as flat objects with numeric values: `{"temperature": 22.4, "humidity": 65.1}` → two CHOP channels.
- **Table DATs** are great for storing message history — `appendRow()` on every message, set a row limit to avoid memory growth.
- For **multi-channel subscriptions**, maintain a dict of last values and expose a `get_value(channel, key)` helper function that the Script CHOP can call.