consolepi 0.0.11
Acts as a serial Console Server, allowing you to remotely connect to ConsolePi via Telnet/SSH/bluetooth to gain Console Access to devices connected to local or remote ConsolePis via USB to serial adapters (i.e. Switches, Routers, Access Points... anything with a serial port).
Requires Python: >=3.8

# ConsolePi

Acts as a serial Console Server, allowing you to remotely connect to ConsolePi via Telnet/SSH/bluetooth to gain Console Access to devices connected to local **or remote** ConsolePis via USB to serial adapters (i.e. Switches, Routers, Access Points... anything with a serial port).

***TL;DR:***
Single Command Install Script. Run from an internet connected RaspberryPi running RaspiOS (ideally a fresh image).  *See [Known Issues](#known-issues) regarding changes in Raspberry Pi OS 12(bookworm).*

```bash
wget -q https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/installer/install.sh -O /tmp/ConsolePi && sudo bash /tmp/ConsolePi && sudo rm -f /tmp/ConsolePi
```

![consolepi animated Demo](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/readme_content/main-demo.gif)


>*Making multiple ConsolePis?  Want to re-image, but pull-over all of your existing configs?  Check out the [image creator](#3-consolepi-image-creator)!!!*

------
<!-- prettier-ignore-start -->
# Contents
- [Known Issues](#known-issues)
- [What's New](#whats-new)
- [Planned Enhancements](#planned-enhancements)
- [Features](#features)
  - [Serial Console Server](#serial-console-server)
    - [Guidance on LAME USB to RS232 adapters](#guidance-on-lame-usb-to-rs232-adapters)
  - [AutoHotSpot](#autoHotSpot)
  - [Automatic VPN](#automatic-vpn)
  - [Automatic PushBullet Notifications](#automatic-pushbullet-notification)
  - [Automatic Wired DHCP Fallback](#automatic-wired-dhcp-fallback)
  - [Clustering / Cloud Sync](#consolepi-cluster--cloud-sync)
    - [Supported Cluster Methods](#supported-cluster-sync-methods)
      - [Google Drive](#google-drive)
      - [mDNS / API](#mdns--api)
      - [Manual](#local-cloud-cache)
  - [Power Outlet Control](#power-control)
  - [Manual Host Entries](#manual-host-entries)
  - [ZTP Orchestration](#ztp-orchestration)
  - [ConsolePi Restful API](#consolepi-api)
  - [ConsolePi Extras](#consolepi-extras)
- [Installation](#installation)
  - [Automated Installation](#1-automated-installation)
    - [Silent Install](#silent-install)
  - [Semi-Automatic Install](#2-semi-automatic-install)
  - [ConsolePi Image Creator](#3-consolepi-image-creator)
  - [Alternative Hardware Installs](#alternative-hardware-installs)
- [ConsolePi Usage](#consolepi-usage)
  - [Configuration](#configuration)
    - [menu sorting and connection settings](#consolepi-menu-sorting-and-connection-settings)
    - [Manual Host Entries](#configuring-manual-host-entries)
    - [Local UART support (GPIO)](#local-uart-support)
    - [Power Control Setup](readme_content/power.md#power-control-setup)
      - [GPIO Connected Relays](readme_content/power.md##gpio-connected-relays)
      - [espHome flashed WiFi Smart Outlets](readme_content/power.md##esphome-flashed-wifi-smart-outlets)
      - [Tasmota flashed WiFi Smart Outlets](readme_content/power.md##tasmota-flashed-wifi-smart-outlets)
      - [DLI Web/Ethernet Power Switch](readme_content/power.md##dli-webethernet-power-switch)
    - [ZTP Orchestration](readme_content/ztp.md)
    - [OVERRIDES](#overrides)
  - [Console Server](#console-server)
    - [TELNET](#telnet)
    - [SSH / BlueTooth (`consolepi-menu`)](#ssh--bluetooth)
  - [Convenience Commands](#convenience-commands)
- [Upgrading ConsolePi](#upgrading-consolepi)
- [Tested Hardware/Software](#tested-hardware--software)
- [ConsolePi @ Work! (Image Gallery)](#consolepi-@-work)
- [Credits](#credits)
<!-- prettier-ignore-end -->
------


# Known Issues

- :bangbang: *Not an issue, but a breaking change in ser2net if you've upgraded.*  Bullseye updates the ser2net available from the package repo from 3.x to 4.x this is a significant change.  Functionality wise it's a very good thing, ser2net 4.x brings a lot more flexibility and connectivity options, but it means if you already have aliases that were configured (via `consolepi-addconsole` or the rename(`rn`) option in `consolepi-menu`) against the ser2net v3 config `/etc/ser2net.conf`... Those are no longer what ser2net uses for the TELNET connections.

  Good news though.  As of v2023-6.0 the various options mentioned above for adding/updating/renaming adapter aliases now work with the v4 config (`/etc/ser2net.yaml`).  Not only that, but there is now a migration command `consolepi-convert` which will migrate an existing ser2net v3 config to v4.

  if the v3 file still exists, the menu and rename options will continue to use it (it will warn you on menu launch if the v3 config is still in place with ser2netv4 installed).  If you run `consolepi-convert` it will migrate the config, and if ser2net v4 is installed it will stash the old v3 config in the bak dir.

- ser2net v4 config parsing errors: I have seen on multiple occasions ser2net throw parsing errors after bootup.  `systemctl restart ser2net` has resolved it, so it's not an actual file format issue.  Investigating why this occasionally happens.
  The errors look something like this:
  ```shell
    ---------------- // STATUS OF ser2net.service \\ ---------------
  ● ser2net.service - Serial port to network proxy
      Loaded: loaded (/lib/systemd/system/ser2net.service; enabled; vendor preset: enabled)
      Active: active (running) since Tue 2023-07-11 14:34:27 CDT; 1 weeks 0 days ago
        Docs: man:ser2net(8)
    Main PID: 407 (ser2net)
        Tasks: 1 (limit: 2057)
          CPU: 131ms
      CGroup: /system.slice/ser2net.service
              └─407 /usr/sbin/ser2net -n -c /etc/ser2net.yaml -P /run/ser2net.pid

  Jul 11 14:34:28 ConsolePi3 ser2net[407]: Invalid port name/number: Invalid data to parameter on line 361 column 0
  Jul 11 14:34:28 ConsolePi3 ser2net[407]: Invalid port name/number: Invalid data to parameter on line 369 column 0
  Jul 11 14:34:28 ConsolePi3 ser2net[407]: Invalid port name/number: Invalid data to parameter on line 377 column 0
  Jul 11 14:34:28 ConsolePi3 ser2net[407]: Invalid port name/number: Invalid data to parameter on line 385 column 0
  Jul 11 14:34:28 ConsolePi3 ser2net[407]: Invalid port name/number: Invalid data to parameter on line 393 column 0
  Jul 11 14:34:28 ConsolePi3 ser2net[407]: Invalid port name/number: Invalid data to parameter on line 401 column 0
  Jul 11 14:34:28 ConsolePi3 ser2net[407]: Invalid port name/number: Invalid data to parameter on line 409 column 0
  Jul 11 14:34:28 ConsolePi3 ser2net[407]: Invalid port name/number: Invalid data to parameter on line 420 column 0
  Jul 11 14:34:28 ConsolePi3 ser2net[407]: Invalid port name/number: Invalid data to parameter on line 431 column 0
  Jul 11 14:34:28 ConsolePi3 ser2net[407]: Invalid port name/number: Invalid data to parameter on line 442 column 0
  ```
  Again simply restarting the service seems to resolve the issue.  I suspect it may be a race-condition on bootup with the initialization of udev.

  > ℹ  You can disable ser2net if you never use TELNET to connect directly to the adapters on the ConsolePi.  The Rename utility will still use and update the ser2net conf file *(so you don't have to configure the settings in 2 places if you do happen to use it)*.  `sudo systemctl disable ser2net` will disable ser2net, `consolepi-upgrade` won't re-enable it if it was disabled when the you run the upgrade.

# What's New
## June 2025 (v2025-3.8)

  - 🐛 Fix rename of remote adapters
  - 🐛 Fix menu bug corner case that could omit entries if last section of page needs to be split
  - 🩹 Prevent use of invalid character (`.`) in new name for adapters (`.` not allowed by ser2net)
  - 🩹 Fix menu display issue where first launch of the menu would display sections in 3 columns, then any activity in the menu would re-display the menu more vertically.
    - *This didn't result in any entries being omitted, just needlessly changed the way it was displayed*
  - 🩹 Handle SSIDs with spaces in `consolepi-wlanscan`

## Dec 2024 (v2024-3.6 installer v84)

  - 🐛fixes bug re disabling static fallback.
  - 🩹 remove static fallback file if wired_dhcp=false and they have the old template
  - 🦺 ensure wlan_ip / wired_ip has value when using consolepi-upgrade.  Rare occasion where user changes ConsolePi.yaml to `wired_dhcp: true`, but does not update the config with IP addresses.
  - 🔊 fix wired_dhcp log referencing wrong interface in log (wrong variable)
  - ♻️ change logic in disable wired dhcp for NetworkManager
  - 🔧 Update wired static fallback.  set autoconnect priority to lowest possible.
  - ⬆️ upgrade aiohttp dependency

## Oct 2024 (v2024-3.4 installer v83)
  - :pushpin: pin cryptography they've pushed a release with failed build (43.0.3 failes build on piwheels)
  - :bookmark: Installer v83
    - ✨ add flag for installer to skip bluetooth setup
    - ✨ show model info for non rpi if possible
  - :speech_balloon: Simplify `consolepi-status` output.
  - :memo: Documentation updates/improvements


Prior Changes can be found in the - [ChangeLog](changelog.md)

# Planned enhancements
  - Complete automation of feature to provide ssh direct to adapter discussed in [issue #119](https://github.com/Pack3tL0ss/ConsolePi/issues/119)
    - *Still plan to implement this in some form, despite how long it's taken 🤪*
  - Investigate outlet control and PoE port control via HomeAssistant.
  - Non RPI & wsl (menu accessing all remotes) support for the installer.  Should work now, but needs further testing.
  - Ability to pick a non sequential port when using `rn` in menu or `consolepi-addconsole`
  - ✨ *Most excited about* launch menu in byobu session (tmux).  With any connections in a new tab.
  - Eventually... formatting tweaks, and TUI.  Also consolepi-commands turn into `consolepi command [options]` with auto complete and help text for all (transition to typer CLI).

# Features
## **Feature Summary Image**
![consolepi-menu image](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/readme_content/ConsolePi_features.jpg)

## Serial Console Server
**This is the core feature of ConsolePi.**
  - Connect USB to serial adapters to ConsolePi (or use the onboard UART(s)), then access the devices on those adapters via the ConsolePi.
  - Supports TELNET directly to the adapter.  Use `consolepi-addconsole` or the rename(`rn`) option in `consolepi-menu` to create predictable alias for a USB to serial adapter (vs the default /dev/ttyUSB0...) and assign the TELNET port associated with it.
  - Connect to ConsolePi via SSH or BlueTooth and select the adapter from `consolepi-menu`.  The menu will show connection options for any locally connected adapters, as well as connections to any remote ConsolePis discovered via Cluster/sync.  The menu has a lot of other features beyond connecting to local adapters, as shown in the image above.
  <!-- - ssh directly to an adapter by connecting via ssh to port 2222 and providing the adapter name i.e. `ssh -t consolepi4 -p 2202 r1-8360-TOP` -->
- When connecting to the ConsolePi via bluetooth, default behavior is to auto-login and launch a limited function menu.  Given this user is automatically logged in, the user has limited rights, hence the limited function menu (allows access to locally attached adapters).
- The `consolepi` user can also be configured to auto-launch the menu on login (option during install), this user doesn't auto-login, so it's created with typical rights and launches the full menu.
> To disable auto-login via Bluetooth, modify /etc/systemd/system/rfcomm.service and remove `-a blue` from the end of the `ExecStart` line.  Then issue the following to create an empty file telling `consolepi-upgrade` not to update the file on upgrade: `touch /etc/ConsolePi/overrides/rfcomm.service`.


### Guidance on LAME USB to RS232 adapters

There are some lame adapters that don't burn a serial # to the chip, this makes assigning a unique name/TELNET port more challenging.  Refer to the [adapters](adapters.md) page for more guidance.

>Spoiler alert, Adapters based on an FTDI chipset FTW!


## AutoHotSpot

This is handled by NetworkManager, but the installer deploys the appropriate config (if you've chosen to enable the feature).  On boot NetworkManager will first look for any configured SSIDs (you need to build those profiles), if it can't find or is unable to connect it will fallback to hotspot.  Any profiles you setup should have an autoconnect-priority=N where N is anything > 0.  Consolpi will also configure DHCP for hotspot clients such that if the ConsolePi has no network reachability (it is isolated) it will not distribute a gateway to hotspot clients.  This is specifically done so you can connect to a ConsolePi that is not network connected (to access connected serial ports) from a PC via a second NIC.  This way it won't compete with your existing default-route (which would black-hole all your traffic).

> If a domain is provided to the wired port via DHCP, and the hotspot is enabled ConsolePi will distribute that same domain via DHCP to clients.

> On systems initially installed prior to Raspberry Pi OS 12 (bookworm) (Uses dhcpcd vs NetworkManager), autohotspot is managed by the `autohotspot` service which runs a script on boot, but functionality is the same.

## Automatic VPN

When an interface receives an IP address ConsolePi will Automatically connect to a VPN server under the following conditions:
- It's configured to use the Auto VPN feature (ConsolePi.yaml)
- A NetworkManager connection profile has been created with `type=vpn`
- ConsolePi is not on the users home network (determined by the 'domain' handed out by DHCP)
- The internet is reachable.

> You need to configure the NetworkManager connection profile for your VPN.  It is best to manually test it using `nmcli con up <connection id>` to verify the configuration.  You may need to include the `--ask` option when you manually test it so NetworkManager will store any secrets involved.  Refer to NetworkManager documentation for more detail.

## Automatic PushBullet Notification

*(Requires a PushBullet Account, API key, and the app / browser extension.)*

When ConsolePi receives a dynamic IP address.  A message is sent via PushBullet API with the IP so you know how to reach ConsolePi.

![Push Bullet Notification image](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/readme_content/ConsolePiPB1.png)

An additional message is sent once a tunnel is established if the Automatic VPN feature is enabled.

![Push Bullet Notification image](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/readme_content/ConsolePiPB2.png)

Each Time a Notification is triggered **all** interface IPs are sent in the message along with the ConsolePi's default gateway(s).

## Automatic Wired DHCP Fallback

**Use with caution**

The Wired DHCP Fallback function configures the wired interface of ConsolePi to fallback to static when it fails to get an address as a client via DHCP.  It will then enable a DHCP Server on the wired interface (with the ConsolePi as the gateway for the clients).

This is useful when configuring factory-default devices, or on an isolated staging network.  *Caution should be taken on production networks*.

This function also:
- Configures traffic from the wired interface to NAT out of the WLAN interface if the WLAN has an internet connection. (The reverse of Auto-HotSpot)
- Optionally, with `ovpn_share: true` set in the optional `OVERRIDES:` section of ConsolePi.yaml wired devices will share access to the OpenVPN tunnel if established (via Auto-OpenVPN).
  > NAT & ovpn_share override with the new NetworkManager based automations has not been tested.  iptables is not installed with Raspberry Pi OS 12 (bookworm)

> The [ZTP Orchestration](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/readme_content/ztp.md) feature will enable wired fallback to static/DHCP Server when you run `consolepi-ztp`.  `consolepi-ztp -end` restores everything to pre-ZTP state.  You do not need to enable it if ZTP is your only need for it `consolepi-ztp` will handle that.

## ConsolePi Cluster / Cloud Sync

The Cluster feature allows you to have multiple ConsolePis connected to the network, or to each other (i.e. first ConsolePi in hotspot mode, the others connected as clients to that hotspot).  A connection to any one of the ConsolePis in the Cluster will provide options to connect to any local serial adapters, as well as those connected to the other ConsolePis in the cluster (via the `consolepi-menu` command).


![consolepi-menu image](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/readme_content/consolepi-use-diagram.jpg)

**Another look at the menu**

![consolepi-menu image](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/readme_content/menu.png)

> In this example only 1 adapter is connected locally (menu item 1), Items 2 - 26 are adapters discovered on other ConsolePis on the network. Items 27 - 50 are [Manual Host Entries](configuring-manual-host-entries).


### Supported Cluster Sync Methods:

#### Google Drive:
   > Read The [Google Drive Setup](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/readme_content/gdrive.md) for instructions on setting up Google Drive and authorizing ConsolePi to leverage the API.
 - Google Drive/Google Sheets is currently the only external method supported.  Given this gets the job done, it unlikely more external methods will be added.

 - The ConsolePi will automatically exchange information with `ConsolePi.csv` in your gdrive under the following scenarios (*all assume the function is enabled in the config*):
  1. When the ConsolePi receives an IP address, and can reach the google API endpoints.

  2. When consolepi-menu is launched and the `'r'` (refresh) option is selected.

  3. When a USB to Serial adapter is added or removed.  (This happens on a 30 second delay, so if multiple add/removes are made in a 30 second window, only 1 update to the cloud will occur, that update will include everything that happened within the 30 second window)

      > Note: The plan is to disable this update scenario in a future release, as we only need address information for the remote, not a full snapshot of the data.  The remote is queried via API to ensure reachability and get the current list of adapters available, so we no longer need that data stored in the cloud.

  >The Gdrive function uses the hostname as a unique identifier.  If all of your ConsolePis have the same hostname only one of them will be synchronized.  **Make Hostnames unique for each ConsolePi**
  >
  >In all of the above a local cloud cache which includes data for any remote ConsolePis pulled from ConsolePi.csv is updated for the sake of persistence and speed.  The local cloud cache is what is referenced when the menu is initially launched

#### mDNS / API
* ConsolePis advertise themselves on the local network via mDNS (bonjour, avahi, ...)

* 3 daemons run on ConsolePi one that advertises details via mdns and updates anytime a change in available USB-serial adapters is detected, a browser service which browses for remote ConsolePis registered on the network, and the API described below.  The browser service updates the local cloud cache anytime a new ConsolePi is detected.

  > The API described [here](#api) comes into play when enough adapters are plugged in, such that the data payload would be over what's allowed via mDNS.  In the event this occurs... well here is an example:  *ConsolePi-A* has a has enough USB to Serial adapters plugged in to be over the data limit allowed via mDNS, the consolepi-mdnsreg service will detect this and fall-back to advertising *ConsolePi-A* without the adapter data.  *ConsolePi-B*, and *ConsolePi-C* are on the network and discover *ConsolePi-A*.  B and Cs consolepi-mdnsbrowse service will detect that the adapter data was stripped and request the adapter data from A via the API.
  >
  >When `consolepi-menu` is launched any remotes in the cache are queried via the API to ensure an up to date listing of available adapters.  If for some reason it doesn't respond to the API a secondary check is done to verify it is listening on the SSH port.  If it fails both it will be listed as unreachable and will not appear in the menu.  If it fails API, but is listening on the SSH port it won't be in the adapter menu (as we didn't get any adapter data), but will show up in the `rs` (remote shell) menu.

#### Local Cloud Cache
  - local cloud cache:  For both of the above methods, a local file `/etc/ConsolePi/cloud.json` is updated with details for remote ConsolePis.  This cache file can be modified or created manually.  If the file exists, the remote ConsolePis contained within are checked for reachability and added to the menu on launch.

 - The rename option in `consolepi-menu` or the `consolepi-addconsole` command supports assignment of custom aliases used to predictably identify the serial adapters with friendly names (udev rules).  If configured these names are used in `consolepi-menu`, the default device name is used if not (i.e. ttyUSB0), but that's less predictable.

 - `consolepi-menu` does not attempt to connect to the cloud on launch, it retrieves remote data from the local cache file only, verifies the devices are reachable, and if so adds them to the menu.  To trigger a cloud update use the `r` refresh option.
  >Note: that ConsolePi will automatically update the local cache file when it gets an IP address, or adapters are added/removed, so the refresh should only be necessary if other ConsolePis have come online since the the menu was launched.  Additionally ConsolePis will automatically discover each other via mdns if on the same network, this will automatically update the local-cache if a new remote ConsolePi is discovered.

 - Read The [Google Drive Setup](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/readme_content/gdrive.md) for instructions on setting up Google Drive and authorizing ConsolePi to leverage the API.

   #### If you are configuring multiple ConsolePis to use this cluster, you should consider using the [ConsolePi Image Creator](#3.-consolepi-image-creator).  Once You've installed the first ConsolePi, leverage the `consolepi-image` command to pre-stage the micro-sd cards for the other ConsolePis you will be creating.  This script is handy, if duplicating the install across multiple ConsolePis.  It can pre-stage the entire configuration and cut out some install time.

## **Power Control**

The Power Control Function allows you to control power to external outlets.  ConsolePi supports:
  - [digital Loggers](https://www.digital-loggers.com/index.html) Ethernet Power Switch/Web Power Switch (including older models lacking rest API).
  - External relays controlled by ConsolePi GPIO ( Like this one [Digital-Loggers IoT Relay](https://dlidirect.com/products/iot-power-relay) ).
  - [espHome](https://esphome.io) flashed WiFi smart outlets (i.e. SonOff S31).  These are low cost outlets based on ESP8266/ESP32 microcontrollers.
  - [Tasmota](https://blakadder.github.io/templates/) flashed WiFi smart [outlets](https://blakadder.github.io/templates/) These are also esp8266 based outlets similar to espHome.

      > espHome/Tasmota were chosen because it allows for local control without reliance on a cloud service.  So your 'kit' can include a small relatively portable smart outlet which can be programmed to connect to the ConsolePi hotspot.  Then ConsolePi can control that outlet even if an internet connection is not available.
- If the function is enabled and outlets are defined, an option in `consolepi-menu` will be presented allowing access to a sub-menu where those outlets can be controlled (toggle power on/off, cycle).
- Outlets can be linked to Console Adapter(s) (best if the adapter is pre-defined using `consolepi-addconsole` or `rn` option in the menu) or manually defined host connections.  If there is a link defined between the outlet and the adapter/host, anytime you initiate a connection to the adapter/host via `consolepi-menu` ConsolePi will ensure the outlet is powered on.  Otherwise if the link is defined you can connect to a device and power it on, simply by initiating the connection from the menu **Only applies when connecting via `consolepi-menu`**.

    > The power sub-menu **currently** only appears in the menu on the ConsolePi where the outlets are defined (Menu does not display outlets defined on remote ConsolePis).  The auto-power-on when connecting to an adapter linked to an outlet works for both local and remote connections (establishing a connection to an adapter on a remote ConsolePi (clustering / cloud-sync function) via another ConsolePis menu)

### The Power Control Menu
*Defined Outlets will show up in the Power Menu*

![consolepi-menu image](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master//readme_content/powermenu.png)


### Additional Controls for Digital Loggers
*You may have an 8 port digital-loggers web power switch, with only some of those ports linked to devices.  Any Outlets linked to devices will show up in the Power Menu, **All** of the Ports for the DLI (regardless of linkage) will show up in the dli menu*

![consolepi-menu image](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master//readme_content/dlimenu.png)

### Outlet Linkages

Example Outlet Linkage.  In this case the switch "rw-6200T-sw" has 2 ports linked.  Both are on a dli web power switch.  One of the ports is for this switch, the other is for the up-stream switch that serves this rack.  When connecting to the switch, ConsolePi will ensure the linked outlets are powered ON.  *ConsolePi does **not** power-off the outlets when you disconnect.*
```shell

------------------------------------------------------------------------------------
  Ensuring r2-6200T-sw Linked Outlets (labpower2:[1, 4]) are Powered ON
------------------------------------------------------------------------------------

Connecting To r2-6200T-sw...
picocom v3.1
...
```
Refer to [Power Control Setup](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/readme_content/power.md) for details on how to setup Power Control.

## Manual Host Entries
The Manual Host Entries Feature allows you to manually define other SSH or TELNET endpoints that you want to appear in the menu.  These entries will appear in the `rs` (remote shell) menu by default, but can also show-up in the main menu if `show_in_main: true`.  Manual host entries support outlet bindings (Auto Power On when connecting through the menu).

Refer to [Manual Host Entries](#configuring-manual-host-entries) in the Configuration section foe details on how to configure.

## ZTP Orchestration

ConsolePi supports Zero Touch Provisioning(ZTP) of devices via wired ethernet/DHCP.  The feature uses DHCP to trigger ZTP, and supports config file generation using jinja2 templates.  For more details see [`ConsolePi ZTP Orchestration`](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/readme_content/ztp.md).

## ConsolePi API

ConsolePi includes an API with the following available methods
> All Are GET methods the default port is 5000, which can be overridden in the config via the `api_port` key in the optional `OVERRIDES` stanza. See [ConsolePi.yaml.example](ConsolePi.yaml.example).

/api/v1.0/
* adapters: returns list of local adapters
* remotes: returns the local cloud cache
* interfaces: returns interface / IP details
* details: full json representing all local details for the ConsolePi

The swagger interface is @ `/api/docs` or `/api/redoc`.  You can browse/try the less common API methods there.

The API is used by ConsolePi to verify reachability and ensure adapter data is current on menu-load.

> The API is currently unsecured, it uses http, and Auth is not implemented *yet*.  It currently only supports GET requests and doesn't provide any sensitive (credential) data.

## ConsolePi Extras
Toward the end of the install, and via `consolepi-extras` anytime after the install, you are provided with options to automate the deployment (and removal for most) of some additional tools.  This is a selection of tools not required for ConsolePi, but often desired, or useful for the kind of folks that would be using ConsolePi.

![`consolepi-extras`](https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/readme_content/consolepi-extras.png)

>Note: speed test (locally hosted browser based speed-test (server)), is only presented as an option for Pi4/CM4+.  Devices prior had a slow ethernet port, so it wouldn't make sense to use them as a speed-test server.

# Installation

If you have a Linux system available you can use [ConsolePi image creator](#3.-consolepi-image-creator)  to burn the image to a micro-sd, enable SSH, mass-import configurations (if ran from a ConsolePi, optional), and PreConfigure ConsolePi settings (optional).  This script is especially useful for doing headless installations.

**The Following Applies to All Automated Installation methods**
> Note Previous versions of ConsolePi supported import from either the users home-dir (i.e. `/home/wade`) or from a `consolepi-stage` subdir in the users home-dir (i.e. `/home/wade/ConsolePi-stage`).  The import logic directly from the home-dir has not been removed, but going forward any new imports will only be tested using the `consolePi-stage` directory for simplicity.

ConsolePi will **optionally** use pre-configured settings for the following if they are placed in the a `consolepi-stage` subdir in the users home folder (i.e. `/home/wade/consolepi-stage`).  This is optional, the installer will prompt for the information if not pre-configured.  It will prompt you to verify either way.  *Imports only occur during initial install not upgrades.*

staged-file import also supports host specific folders within `consolepi-stage`.  The installer will first check if there is a folder that matches the CosnolePis hostname (you can set via the --hostname flag when you run the installer, if it's still the default.).  So if i'm installing on a system already set with the hostname `ConsolePi5` or i've launched the installer with the `--hostname ConsolePi5` option.  Any imports described below found in `consolepi-stage/ConsolePi5` will take precedence over anything found in the root of `consolepi-stage`

**Examples below use `consolepi-stage/...` for brevity but `consolepi-stage/HOSTNAME/...` is valid for all examples**

- ConsolePi.yaml: This is the main configuration file where all configurable settings are defined.  If provided in the `consolepi-stage` dir the installer will ask for verification then create the working config `/etc/ConsolePi/ConsolePi.yaml`

- 10-ConsolePi.rules: udev rules file used to automatically map specific adapters to user defined aliases, which map to specific TELNET ports.  This file is created automatically during the install if you don't skip the *predictable serial port/names* workflow toward the end.  It's also available after the install via the `rn` (rename) option in the menu, or via the `consolepi-addconsole` command.  This is **highly recommended** in most use cases, and is explained further [here](#telnet).

- ser2net.yaml|ser2net.conf: ser2net configuration will be cp to /etc if found in the stage-dir.

- NetworkManager profiles (should be in `consolepi-stage/NetworkManager/system-connections` or in a host specific folder as described above).  Any profiles found will be cp to /etc/NetworkManager/system-connections and ownership permission requirements will be corrected if necessary.

- certificates (EAP-TLS SSIDs) should be pre-staged in `consolepi-stage/cert`, the installer will scan the pre-staged connection profiles, and pre-stage them in the directory defined within the profile.  It will also verify/adjust the permissions on the private key file.

- authorized_keys/known_hosts: If either of these ssh related files are found they will be placed in both the /home/consolepi/.ssh and /root/.ssh directories (ownership is adjusted appropriately).

- Users home directories.  You can pre-stage random files by placing them in `consolepi-stage/home` or in the case or the root user `consolepi-stage/root`.  The installer will import and set ownership for the default consolepi user, root, any users created during the install, and the current user (if happens to be something different).
> When re-imaging a ConsolePi, it's useful to import the users `.ssh` directory so thier ssh keys stay consistent with the previous image.

- rpi-poe-overlay.dts: This is a custom overlay file for the official Rpi PoE hat.  If the dts is found in the stage dir, a dtbo (overlay binary) is created from it and placed in /boot/overlays.  A custom overlay for the PoE hat can be used to adjust what temp triggers the fan, and how fast the fan will run at each temp threshold.
> Refer to google for more info, be aware some apt upgrades update the overlays overwriting your customization.  I use a separate script I run occasionally which creates a dtbo then compares it to the one in /boot/overlays, and updates if necessary (to revert back to my custom settings)

- autohotspot-dhcp(directory): If you have a autohotspot-dhcp directory inside the `consolepi-stage` dir, it's contents are copied to /etc/ConsolePi/dnsmasq.d/autohotspot.  This is useful if you have additional configs you want to use for autohotspot, dhcp-reservations, etc.  The main config for the autohotspot feature `autohotspot` is still managed by ConsolePi.

- wired-dhcp(directory): If you have a wired-dhcp directory inside the `consolepi-stage` dir, it's contents are copied to /etc/ConsolePi/dnsmasq.d/wired-dhcp.  This is useful if you have additional configs you want to use for wired-dhcp, dhcp-reservations, etc.  The main config for the wired-dhcp feature `wired-dhcp` is still managed by ConsolePi.

- ztp(directory): if a `ztp` directory is found in the `consolepi-stage` dir, it's contents are copied to /etc/ConsolePi/ztp.  This is where your template/variable files, and custom_parsers are configured.

- consolepi-post.sh: Custom post install script.  This custom script is triggered after all install steps are complete.  It runs just before the post-install message is displayed.  Use it to do anything the installer does not cover that you normally setup on your systems.  For Example my consolepi-post.sh script does the following:
  - generates certificates for cockpit if it's installed on the system
  - verifies cloud credentials were imported during the install, and if I forgot to pre-stage them it sftps them from another ConsolePi on the network.
  - Installs python3-cryptography used by one of my other scripts that generates EAP-TLS certificates.
  - Pulls various common .bash_aliases and other *stuff* from my NAS.
  - modify /etc/nanorc to my liking

  This is just an optional mechanism to automatically prep whatever it is you normally prep on your systems after the install completes.  The custom Post install script is only executed on initial install, not on upgrade (unless you supply the `-p|--post` flag to `consolepi-upgrade`).

  > Some functions/variables available to the script (will be in the environment) that you could leverage:
  > - $iam (variable) is the user (script is ran as root hence the sudo -u examples above to run a command as the user that launched the installer)
  > - $logit (function) logit "message to log" ["log-level" defaults to "INFO"] *log-lvl "ERROR" results in script aborting if it hits, so don't set the log-lvl to ERROR*

**To enable the Clustering / Cloud-Sync function see the description [above](#consolepi-cluster--cloud-sync) and the prerequisite [Google Drive Setup](readme_content/gdrive.md)  instructions.**

## **1. Automated Installation**

Install RaspberryPi OS on a raspberryPi and connect it to the network.

Use the command string below to kick-off the automated installer.  The install script is designed to be essentially turn-key.  It will prompt to change hostname, set timezone, and set the consolepi users password.  Be sure to checkout the [image creator script](#3-consolepi-image-creator) if doing a headless install, creating multiple ConsolePis, or want to re-image an existing ConsolePi.

```
wget -q https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/installer/install.sh -O /tmp/ConsolePi && sudo bash /tmp/ConsolePi && sudo rm -f /tmp/ConsolePi
```

>Command String Breakdown:
>
>`wget -q https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/installer/install.sh -O /tmp/ConsolePi` --> Pull current install script from GitHub and place in /tmp/ConsolePi
>
> `sudo bash /tmp/ConsolePi` --> Execute ConsolePi install script with sudo privs
>
> `sudo rm -f /tmp/ConsolePi` --> Remove the install script (script execution is complete at this point)
>
> The `&&` between each command just means the command will execute given there was not an error returned from the previous command.

### Silent Install
A Silent install (Installation runs without prompts) is possible cmd line arguments provided to the installer or a config file, where the path to the config is provided to the installer via the `-C </path/to/config/file.conf>` argument.  A pre configured ConsolePi.yaml should also exist in the `consolepi-stage` dir described above.

Refer to [/etc/ConsolePi/installer/install.conf.example](installer/install.conf.example) for an example config.  This command string will download it to your home dir as install.conf and open it in nano for editing.
```
wget -q https://raw.githubusercontent.com/Pack3tL0ss/ConsolePi/master/installer/install.conf.example -O ~/install.conf && nano ~/install.conf
```

> The output below shows `consolepi-upgrade` as the command to launch, the command will be `consolepi-install` on an image created using the [image creator script](#3-consolepi-image-creator).  If neither is the case you would call the installer directly (with sudo) and pass in the args (The [TL;DR](#consolepi) string at the top of this README can be modified to pass in the arguments)

```
wade@ConsolePi-dev:~$ consolepi-upgrade --help

USAGE: consolepi-upgrade [OPTIONS]

Available Options
 -h|--help                               Display this help text.
 -s|--silent                             Perform silent install no prompts, all variables reqd must be provided via pre-staged configs.
 -C | --config <path/to/config>          Specify config file to import for install variables (see /etc/ConsolePi/installer/install.conf.example).
    Copy the example file to your home dir and make edits to use.
 -6|--no-ipv6                            Disable IPv6 (Only applies to initial install).
 -R|--reboot                             reboot automatically after silent install (Only applies to silent install).
 -w|--wlan_country <2 char country code> wlan regulatory domain (Default: US).
 --locale <2 char country code>          Update locale and keyboard layout. i.e. '--locale us' will result in locale being updated to en_US.UTF-8.
 --us                                    Alternative to the 2 above, implies us for wlan country, locale, and keyboard..
 -H|--hostname <hostname>                If set will bypass prompt for hostname and set based on this value (during initial install).
 --tz <tz>                               Configure TimeZone i.e. America/Chicago.
 -L|--auto-launch                        Automatically launch consolepi-menu for consolepi user.  Defaults to False..
 -p|--passwd '<password>'                Use single quotes: The password to configure for consolepi user during install..
    Any manually added users should be members of 'dialout' and 'consolepi' groups for ConsolePi to function properly

The Following optional arguments are more for dev, but can be useful in some other scenarios
 -P|--post                               ~/consolepi-stage/consolepi-post.sh if found is executed after initial install.  Use this to run after upgrade..
 --no-apt                                Skip the apt update/upgrade portion of