Friday, October 6, 2023

Connect to Fluvius smart energy meter via P1 and publish data via MQTT broker to Home Assistant

I created a python script that runs as a service to read out the Fluvius energy meter via the P1 port to USB on a Raspberry Pi. See installation notes here: https://github.com/smartathome/fluvius2mqtt/tree/main

The output into Home Assistant looks like this:



Monday, October 2, 2023

Install SBFSpot to read out SMA Sunny Boy data and make data available over MQTT for Home Assistant

I want to read out a SMA Sunny Boy data and make it available via MQTT to Home Assistant. Installation of SBFSpot on a Rasberry Pi is really easy. The tool to be used for this is SBFSpot. The SMA procotol used is proprietary and happens over 502/tcp.

  1. Install command line:
    curl -s https://raw.githubusercontent.com/sbfspot/sbfspot-config/master/sbfspot-config | sudo bash
  2. Follow the configuration tutorial and setup wizard
  3. Edit the configuration file for MQTT:
    sudo vi cd /usr/local/bin/sbfspot.3/SBFspot.cfg
    to change the MQTT configuration and set the following MQTT_Data:
    MQTT_Data=Timestamp,SunRise,SunSet,InvSerial,InvName,InvSwVer,InvTime,InvStatus,InvTemperature,InvGridRelay,EToday,ETotal,PACTot,UDC1,UDC2,IDC1,IDC2,PDC1,PDC2,GridFreq
  4. Test:
    /usr/local/bin/sbfspot.3/SBFspot -v -finq -nocsv -mqtt
  5. Note the SMA inverter serial for adding to your Home Assistant configuration
    x
  6. Configure Home Assistant and add these sensors to your configuration.yaml. Remember to replace the SERIAL with the actual serial number from your inverter.



Monday, September 25, 2023

Advanced installation of a Raspberry Pi with Raspbian Bullseye

When installing a Raspberry Pi, I have a checklist of steps I take each time to ensure my Raspberry Pi's are (mostly) configured in the same way. They have the same way to backup their data, use the same user configurations (ntp, syslog, sendmail...) and have the same security provisioning. We will also introduce logs into memory with Log2Ram, to avoid too much SD card writing/wearing, which will eventually break your RPi. Feel free to comment on any step that is documented here. Some steps might be optional or unnecessary in your case.

  1. Do the physical installation, plugin the network and HDMI cables (except the power cable of course) and screw your RPi into a cover or box.
  2. Prepare SD card on Mac with Raspberry Pi Imager
  3. Plugin the SD card into your RPi and now also plugin the power cable. Boot your RPi for the first time now. Create a user with password for using later. (e.g. user:pi, password:raspberry)
  4. When booted, you'll be provided with a prompt to login for the first time. Mind the QWERTY keyboard layout.
  5. Run the setup tool
    sudo raspi-config
  6. Configure the setup tool
    1. Set the hostname (1 System Options > S4 Hostname)
    2. Expand Filesystem (6 Advanced Options > A1 Expand file system)
    3. Change Timezone, set Keyboard Layout (if needed) and change Wifi Country (5 Localization Options > L2 Change Timezone, L3 Change Keyboard Layout, L4 Change Wi-fi Country)
    4. Enable SSH (3 Interfacing Options > I2 SSH)
    5. Press 'Finish' and Reboot
  7. After reboot, login again via SSH and change your user password:
    passwd
  8. Generate a SSH key-gen pair, which is more robust than the default one.
    ssh-keygen -o -a 100 -t ed25519
  9. Change the root password
    sudo passwd root
  10. Set the ETH0 IP address to a fixed IP. I hardly ever use the Wifi module in a Raspberry Pi
    sudo vi /etc/network/interfaces
    Add at the end of the file the following:
    # Added by user on 2023-XX-XX
    auto eth0
    iface eth0 inet static
            address 192.168.0.240/24
            network 192.168.0.0
            broadcast 192.168.0.255
            gateway 192.168.0.1
            dns-nameservers 192.168.0.1 8.8.8.8
    # End of Addition
    sudo systemctl restart networking.service
    And test with
    ip add show
    Reboot your RPi again (or do it later if you plan to reboot anyway)
  11. Check for updates & upgrades for Bullseye, but first become root. Don't forget to reboot if kernel patches were installed.
    sudo -i
    apt-get update -y && apt-get upgrade -y
  12. Fix a common issue with Syslog flooding your logs
    sudo sed -i '/# The named pipe \/dev\/xconsole/,$d' /etc/rsyslog.conf
    sudo service rsyslog restart
  13. Alternatively, you could also install Syslog-NG
    sudo apt-get install -y syslog-ng
  14. Install Git
    sudo apt-get install -y git dirmngr
  15. Install Log2Ram as this will allow us to keep logs in memory and reduce the SD card writing significantly. From time to time, the logs are still made persistent to disk.
    cd /home/pi
    git clone https://github.com/azlux/log2ram.git
    cd log2ram
    chmod +x install.sh
    sudo ./install.sh
    Change the log size value to 128M
    sudo vi /etc/log2ram.conf
    Reboot
  16. Install Sendmail and configure to work with a local mail relay server, or alternatively Gmail.
    sudo apt-get install -y sendmail mailutils sendmail-bin
    sudo mkdir -m 700 /etc/mail/authinfo/
    sudo cd /etc/mail/authinfo/
    Create a Sendmail authentication file:
    sudo vi sendmail-auth
    And paste the following info:
    AuthInfo: "U:root" "I:YOUR LOGIN" "P:YOUR PASSWORD"
    Save and exit vi. Next do the makemap:
    sudo makemap hash sendmail-auth < sendmail-auth
    sudo chmod 400 sendmail-auth
    Change the Sendmail configuration now
    sudo vi /etc/mail/sendmail.mc
    Add the following below right above first "MAILER_DEFINITIONS" line:
    # Added by yourname on 2018-XX-XX
    define(`SMART_HOST',`[192.168.Y.XX]')dnl
    define(`RELAY_MAILER_ARGS', `TCP $h 587')dnl
    define(`ESMTP_MAILER_ARGS', `TCP $h 587')dnl
    define(`confAUTH_OPTIONS', `A p')dnl
    TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
    define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
    FEATURE(`authinfo',`hash -o /etc/mail/authinfo/sendmail-auth.db')dnl
    # End of Addition
    Apply the changes to the configuration and restart Sendmail:
    sudo make -C /etc/mail
    sudo /etc/init.d/sendmail reload
    Test if you can send an email to yourself:
    echo "Just testing my Sendmail email relay" | mail -s "Sendmail email relay" you@here.com
  17. Setup NTP sync
    sudo apt-get install -y ntp ntpdate
    sudo vi /etc/ntp.conf
    And replace the XX with your country code
    0.XX.pool.ntp.org
    sudo /etc/init.d/ntp stop
    And query to see NTP being in sync
    sudo ntpd -gq
    sudo /etc/init.d/ntp start
    sudo ntpd -pn
  18. Setup SNMP
    sudo apt-get install snmp snmpd
    sudo vi /etc/snmp/snmpd.conf
    And put the following configuration lines
    agentAddress udp:161
    rocommunity public 192.168.X.0/24
    Restart your SNMP daemon
    sudo /etc/init.d/snmpd restart
    And test on your local machine
    snmpwalk -Os -c public -v 1 localhost
  19. Setup NFS backup share, install a backup tool, rsnapshot and configure
    Fix rpcbind issue (Make yourself root first)
    su -
    cat >/etc/systemd/system/nfs-common.service <<\EOF
    [Unit]
    Description=NFS Common daemons
    Wants=remote-fs-pre.target
    DefaultDependencies=no
    
    [Service]
    Type=oneshot
    RemainAfterExit=yes
    ExecStart=/etc/init.d/nfs-common start
    ExecStop=/etc/init.d/nfs-common stop
    
    [Install]
    WantedBy=sysinit.target
    EOF

    cat >/etc/systemd/system/rpcbind.service <<\EOF
    [Unit]
    Description=RPC bind portmap service
    After=systemd-tmpfiles-setup.service
    Wants=remote-fs-pre.target
    Before=remote-fs-pre.target
    DefaultDependencies=no
    
    [Service]
    ExecStart=/sbin/rpcbind -f -w
    KillMode=process
    Restart=on-failure
    
    [Install]
    WantedBy=sysinit.target
    Alias=portmap
    EOF

    cat >/etc/tmpfiles.d/rpcbind.conf <<\EOF
    #Type Path        Mode UID  GID  Age Argument
    d     /run/rpcbind 0755 root root - -
    f     /run/rpcbind/rpcbind.xdr 0600 root root - -
    f     /run/rpcbind/portmap.xdr 0600 root root - -
    EOF
    
    systemctl enable rpcbind.service
    systemctl enable nfs-common 
    Install raspiBackup  (from this website)
    sudo mkdir -p /backup 
    Avoid accidental file storage, when folder is not mounted
    And put the following configuration lines
    sudo chattr +i /backup
    sudo vi /etc/fstab 
    And add
    server.yourdomain.com:/volume1/backups/host.yourdomain.com/backup      nfs     rsize=8912,wsize=8912,timeo=14     0       0
    sudo mount /backup
    Now install the raspiBackup tool
    curl -s https://raw.githubusercontent.com/framps/raspiBackup/master/installation/install.sh | sudo bash
    Go through the configuration tool, later on you can go back to it via: raspiBackupInstallUI.sh
    -Backup versions: smart strategy
    -Backup to tar
    -No compression
    -Backup mode standard
    -Email notification set
    Uncomment the crontab (backup will run every Sunday at 5am):
    sudo vi /etc/cron.d/raspiBackup 
    And finally test
    sudo raspiBackup
  20. Generate an SSH keypair for easy login
    ssh-keygen
    ssh-copy-id -p 22 admin@server.yourdomain.com 
    Log into your server, make yourself root and copy the public key into the raspberry
    cat /root/.ssh/id_rsa.pub | ssh user@hhost.yourdomain.com "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys" 
    Test if it's working by using:
    ssh user@host.yourdomain.com 
  21. Setup unattended upgrade based on this tutorial
    sudo apt update
    sudo apt install unattended-upgrades 
    Configure unattended upgrades and uncomment:
    sudo vi /etc/apt/apt.conf.d/50unattended-upgrades
    
    "origin=Debian,codename=${distro_codename}-updates";
    "origin=Debian,codename=${distro_codename}-proposed-updates";
    "origin=Debian,codename=${distro_codename},label=Debian";
    "origin=Debian,codename=${distro_codename},label=Debian-Security";
    "origin=Debian,codename=${distro_codename}-security,label=Debian-Security"; 
    And uncomment:
    Unattended-Upgrade::Remove-Unused-Dependencies "false";
    Now enable Automatic Updates (and press Yes)
    sudo dpkg-reconfigure --priority=low unattended-upgrades
    To view the unattended upgrades:
    sudo systemctl status unattended-upgrades.service
    -



Thursday, September 21, 2023

Watchtower not working on Synology DSM 6.0.x as a Docker container

When installing the standard Watchtower container onto DSM 6.0 with Docker, I ran into 2 issues which caused the container not to run and constantly restart itself (see container logs for the error messages):

  1. Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
  2. Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.23

Both issues have a solution. First being that an older docker image has to be used, second being that DSM UI somewhere filters out a file mapping when creating the container.

Fix for issue 1: Create a symlink

sudo ln -s /var/run/docker.sock /volume1/docker/docker.sock

Fix for issue 2: Create a new docker container

sudo docker run -d --name watchtower-new -v /volume1/docker/docker.sock:/var/run/docker.sock centurylink/watchtower --interval 82800 --cleanup

Thanks to these links for finding a solution

Monday, September 18, 2023

ipmitool not working on Supermicro server with Debian 12

I have a server with a Supermicro X7SBI motherboard. It has a dedicated IPMI port but I could no do anything with it, as this was a second hand server and the IPMI had a fixed IP address in the 83... range and the admin password was unknown to me. I thought that resetting it was straightforward, but ran into a few issues:

First issue: Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

So at first, I thought that the device itself was broken. I unplugged it, checked it, but I could see the IP address and the webserver was accessible, so that seems ok. So I tried to factory reset the BIOS, still no help. Then, after some Googling, I found out that 2 fixes were needed.

1. Adapt the GRUB boot parameters for this type of board

//add logs

These articles explain what needs to be done: different ipmi_si port address and add kernel boot parameter

Run everything as root:

vi /etc/default/grub and change GRUB_CMDLINE_LINUX_DEFAULT="quiet" to

GRUB_CMDLINE_LINUX_DEFAULT="quiet ipmi_si.type=kcs ipmi_si.ports=0xca2"

Save and exit, then update your GRUB

update-grub

Reboot your sever, now the device should be found under /dev/ipmi0

Second issue: Still, there is a second problem with this version of ipmitool for Debian 12

root@debian:~# ipmitool raw 0x6 0x46 0x02
IANA PEN registry open failed: No such file or directory
61 64 6d 69 6e 00 00 00 00 00 00 00 00 00 00 00
root@debian:~# lsmod | grep ipmi
ipmi_si 73728 0
ipmi_devintf 20480 0
ipmi_msghandler 77824 2 ipmi_devintf,ipmi_si
root@debian:~# ipmitool lan set 1 ipsrc static
IANA PEN registry open failed: No such file or directory

2. Copy missing enterprise numbers

After some Googling, I found out there is some config missing for ipmitool. This command will solve that:

wget -O /usr/share/misc/enterprise-numbers.txt https://jff.email/cgit/ipmitool.git/plain/debian/enterprise-numbers.txt?h=debian/1.8.19-5


After which I could reset the ipmi ADMIN password and set the static IP address. Yeehaa!

root@debian:~# ipmitool user set name 2 ADMIN
root@debian:~# ipmitool user set password 2
Password for user 2:
Password for user 2:
Set User Password command successful (user 2)
root@debian:~# ipmitool lan set 1 ipsrc static
root@debian:~# ipmitool lan set 1 ipaddr 192.168.0.199
Setting LAN IP Address to 192.168.0.199
root@debian:~# ipmitool lan set 1 defgw ipaddr 192.168.0.1
Setting LAN Default Gateway IP to 192.168.0.1
root@debian:~# ipmitool lan print
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 PASSWORD
Auth Type Enable : Callback :
: User : MD5
: Operator : MD5
: Admin : MD5
: OEM : MD5
IP Address Source : Static Address
IP Address : 192.168.0.199
Subnet Mask : 255.255.255.0
MAC Address : 00:30:48:95:76:55
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Enabled
Gratituous ARP Intrvl : 2.0 seconds
Default Gateway IP : 192.168.0.1
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
RMCP+ Cipher Suites : 0,1,2,3,6,7,8,11,12
Cipher Suite Priv Max : uuuOXXuuOXXuOXX
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM

Wednesday, August 30, 2023

Unifi Controller: update your SSL certificate

I am running my Unifi Controller (UC) inside a Docker container on my Synology from the excellent image provided by Jacob Alberty. I needed to update my SSL certificate and it took some time for me to figure out how to do that again.

My setup is that I have a number of folders mapped from the container to a volume on my NAS. /volume1/docker/unifi contains 3 folders: cert, data and logs. The cert folder contains the necessary SSL certificate. All my files are with extension *.pem and are also in the PEM format. Although the logging later on will hint that you should use the PKCS12 format, PEM works fine as well.
The files that are needed are:

  • cert.pem  # The Certificate
  • privkey.pem # Private key for the cert
  • chain.pem # full cert chain (this includes your intermediate pasted first, and right under it your root certificate)
  1. Login to your Synology through SSH
  2. Idenfity the Docker container ID of your UC
  3. docker ps
    Output:
    CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS                                                                                                                                                                                NAMES
    78bd86bc763a        7cdd2aec3527                   "/usr/local/bin/docke"   12 months ago       Up 23 minutes       0.0.0.0:6789->6789/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:8443->8443/tcp, 0.0.0.0:8843->8843/tcp, 0.0.0.0:3478->3478/udp, 0.0.0.0:10001->10001/udp, 6789/udp, 0.0.0.0:8880->8880/tcp   unifi
  4. Create a backup of your running SSL certificate into the folder _backups
  5. cd /volume1/docker/unifi/cert && mv *.pem _backups/
  6. Copy your new files onto your Synology (/usr/syno/etc/certificate/system/default) and keep the naming as mentioned above: cert.pem, privkey.pem and chain.pem.
  7. Check the permissions and set them to user only
  8. chmod 400 *.pem*
  9. Restart the Docker container (through command line or the Synology webconsole)
  10. Monitor the progress of the container rebooting. Replace 78bd86bc763a with your actual container ID. The below highlights that a new certificate was found and it is being updated in the UC. In the case where you keep seeing the old certificate after rebooting, you have probably updated the cert in the wrong place. Try again.
  11. docker logs -f 78bd86bc763a
    Logging:
    [2019-02-09 08:08:13,866]  Starting unifi controller service.
    [2019-02-09 08:08:13,868]  WARNING: Running UniFi in insecure (root) mode
    [2019-02-09 08:12:27,477]  Exit signal received, shutting down
    [2019-02-09 08:12:33,211]  Cert directory found. Checking Certs
    [2019-02-09 08:12:33,440]  Cert has changed, updating controller...
    [2019-02-09 08:12:33,443]  Using openssl to prepare certificate...
    [2019-02-09 08:12:33,488]  Removing existing certificate from Unifi protected keystore...
    [2019-02-09 08:12:33,903]  Inserting certificate into Unifi keystore...
    Importing keystore /tmp/tmp.cuWCGLJI1p to /unifi/data/keystore...
    
    Warning:
    The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /unifi/data/keystore -destkeystore /unifi/data/keystore -deststoretype pkcs12".
    [2019-02-09 08:12:34,684]  Cleaning up temp files
    [2019-02-09 08:12:34,688]  Done!
    [2019-02-09 08:12:34,692]  Starting unifi controller service.
    [2019-02-09 08:12:34,693]  WARNING: Running UniFi in insecure (root) mode
  12. Try to browse to the UC and check the certificate in your browser. Keyboard shortcut Command+Option+J in Chrome on Mac.

Monday, August 28, 2023

Factory reset your Unifi AP or switch and adopt them to your new Unifi Controller setup with the wrong network IP

I am reinstalling my Unifi Controller on Docker and need to re-adopt my Unifi AP devices. There are several ways to do this:

  1. Hard reset your device with a staple
  2. SSH into the device and reset via a command
  3. Forget the device in the Unifi Controller (was never an option for me, but seems possible for others)

With option 1, I ran into the issue that my network is different from the default one that a Unifi AP resets to (default IP = 192.168.1.20) and I wanted a way to change that back to the fixed IP address it had before. Thanks to this article for some help. Later on, it turned out there was even a more convenient way for me to do this, see below.

  1. Reset your Unifi AP device with a staple by pressing the Reset button. When it concerns a switch, follow the special procedure which requires to unplug the power cord and UTP cables, see more information here.
  2. Provide a laptop where you can change the Ethernet IP to a fixed on e.g. 192.168.1.2 (has to be in the 192.168.1.x/24 range) and connect the laptop with your Unifi AP or switch directly through an UTP cable 
  3. SSH into the Unifi AP or switch (ssh -l ubnt 192.168.1.20) with default password ubnt
  4. Change the config file:
  5. vi /etc/udhcpc/udhcpc
    From:
    UDHCPC_FALLBACK_IP="192.168.1.20"UDHCPC_FALLBACK_NETMASK="255.255.255.0"
    Into:
    UDHCPC_FALLBACK_IP="NEW_IP"UDHCPC_FALLBACK_NETMASK="255.255.255.0"
  6. Save your file and wait
A couple of seconds later, the Unifi AP or switch will get a new IP address and you can disconnect the direct UTP cable and hook up the AP or switch again to your normal network. It should be accessible now through the Controller UI and you can adopt it again.

Unfortunately, this did not work 100% for one of my switches, so I searched for a better way.

Now, option 2 is even better and didn't require me to unplug the device and work with 2 UTP cables.
  1. Check in the Unifi Controller for the listed devices that still need to be adopted


  2. SSH into the device with you known (old) admin account and password
  3. Reset the device configuration
    set-default


  4. Wait for the device to come up again in the Unifi Controller and check it's IP address that should have changed now. Select the device and click on 'Adopt'.
  5. SSH into the device with this IP address and the default login: ubnt/ubnt.
  6. Perform an adoption via:
    ping -c 4 unifi.yourdomain.com
    mca-cli
    set-inform http://unifi.yourdomain.com:8080/inform
  7. Check in the Unifi Controller but normally it should start adopting or updating its configuration straight away!