DSC Neo Alarm Integration for Home Assistant

The biggest changes were made in the readData() interrupt service routine. Previously, in the rare event data was randomly missed or misread, the messages or the reading of the high/low bits could get out of sync. The previous code relied upon always receiving exactly 32 characters per message to know when to end a message and begin a new one. This generally always works but after a long time it could randomly get out of sync. It is suspected that other internal messages causing interrupts to happen too close to the message we are trying to capture would cause a misread. It was discovered that most often the low bits of the last space character (0x20) in the message would then end up in the highbits of the first character of the next message, desyncing all messages thereafter. When the code was previously checking and ignoring any 0x0 reads in the high bits (as all ASCII characters expected in a message are within the range of 0x20 to 0x7E)  the issue was always avoided as the 0x0 from the end of the previous message pushed into the highbits would get ignored, re-syncing the message. Although this worked well, it was not an ideal solution as it relied upon one type of desync where the low bits of the last space character (0x20) was pushed into the next message. A different last character at the end of the message that did not end in 0 could ultimately cause a desync that would not be fixable by the highbits 0 check.  Ideally we want the ESP32 to be able to correctly delimit the beginning of messages, and not rely on case specific desyncs and receiving 32 characters to know when the next message starts. 

This was solved by having the code check the internal register instructions to the LCD controller for an internal instruction indicating a new message. The controller sends 0x00000001 to clear the screen before each message is sent. The way the decoder is set up it reads this with the least significant bit first, as 0x10000000 or 0x80. We can use this internal register instruction 0x80 as a delimiter to indicate the beginning of new messages and correctly ensure the messages and the reading of the high and low 4-bit nibbles always remain in sync. 

The way this is checked is when the enable pin triggers an interrupt, the code first checks if the keypad is writing data to the LCD (RS_PIN == HIGH). However, if it is not writing messages to the LCD, then we know RS_PIN is low, and we can add an ELSE IF to further check if the interrupt is an internal register instruction (RW_PIN == LOW). We can then check if the instruction is 0x80, indicating a new message is coming, and reset the necessary variables and flags to have everything synced for the new message. We have to check the RW pin to differentiate internal register instructions (RW == LOW) from busy flag checks (RW == HIGH).

Some further optimizations have also been made to the code. One is that I have gone back to reading the data pins sequentially as it has been found more reliable. It is best advised to use the new code in its entirety. 

New code posted below and also in the Instructions section.

//ESP32 DSC NEO KEYPAD DECODER & BUTTON PUSHER BY VALDEZ - NOV/2023
//CODE IS DESIGNED FOR USE WITH ESP32-S3-WROOM-2-DEVKITC-1-N32R8V AS PER SCHEMATIC
//LIBRARIES MARKED #INCLUDE <*.h> BELOW SUCH AS PUBSUBCLIENT MAY NEED TO BE INSTALLED THROUGH YOUR LIBRARY MANAGER OF YOUR IDE
//DECODER CODE MODIFIED FROM LEONARDO MARTINS 19/01/2019 HD44780 DECODER CODE
//OTA UPDATING ENABLED AFTER FIRST FLASH
//THANKS TO USER KUSHKI7 FOR ADDITIONAL HELP WITH CODE OPTIMIZATIONS & DEBUGGING  

#include <PubSubClient.h>
#include <Wire.h>
#include <WiFi.h>
#include <ESPmDNS.h>
#include <NetworkUdp.h>
#include <ArduinoOTA.h>

//DEFINE HD44780 DECODER INPUT PINS (RS, RW, EN, DB4-7)
# define RS_PIN 2 // RS (Register Select) pin
# define RW_PIN 42 // RW (Read/Write) pin 
# define EN_PIN 1 // EN (Enable) pin
# define DB4 4
# define DB5 5
# define DB6 6
# define DB7 7

//Define Pushbutton Matrix Pins
# define X1 10
# define X2 11
# define X3 12
# define X4 13
# define X5 14
# define Y1 18
# define Y2 8
# define Y3 3
# define Y4 46
# define Y5 9

# define LED_BUILTIN 38 //define onboard LED pin

//declare functions
void IRAM_ATTR readData(void); //the ISR is to run from RAM not flash
void connect_wifi(void);
void connect_mqtt(void);
void callback(char* topic, byte* message, unsigned int length);
void buttonPush(uint8_t X, uint8_t Y, int time);
void dataSpill(void);


//********************************************************************
//******* ENTER YOUR WIFI AND MQTT CREDENTIALS MANUALLY HERE**********

//WIFI SSID/Password credentials
const char* ssid = "ENTER WIFI SSID HERE";
const char* pass = "ENTER WIFI PASSWORD HERE";

//MQTT SERVER CREDENTIALS
const char* mqtt_server = "homeassistant.local";
const int mqtt_port = 1883;
const char* mqtt_username = "ENTER MQTT USERNAME HERE";
const char* mqtt_password = "ENTER MQTT PASSWORD HERE";

//************************************************************************
//************************************************************************

//DECLARE GLOBAL VARIABLES FOR HD44780 DECODER
volatile byte data = 0; // Storage for data
volatile bool highBits = true;  // flag to indicate high or low 4 bits being read 
volatile bool dataReady = false; // Flag to indicate when data is ready to read
volatile bool highBitsInt = true; 
volatile char dataBuffer[32] = {0}; //databuffer to store the 32 char message sent to LCD
volatile byte charCount = 0; // character counter
volatile byte internalReg = 0; //internal register instruction storage

//WIRELESS SETUP
WiFiClient wifiClient; //create wifi instance wifiClinet
PubSubClient mqttClient(wifiClient); //create mqtt instance mqttClient
int countx = 0; //counter for reconnection attempts

//MAIN PROGRAM SETUP
void setup() {

  Serial.begin(115200); // Start serial monitor communication
  
  neopixelWrite(LED_BUILTIN,0,0,0); // initialize onboard led to off
  
  //initializes keypad matrix output pins
  pinMode(X1, OUTPUT);
  pinMode(X2, OUTPUT);
  pinMode(X3, OUTPUT);
  pinMode(X4, OUTPUT);
  pinMode(X5, OUTPUT);
  pinMode(Y1, OUTPUT);
  pinMode(Y2, OUTPUT);
  pinMode(Y3, OUTPUT);
  pinMode(Y4, OUTPUT);
  pinMode(Y5, OUTPUT);
  digitalWrite(X1, LOW);
  digitalWrite(X2, LOW);
  digitalWrite(X3, LOW);
  digitalWrite(X4, LOW);
  digitalWrite(X5, LOW);
  digitalWrite(Y1, LOW);
  digitalWrite(Y2, LOW);
  digitalWrite(Y3, LOW);
  digitalWrite(Y4, LOW);
  digitalWrite(Y5, LOW);

  //INITIALIZE INPUT PINS
  pinMode(RS_PIN, INPUT);
  pinMode(RW_PIN, INPUT);
  pinMode(EN_PIN, INPUT);
  pinMode(DB4, INPUT);
  pinMode(DB5, INPUT);
  pinMode(DB6, INPUT);
  pinMode(DB7, INPUT);

  // Setup the interrupt
  //note the HD44780 reads the data on the falling edge but the data is setup at the time of a rising edge which allows extra time to be captured
  attachInterrupt(digitalPinToInterrupt(EN_PIN), readData, RISING); 

  connect_wifi(); //connect wifi
  connect_mqtt(); //connect mqtt
  
  ArduinoOTA.begin(); //initialze OTA firmware updating
  
  // OTA UPLOADING DEFAULT INFORMATION (DEFAULT PASSWORD IS "admin" unless changed)
  // UNCOMMENT BELOW IF WANT TO CHANGE OTA AUTHENTICATION SETTINGS FROM DEFAULT
  // ArduinoOTA.setPort(3232);   // Port defaults to 3232
  // ArduinoOTA.setHostname("myesp32");  // Hostname defaults to esp3232-[MAC]
  // ArduinoOTA.setPassword("admin");   // No authentication by default

}

//MAIN PROGRAM LOOP
void loop() {
    
  ArduinoOTA.handle(); //prioritize OTA updater

  //reconnect wifi if disconnected  
  if (!WiFi.isConnected()) {
    connect_wifi();
  }

  //reconnect mqtt if disconnected 
  if (!mqttClient.connected()) {
    connect_mqtt();
  }
  
  mqttClient.loop(); //mqtt message check
  
  //send LCD message to MQTT if ready 
  if (dataReady){
  dataSpill(); 
  }

}

//WIFI SETUP & CONNECT FUNCTION
void connect_wifi() {
  neopixelWrite(LED_BUILTIN,0,0,0);
  delay(1000);
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);
  WiFi.begin(ssid, pass);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
  neopixelWrite(LED_BUILTIN,32,0,32);

};
    
//MQTT SETUP & CONNECT FUNCTION
void connect_mqtt() {
    mqttClient.setServer(mqtt_server, mqtt_port); 
    mqttClient.setCallback(callback);
    mqttClient.connect("esp32-DSC-Client", mqtt_username, mqtt_password);
    
    countx++; //increases counter each time it attempts a connection
    
    //attempt 15 connection attempts (30 seconds) then restart
    if (mqttClient.connected()) {
      Serial.println("connected to MQTT");
      mqttClient.subscribe("esp32/output"); //topic subscribed to for button pushing messages
      neopixelWrite(LED_BUILTIN,0,0,32); //if MQTT connects change LED to blue
      countx = 0;
    } else {
      if (countx >= 15) {
      ESP.restart();
      }
      Serial.print("MQTT connection failed, try again in 2 seconds");
 
      // Wait 2 seconds while blinking orange LED
      delay(500);
      neopixelWrite(LED_BUILTIN,0,0,0);
      delay(500);
      neopixelWrite(LED_BUILTIN,32,16,0);
      delay(500);
      neopixelWrite(LED_BUILTIN,0,0,0);
      delay(500);
      
    }
}


//CALLBACK FUNCTION TO RESPOND TO MQTT MESSAGES RECEIVED - PUSHES BUTTONS DEPENDING ON PAYLOAD RECEIVED
void callback(char* topic, byte* message, unsigned int length) {
  Serial.print("Message arrived on topic: ");
  Serial.print(topic);
  Serial.print(". Message: ");
  String messageTemp;

  //PRINT OUT THE RECEIVED MSG TO SERIAL MONITOR
  for (int i = 0; i < length; i++) {
    Serial.print((char)message[i]);
    messageTemp += (char)message[i];
   }
    Serial.println("");

  //CODE TO CALL BUTTONPUSH CONDITIONAL ON THE PAYLOAD RECEIVED   
   if (messageTemp == "1"){
      buttonPush(X1, Y1, 100);
    } else if (messageTemp == "2"){
      buttonPush(X2,Y1,100);
    } else if (messageTemp == "3"){
      buttonPush(X3,Y1,100);
    } else if (messageTemp == "4"){
      buttonPush(X1,Y2,100);
    } else if (messageTemp == "5"){
      buttonPush(X2,Y2,100);
    } else if (messageTemp == "6"){
      buttonPush(X3,Y2,100);
    } else if (messageTemp == "7"){
      buttonPush(X1,Y3,100);
    } else if (messageTemp == "8"){
      buttonPush(X2,Y3,100);
    } else if (messageTemp == "9"){
      buttonPush(X3,Y3,100);
    } else if (messageTemp == "*"){
      buttonPush(X1,Y4,100);
    } else if (messageTemp == "0"){
      buttonPush(X2,Y4,100);
    } else if (messageTemp == "#"){
      buttonPush(X3,Y4,100);
    } else if (messageTemp == "STAY"){
      buttonPush(X4,Y1,2000);
    }  else if (messageTemp == "AWAY"){
      buttonPush(X4,Y2,2000);
    } else if (messageTemp == "CHIME"){
      buttonPush(X4,Y3,2000);
    } else if (messageTemp == "F4"){ 
      buttonPush(X4,Y4,2000);
    } else if (messageTemp == "<"){
      buttonPush(X5,Y1,100);
    } else if (messageTemp == ">"){
      buttonPush(X5,Y2,100);
    } else if (messageTemp == "NIGHT"){
      buttonPush(X5,Y3, 2000);
    } else if (messageTemp == "reset"){
      ESP.restart(); // ability to restart ESP32 from MQTT with "reset" payload
    } 
   
 }


//BUTTON PUSHING FUNCTION
void buttonPush(uint8_t X, uint8_t Y, int time){
  neopixelWrite(LED_BUILTIN,32,0,32); // blink purple each time a button is pushed
  digitalWrite(X, HIGH);
  digitalWrite(Y, HIGH);
  delay(time); //length of button push
  digitalWrite(X, LOW);
  digitalWrite(Y, LOW);
  neopixelWrite(LED_BUILTIN,0,0,32);
  delay(100); //IF PUSHING MULTIPLE BUTTONS QUICKLY BY AUTOMATION NEED TO HAVE A DELAY IN BETWEEN
}


//ISR FUNCTION TO READ THE 4 bit DATA WHEN THE ENABLE PIN TRIGGERS THE INTERRUPT 
void IRAM_ATTR readData() {

   // Check that data is being written (RS == 1) 
   // store the 4-bit parallel data into the LSB (0000xxxx) of variable part
    if (digitalRead(RS_PIN) == HIGH) {
        byte part = 0; 
        part |= digitalRead(DB4); 
        part |= digitalRead(DB5) << 1;
        part |= digitalRead(DB6) << 2;
        part |= digitalRead(DB7) << 3;


    if (highBits) {   // This is the high bits (MSB) (xxxx0000) part of the 8-bit data
  
        data = part << 4; //assign and shift the 4 bit part to the most sig bit positions of the 8 bit byte

    } else {  // This is the low bits part (LSB) (0000xxxx) 

      data |= part; //copy the 4 bit part to the least sig bit end of the 8 bit byte using bitwise OR
      dataBuffer[charCount++ & 31] = data; //store the full 8 bit character data byte in the buffer array and increment the counter 
      if (!(charCount & 31)) dataReady = true, highBitsInt = true;  // once counter reaches 32 set dataready flag, reset internal register highbits flag to keep in sync. 
      //above statements uses bitwise & (X & (n-1)) similar to modulo (X % n) to repeatedly loop through the array with minimal processing; This only works when n is a power of 2.  
    }
    
    // Flip the highBits flag to alternate inputting the 4 bits from the high bits to low bits
    highBits = !highBits;

  //if controller is not writing to the LCD (RS == LOW), check if it is writing internal register instructions for the new message instruction (0x80) to keep messages in sync
  //maintain a separate highbits flag for internal register instructions so each type of write instruction can re-sync the other  
  } else if  (digitalRead(RW_PIN) == LOW) { 
        byte partInternal = 0;
        partInternal |= digitalRead(DB4);
        partInternal |= digitalRead(DB5) << 1;
        partInternal |= digitalRead(DB6) << 2;
        partInternal |= digitalRead(DB7) << 3;

    if (highBitsInt) {  
  
        internalReg = partInternal << 4; //shift the high bits into the internal register instruction storage

    } else {  
      internalReg |= partInternal; 
        if (internalReg == 0x80) { // if new message instruction detected resync the highBits flag and charCount to keep messages in sync. 
            charCount = 0; 
            highBits = true; 
        }  
    }    

    highBitsInt = !highBitsInt; 
  }

}

//FUNCTION TO WORK WITH THE STORED DATABUFFER AFTER INTERRUPT COMPLETES
//SEND THE MESSAGE TO MQTT WHEN THE DATA READY FLAG IS TRUE
void dataSpill(){

  noInterrupts(); //no interrupts must be set while manipulting volatile data that can also be changed by the ISR
  dataReady = false; //set flag to false so message is not re-sent until next message arrives
  interrupts();

  char *outputx = (char *) dataBuffer; // setup pointer to the databuffer char array to act like a string
  mqttClient.publish("esp32/alarm", outputx); //send the message to mqtt
  
    for(int i = 0; i < 32; i++){
      Serial.print(outputx[i]);                        // prints the lcd message to the serial monitor
    } 
      Serial.printf("\n");

  /*   
    for(int i = 0; i < 32; i++){             //for debugging purposes, prints the hex codes of each char to serial monitor.
      Serial.printf("%02x ", outputx[i] );  //for debugging purposes, prints the hex codes of each char to serial monitor.
      }                                    //for debugging purposes, prints the hex codes of each char to serial monitor.
      Serial.printf("\n");                //for debugging purposes, prints the hex codes of each char to serial monitor.
  */  

}

1. AP Mode has been removed as it was causing issues. Enter your wifi/mqtt credentials manually in the code before compiling. 

2. OTA updating code added so code can be flashed remotely through wifi. Will require a first flash by usb before it will work. Default password is "admin" unless changed in the code. 

3. @kushki7 provided further code optimizations to the interrupt service routine. The ISR now reads all 4 pins simultaneously using a register increasing the read speed. Bitwise & is used like a modulo to loop through the dataBuffer and set the dataReady flag as is much faster than a for loop. It takes minimal processing to compare two bytes with & which is why it increases speed. A further explanation is in the code comments.

4. With @kushki7 optimizations, it appears the integration is fast enough to operate the interrupt on a falling edge and work. The actual HD4470 controller latches the DB4-7 data on a falling edge of the enable pin. However, the data is set up at the time the pin goes high, and triggering the interrupt on the rising edge adds a buffer of safety in case the reading of DB4-7 is delayed. Code is left on a rising edge for this reason, but can make that change if you want. 

5. The interrupt function is set to run off ESP32s RAM instead of Flash using IRAM_ATTR. This is necessary for a properly functioning ISR.

6. Changes to the dataSpill() function were also made. noInterrupts() must wrap any volatile variables changed in dataspill to prevent problems with the ISR trying to change them at the same time. In our case only dataReady flag needs to be wrapped with noInterrupts(). When it was not previously, synchronization errors were happening shifting the messages high/low bits over and out of sync. Lastly the string pointer to the volatile dataBuffer was replaced with a local char array with an extra null character at the end. The dataBuffer is copied into this array and the extra null at the end allows proper handling and sending it to MQTT as a recognized string. 

7. A MQTT reset message has been included in case you ever need to remotely reboot the ESP32. If you send "reset" to the button pushing topic "esp32/output" the ESP32 will reboot itself.  

8. Code was cleaned up in general and some variables and instances redefined to make things more clear so best to use the new code in its entirety

User @kushki7 discovered some bugs when building my design for his own system. After a long debugging and re-programming session with him we have fixed these issues. 

We discovered that the timing of the reading the data from the bus is critical and has to happen as fast as possible. We also found some code bugs in the interrupt routine that were causing issues. 

For the timing issue, we removed the need to check the RW pin as the keypad never reads the data from the LCD controller RAM and so checking when the RW is low when RS is high is not necessary as RW will always be low when RS is high. It just isn't necessary for capturing the write data and the added complexity of checking the RW line adds processing to the interrupt where we want it the fastest as possible - right before reading the data lines.   An updated PCB design may come sometime in the future to reflect this.

The code in instructions section corrects these bugs found, and does a few other optimizations for reading the data. 

Power management - @kushki7 also had a greater strain on power running the ESP32 off his keypads 3.3V+. It ran, but he felt it was pulling a lot of power from the keypad causing his display to slow down and dim and decided to run his ESP32 off an external 5V USB power source. I never had this issue, and it likely depends on each individuals alarm panel setup and power demands. If you want to run the ESP32 off an external power source, disconnect the V+ line from the PCB to the keypad. Gnd must stay connected between the keypad and PCB or the data signals will not make any sense to the ESP32 (the signals are voltages or potential differences between Gnd so the ESP32 and keypad must share a common Gnd).