The biggest changes were made in the readData() interrupt service routine. Previously, in the rare event data was randomly missed or misread, the messages or the reading of the high/low bits could get out of sync. The previous code relied upon always receiving exactly 32 characters per message to know when to end a message and begin a new one. This generally always works but after a long time it could randomly get out of sync. It is suspected that other internal messages causing interrupts to happen too close to the message we are trying to capture would cause a misread. It was discovered that most often the low bits of the last space character (0x20) in the message would then end up in the highbits of the first character of the next message, desyncing all messages thereafter. When the code was previously checking and ignoring any 0x0 reads in the high bits (as all ASCII characters expected in a message are within the range of 0x20 to 0x7E)  the issue was always avoided as the 0x0 from the end of the previous message pushed into the highbits would get ignored, re-syncing the message. Although this worked well, it was not an ideal solution as it relied upon one type of desync where the low bits of the last space character (0x20) was pushed into the next message. A different last character at the end of the message that did not end in 0 could ultimately cause a desync that would not be fixable by the highbits 0 check.  Ideally we want the ESP32 to be able to correctly delimit the beginning of messages, and not rely on case specific desyncs and receiving 32 characters to know when the next message starts. 

This was solved by having the code check the internal register instructions to the LCD controller for an internal instruction indicating a new message. The controller sends 0x00000001 to clear the screen before each message is sent. The way the decoder is set up it reads this with the least significant bit first, as 0x10000000 or 0x80. We can use this internal register instruction 0x80 as a delimiter to indicate the beginning of new messages and correctly ensure the messages and the reading of the high and low 4-bit nibbles always remain in sync. 

The way this is checked is when the enable pin triggers an interrupt, the code first checks if the keypad is writing data to the LCD (RS_PIN == HIGH). However, if it is not writing messages to the LCD, then we know RS_PIN is low, and we can add an ELSE IF to further check if the interrupt is an internal register instruction (RW_PIN == LOW). We can then check if the instruction is 0x80, indicating a new message is coming, and reset the necessary variables and flags to have everything synced for the new message. We have to check the RW pin to differentiate internal register instructions (RW == LOW) from busy flag checks (RW == HIGH).

Some further optimizations have also been made to the code. One is that I have gone back to reading the data pins sequentially as it has been found more reliable. It is best advised to use the new code in its entirety. 

New code posted below and also in the Instructions section.

//ESP32 DSC NEO KEYPAD DECODER & BUTTON PUSHER BY VALDEZ - NOV/2023
//CODE IS DESIGNED FOR USE WITH ESP32-S3-WROOM-2-DEVKITC-1-N32R8V AS PER SCHEMATIC
//LIBRARIES MARKED #INCLUDE <*.h> BELOW SUCH AS PUBSUBCLIENT MAY NEED TO BE INSTALLED THROUGH YOUR LIBRARY MANAGER OF YOUR IDE
//DECODER CODE MODIFIED FROM LEONARDO MARTINS 19/01/2019 HD44780 DECODER CODE
//OTA UPDATING ENABLED AFTER FIRST FLASH
//THANKS TO USER KUSHKI7 FOR ADDITIONAL HELP WITH CODE OPTIMIZATIONS & DEBUGGING  

#include <PubSubClient.h>
#include <Wire.h>
#include <WiFi.h>
#include <ESPmDNS.h>
#include <NetworkUdp.h>
#include <ArduinoOTA.h>

//DEFINE HD44780 DECODER INPUT PINS (RS, RW, EN, DB4-7)
# define RS_PIN 2 // RS (Register Select) pin
# define RW_PIN 42 // RW (Read/Write) pin 
# define EN_PIN 1 // EN (Enable) pin
# define DB4 4
# define DB5 5
# define DB6 6
# define DB7 7

//Define Pushbutton Matrix Pins
# define X1 10
# define X2 11
# define X3 12
# define X4 13
# define X5 14
# define Y1 18
# define Y2 8
# define Y3 3
# define Y4 46
# define Y5 9

# define LED_BUILTIN 38 //define onboard LED pin

//declare functions
void IRAM_ATTR readData(void); //the ISR is to run from RAM not flash
void connect_wifi(void);
void connect_mqtt(void);
void callback(char* topic, byte* message, unsigned int length);
void buttonPush(uint8_t X, uint8_t Y, int time);
void dataSpill(void);


//********************************************************************
//******* ENTER YOUR WIFI AND MQTT CREDENTIALS MANUALLY HERE**********

//WIFI SSID/Password credentials
const char* ssid = "ENTER WIFI SSID HERE";
const char* pass = "ENTER WIFI PASSWORD HERE";

//MQTT SERVER CREDENTIALS
const char* mqtt_server = "homeassistant.local";
const int mqtt_port = 1883;
const char* mqtt_username = "ENTER MQTT USERNAME HERE";
const char* mqtt_password = "ENTER MQTT PASSWORD HERE";

//************************************************************************
//************************************************************************

//DECLARE GLOBAL VARIABLES FOR HD44780 DECODER
volatile byte data = 0; // Storage for data
volatile bool highBits = true;  // flag to indicate high or low 4 bits being read 
volatile bool dataReady = false; // Flag to indicate when data is ready to read
volatile bool highBitsInt = true; 
volatile char dataBuffer[32] = {0}; //databuffer to store the 32 char message sent to LCD
volatile byte charCount = 0; // character counter
volatile byte internalReg = 0; //internal register instruction storage

//WIRELESS SETUP
WiFiClient wifiClient; //create wifi instance wifiClinet
PubSubClient mqttClient(wifiClient); //create mqtt instance mqttClient
int countx = 0; //counter for reconnection attempts

//MAIN PROGRAM SETUP
void setup() {

  Serial.begin(115200); // Start serial monitor communication
  
  neopixelWrite(LED_BUILTIN,0,0,0); // initialize onboard led to off
  
  //initializes keypad matrix output pins
  pinMode(X1, OUTPUT);
  pinMode(X2, OUTPUT);
  pinMode(X3, OUTPUT);
  pinMode(X4, OUTPUT);
  pinMode(X5, OUTPUT);
  pinMode(Y1, OUTPUT);
  pinMode(Y2, OUTPUT);
  pinMode(Y3, OUTPUT);
  pinMode(Y4, OUTPUT);
  pinMode(Y5, OUTPUT);
  digitalWrite(X1, LOW);
  digitalWrite(X2, LOW);
  digitalWrite(X3, LOW);
  digitalWrite(X4, LOW);
  digitalWrite(X5, LOW);
  digitalWrite(Y1, LOW);
  digitalWrite(Y2, LOW);
  digitalWrite(Y3, LOW);
  digitalWrite(Y4, LOW);
  digitalWrite(Y5, LOW);

  //INITIALIZE INPUT PINS
  pinMode(RS_PIN, INPUT);
  pinMode(RW_PIN, INPUT);
  pinMode(EN_PIN, INPUT);
  pinMode(DB4, INPUT);
  pinMode(DB5, INPUT);
  pinMode(DB6, INPUT);
  pinMode(DB7, INPUT);

  // Setup the interrupt
  //note the HD44780 reads the data on the falling edge but the data is setup at the time of a rising edge which allows extra time to be captured
  attachInterrupt(digitalPinToInterrupt(EN_PIN), readData, RISING); 

  connect_wifi(); //connect wifi
  connect_mqtt(); //connect mqtt
  
  ArduinoOTA.begin(); //initialze OTA firmware updating
  
  // OTA UPLOADING DEFAULT INFORMATION (DEFAULT PASSWORD IS "admin" unless changed)
  // UNCOMMENT BELOW IF WANT TO CHANGE OTA AUTHENTICATION SETTINGS FROM DEFAULT
  // ArduinoOTA.setPort(3232);   // Port defaults to 3232
  // ArduinoOTA.setHostname("myesp32");  // Hostname defaults to esp3232-[MAC]
  // ArduinoOTA.setPassword("admin");   // No authentication by default

}

//MAIN PROGRAM LOOP
void loop() {
    
  ArduinoOTA.handle(); //prioritize OTA updater

  //reconnect wifi if disconnected  
  if (!WiFi.isConnected()) {
    connect_wifi();
  }

  //reconnect mqtt if disconnected 
  if (!mqttClient.connected()) {
    connect_mqtt();
  }
  
  mqttClient.loop(); //mqtt message check
  
  //send LCD message to MQTT if ready 
  if (dataReady){
  dataSpill(); 
  }

}

//WIFI SETUP & CONNECT FUNCTION
void connect_wifi() {
  neopixelWrite(LED_BUILTIN,0,0,0);
  delay(1000);
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);
  WiFi.begin(ssid, pass);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
  neopixelWrite(LED_BUILTIN,32,0,32);

};
    
//MQTT SETUP & CONNECT FUNCTION
void connect_mqtt() {
    mqttClient.setServer(mqtt_server, mqtt_port); 
    mqttClient.setCallback(callback);
    mqttClient.connect("esp32-DSC-Client", mqtt_username, mqtt_password);
    
    countx++; //increases counter each time it attempts a connection
    
    //attempt 15 connection attempts (30 seconds) then restart
    if (mqttClient.connected()) {
      Serial.println("connected to MQTT");
      mqttClient.subscribe("esp32/output"); //topic subscribed to for button pushing messages
      neopixelWrite(LED_BUILTIN,0,0,32); //if MQTT connects change LED to blue
      countx = 0;
    } else {
      if (countx >= 15) {
      ESP.restart();
      }
      Serial.print("MQTT connection failed, try again in 2 seconds");
 
      // Wait 2 seconds while blinking orange LED
      delay(500);
      neopixelWrite(LED_BUILTIN,0,0,0);
      delay(500);
      neopixelWrite(LED_BUILTIN,32,16,0);
      delay(500);
      neopixelWrite(LED_BUILTIN,0,0,0);
      delay(500);
      
    }
}


//CALLBACK FUNCTION TO RESPOND TO MQTT MESSAGES RECEIVED - PUSHES BUTTONS DEPENDING ON PAYLOAD RECEIVED
void callback(char* topic, byte* message, unsigned int length) {
  Serial.print("Message arrived on topic: ");
  Serial.print(topic);
  Serial.print(". Message: ");
  String messageTemp;

  //PRINT OUT THE RECEIVED MSG TO SERIAL MONITOR
  for (int i = 0; i < length; i++) {
    Serial.print((char)message[i]);
    messageTemp += (char)message[i];
   }
    Serial.println("");

  //CODE TO CALL BUTTONPUSH CONDITIONAL ON THE PAYLOAD RECEIVED   
   if (messageTemp == "1"){
      buttonPush(X1, Y1, 100);
    } else if (messageTemp == "2"){
      buttonPush(X2,Y1,100);
    } else if (messageTemp == "3"){
      buttonPush(X3,Y1,100);
    } else if (messageTemp == "4"){
      buttonPush(X1,Y2,100);
    } else if (messageTemp == "5"){
      buttonPush(X2,Y2,100);
    } else if (messageTemp == "6"){
      buttonPush(X3,Y2,100);
    } else if (messageTemp == "7"){
      buttonPush(X1,Y3,100);
    } else if (messageTemp == "8"){
      buttonPush(X2,Y3,100);
    } else if (messageTemp == "9"){
      buttonPush(X3,Y3,100);
    } else if (messageTemp == "*"){
      buttonPush(X1,Y4,100);
    } else if (messageTemp == "0"){
      buttonPush(X2,Y4,100);
    } else if (messageTemp == "#"){
      buttonPush(X3,Y4,100);
    } else if (messageTemp == "STAY"){
      buttonPush(X4,Y1,2000);
    }  else if (messageTemp == "AWAY"){
      buttonPush(X4,Y2,2000);
    } else if (messageTemp == "CHIME"){
      buttonPush(X4,Y3,2000);
    } else if (messageTemp == "F4"){ 
      buttonPush(X4,Y4,2000);
    } else if (messageTemp == "<"){
      buttonPush(X5,Y1,100);
    } else if (messageTemp == ">"){
      buttonPush(X5,Y2,100);
    } else if (messageTemp == "NIGHT"){
      buttonPush(X5,Y3, 2000);
    } else if (messageTemp == "reset"){
      ESP.restart(); // ability to restart ESP32 from MQTT with "reset" payload
    } 
   
 }


//BUTTON PUSHING FUNCTION
void buttonPush(uint8_t X, uint8_t Y, int time){
  neopixelWrite(LED_BUILTIN,32,0,32); // blink purple each time a button is pushed
  digitalWrite(X, HIGH);
  digitalWrite(Y, HIGH);
  delay(time); //length of button push
  digitalWrite(X, LOW);
  digitalWrite(Y, LOW);
  neopixelWrite(LED_BUILTIN,0,0,32);
  delay(100); //IF PUSHING MULTIPLE BUTTONS QUICKLY BY AUTOMATION NEED TO HAVE A DELAY IN BETWEEN
}


//ISR FUNCTION TO READ THE 4 bit DATA WHEN THE ENABLE PIN TRIGGERS THE INTERRUPT 
void IRAM_ATTR readData() {

   // Check that data is being written (RS == 1) 
   // store the 4-bit parallel data into the LSB (0000xxxx) of variable part
    if (digitalRead(RS_PIN) == HIGH) {
        byte part = 0; 
        part |= digitalRead(DB4); 
        part |= digitalRead(DB5) << 1;
        part |= digitalRead(DB6) << 2;
        part |= digitalRead(DB7) << 3;


    if (highBits) {   // This is the high bits (MSB) (xxxx0000) part of the 8-bit data
  
        data = part << 4; //assign and shift the 4 bit part to the most sig bit positions of the 8 bit byte

    } else {  // This is the low bits part (LSB) (0000xxxx) 

      data |= part; //copy the 4 bit part to the least sig bit end of the 8 bit byte using bitwise OR
      dataBuffer[charCount++ & 31] = data; //store the full 8 bit character data byte in the buffer array and increment the counter 
      if (!(charCount & 31)) dataReady = true, highBitsInt = true;  // once counter reaches 32 set dataready flag, reset internal register highbits flag to keep in sync. 
      //above statements uses bitwise & (X & (n-1)) similar to modulo (X % n) to repeatedly loop through the array with minimal processing; This only works when n is a power of 2.  
    }
    
    // Flip the highBits flag to alternate inputting the 4 bits from the high bits to low bits
    highBits = !highBits;

  //if controller is not writing to the LCD (RS == LOW), check if it is writing internal register instructions for the new message instruction (0x80) to keep messages in sync
  //maintain a separate highbits flag for internal register instructions so each type of write instruction can re-sync the other  
  } else if  (digitalRead(RW_PIN) == LOW) { 
        byte partInternal = 0;
        partInternal |= digitalRead(DB4);
        partInternal |= digitalRead(DB5) << 1;
        partInternal |= digitalRead(DB6) << 2;
        partInternal |= digitalRead(DB7) << 3;

    if (highBitsInt) {  
  
        internalReg = partInternal << 4; //shift the high bits into the internal register instruction storage

    } else {  
      internalReg |= partInternal; 
        if (internalReg == 0x80) { // if new message instruction detected resync the highBits flag and charCount to keep messages in sync. 
            charCount = 0; 
            highBits = true; 
        }  
    }    

    highBitsInt = !highBitsInt; 
  }

}

//FUNCTION TO WORK WITH THE STORED DATABUFFER AFTER INTERRUPT COMPLETES
//SEND THE MESSAGE TO MQTT WHEN THE DATA READY FLAG IS TRUE
void dataSpill(){

  noInterrupts(); //no interrupts must be set while manipulting volatile data that can also be changed by the ISR
  dataReady = false; //set flag to false so message is not re-sent until next message arrives
  interrupts();

  char *outputx = (char *) dataBuffer; // setup pointer to the databuffer char array to act like a string
  mqttClient.publish("esp32/alarm", outputx); //send the message to mqtt
  
    for(int i = 0; i < 32; i++){
      Serial.print(outputx[i]);                        // prints the lcd message to the serial monitor
    } 
      Serial.printf("\n");

  /*   
    for(int i = 0; i < 32; i++){             //for debugging purposes, prints the hex codes of each char to serial monitor.
      Serial.printf("%02x ", outputx[i] );  //for debugging purposes, prints the hex codes of each char to serial monitor.
      }                                    //for debugging purposes, prints the hex codes of each char to serial monitor.
      Serial.printf("\n");                //for debugging purposes, prints the hex codes of each char to serial monitor.
  */  

}