Blatta 9988: deduplication by message hash

Version 9988 includes:

- Overhaul of deduplication algo: Previously timestamps were used to determine whether or not a message was unique. Now a hash of the message is being used. WARNING: Messages from stations that have not yet applied 9988 may not be detected as dupes by 9988 due to the switch from timestamps to hashes. There were multiple issues in previous versions causing messages with identical irc message payloads to appear to be different when hashed.

- Fix for overzealous rubbish message broadcasts.

Additionally, I have started publishing my station's debug log in an effort to aid Blatta testers.

9988-hash-dedup.vpatch

9988-hash-dedup.vpatch.thimbronion.sig

6 Responses to “Blatta 9988: deduplication by message hash”

  1. thimbronion says:

    Traceback (most recent call last):
    File "./blatta", line 145, in
    main(sys.argv)
    File "./blatta", line 140, in main
    server.start()
    File "/home/bmg/applications/blatta/lib/server.py", line 292, in start
    self.clients[x].socket_readable_notification()
    File "/home/bmg/applications/blatta/lib/client.py", line 615, in socket_readable_notification
    self.__parse_read_buffer()
    File "/home/bmg/applications/blatta/lib/client.py", line 115, in __parse_read_buffer
    self.__handle_command(command, arguments)
    File "/home/bmg/applications/blatta/lib/client.py", line 598, in __command_handler
    handler_table[command]()
    File "/home/bmg/applications/blatta/lib/client.py", line 352, in notice_and_privmsg_handler
    }, self.server))
    File "/home/bmg/applications/blatta/lib/server.py", line 209, in peer_message
    message_bytes = self.infosec.get_message_bytes(message, peer)
    File "/home/bmg/applications/blatta/lib/infosec.py", line 60, in get_message_bytes
    self_chain = self.server.state.get_last_message_hash(message.speaker, peer.peer_id)
    AttributeError: 'NoneType' object has no attribute 'peer_id'

  2. thimbronion says:

    speaker, body need to be padded with 0x0 not " ".

  3. thimbronion says:

    Messages should be packed ONCE prior to encryption.

  4. thimbronion says:

    Dupe messages should be logged, not silently disappeared. Bounce count should be logged.

  5. thimbronion says:

    http://logs.nosuchlabs.com/log/asciilifeform/2021-11-17#1066496

    File "./blatta", line 145, in
    main(sys.argv)
    File "./blatta", line 140, in main
    server.start()
    File "/home/Lambert/alcuin/blatta-9988/blatta/lib/server.py", line 292, in start
    self.clients[x].socket_readable_notification()
    File "/home/Lambert/alcuin/blatta-9988/blatta/lib/client.py", line 615, in socket_readable_notification
    self.__parse_read_buffer()
    File "/home/Lambert/alcuin/blatta-9988/blatta/lib/client.py", line 115, in __parse_read_buffer
    self.__handle_command(command, arguments)
    File "/home/Lambert/alcuin/blatta-9988/blatta/lib/client.py", line 598, in __command_handler
    handler_table[command]()
    File "/home/Lambert/alcuin/blatta-9988/blatta/lib/client.py", line 352, in notice_and_privmsg_handler
    }, self.server))
    File "/home/Lambert/alcuin/blatta-9988/blatta/lib/server.py", line 209, in peer_message
    message_bytes = self.infosec.get_message_bytes(message, peer)
    File "/home/Lambert/alcuin/blatta-9988/blatta/lib/infosec.py", line 60, in get_message_bytes
    self_chain = self.server.state.get_last_message_hash(message.speaker, peer.peer_id)
    AttributeError: 'NoneType' object has no attribute 'peer_id'

  6. thimbronion says:

    Traceback (most recent call last):
    File "/home/awt/.pyenv/versions/2.7.14/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
    File "/home/awt/.pyenv/versions/2.7.14/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
    File "/home/awt/src/alcuin/lib/station.py", line 113, in check_embargo_queue
    self.check_for_immediate_messages()
    File "/home/awt/src/alcuin/lib/station.py", line 137, in check_for_immediate_messages
    self.rebroadcast(message)
    File "/home/awt/src/alcuin/lib/station.py", line 185, in rebroadcast
    self.infosec.message(message)
    File "/home/awt/src/alcuin/lib/infosec.py", line 67, in message
    self.state.log(message.speaker, message_bytes, target_peer)
    File "/home/awt/src/alcuin/lib/state.py", line 122, in log
    (handle, peer_id, buffer(message_bytes)))
    ProgrammingError: Recursive use of cursors not allowed.

Leave a Reply