It has been almost 3 years now since I built
Thoughts — the microblogging
platform and I'd say it's been a success. Over the years I've tweaked it a bit
for cost and usability, integrating some quality of life features into the
posting interface (I swapped out my own editor for Trix),
changing the attachment processing pipeline to support videos, and making
a rudimentary swing at rich link previews (that does need a revisit). As
it stands today it costs me between 9¢ and 11¢ per month to run.
One of the things that landed in the ~/TODO after using it for a while
was to add the ability for me to reply to an existing Thought. Sometimes
I throw something out there and later on want to follow up to it but there
wasn't a great way to do that. Thoughts are generally speaking immutable
and independent — and while this is an intended feature it lead to
situations like this.
Alerting in Grafana is so trash that I'm seriously about to write Nagios/Icinga plugins to just watch values from InfluxDB. They should just throw the alert system in the trash and let some software fit for purpose do the work.
Followed later in the feed, and potentially on a different page by...
So yeah, after whining about how bad Grafana's alerting system is and how I was about to rage-write some Icinga checks... Yeah.. I did it. I replaced a whole suite of rules with a 262 line Python script.
I also went and replaced a bunch of Azure Function Monitor rules and alerts with a few calls to check_http.
I've tried several things in various Thoughts to try to make it obvious that one is a reply to another, usually involving linking back but netnews and e-mail have had threading since the 1980s so why the hell don't I?
It is useful at this point to remind ourselves of the original design requirements of the Thoughts system as those have not changed. In fact they informed several decisions I had to make along the way.
- My data MUST be my own.
- URLs do NOT change.
- URLs SHOULD make sense.
- Posting MUST be easy.
Happily the first is easy as the system that periodically renders Thoughts to files on disk isn't going away, it'll just have to be updated to support whatever API changes come along. The next two were accomplished by simply taking the ID of the parent Thought and thinking of that as a thread ID, then coming up with a way to redirect child Thought IDs to the parent thread ID. The final consideration is posting and that was largely solved by thinking about how I wanted to handle threading in general. Since Thoughts are meant to be idempotent we can take inspiration from the way netnews and e-mail work. Since we can assume that the poster doesn't have access to change the data that is already there all the information needed to post a reply must be a part of the existing Thought, hence the use of the ID attribute which is the time of the post in POSIX epoch seconds format.
If you want to get really in depth, I kept the notes I took while I was designing and building the changes.
Thought Changes
Again, taking inspiration from decades old Internet technology I chose to
create an in_reply_to field in the Thought table storage. All the
threading is constructed from this by the new
Thought object
abstraction that I created. ThoughtService
was then converted to leverage the Thought object instead of directly
interacting with the Entity objects from the Table Service.
At first generating these relationships took a long time and so several
iterations and refactors were required to produce something performant
enough to happen on every call. Happily we got there, reducing the call
time from around 3 seconds for the 25 entry RSS feed to about 800 ms.
Further performance was gained by caching the RSS feed as it's the same
for everyone and only needs to be regenerated when I post something new.
I was able to leverage Blob metadata to provide a simple
cache layer
that I was able to plumb into the
RSS function.
Timeline Changes
Given it was written in Python, the RSS timeline was the easiest to update.
The function was taught about the new API and then given the ability to
render the HTML needed to nest children inside <blockquote>
elements. The Thought object was given a UUID property that is generated
deterministically from the thread ID and the highest ID of any child thought.
This way it changes when the Thought is replied to. Most RSS readers
use on the GUID tag in the feed XML as the key to store the read/unread
state. This way when I reply to a Thought it will create a new item in
the feed. Using the same API the method was adapted to JavaScript for
the web display portion. It's a little more complex because there are
several pieces interacting, between the API client, the logic on the
page, and the logic inside the various
Web Components
In the end what you see is mostly driven from
thoughts.js.
Single Thought Display to Thread Display Changes
One of the trickiest parts was deciding how to handle the individual Thought
display. It seems useless to keep the old single Thought display mechanism
as it would destroy the entire purpose of showing the context of related
Thoughts. I did want to simplify the URLs a bit so I didn't have to render
what would end up being duplicates over and over for parents and their
children, so I decided to change the format from /thoughts/<id>.html to
/thoughts/<thread id>.html#id<child id>. This means that the URL
https://www.going-flying.com/thoughts/1658703307.html
becomes
https://www.going-flying.com/thoughts/1658613442.html#id1658703307.
It was simple enough to teach the JavaScript about this new format but I
had to devise a method to make sure the legacy URLs all work. This is
where Apache's mod_rewrite comes to the rescue. I already lean on it
for the single Thought display, internally serving a file called single.html
for any un-rendered Thought. The Apache configuration for that looks like
this.
RewriteCond %{REQUEST_URI} '^/thoughts' [NC]
RewriteCond /var/www/going-flying.com/%{REQUEST_URI} !-f
RewriteRule '^/thoughts/(\d+)\.html' \
'/thoughts/single.html' [L]
This fragment says if a request comes in for something in /thoughts that
looks like <a bunch of numbers>.html check for it on disk in
/var/www/going-flying.com/thoughts/<a bunch of numbers>.html. If it
exists serve that — otherwise internally rewrite the request to
/thoughts/single.html and serve that. The client is none the wiser (that
is the [L] bit) which is important because the JavaScript uses
window.location
to figure out what Thought to load.
I updated the static Thought builder to write out a map file that points each child Thought ID back to it's parent ID. I then added the following prior to this bit which causes Apache to redirect the browser to the new format URL if a mapping is found in the file.
RewriteMap thoughts "txt:/var/www/going-flying.com/thoughts/thread.txt"
RewriteCond %{REQUEST_URI} '^/thoughts' [NC]
RewriteCond ${thoughts:$1} '\d+'
RewriteRule '^/thoughts/(\d+)\.html' \
'/thoughts/${thoughts:$1}.html#id$1' [R=301,NE]
The Apache RewriteMap documentation may be helpful in explaining what is going on here.
The rest was JavaScript and CSS changes to make the display pretty and was mostly done as part of getting the timeline view working. The nice part of the Web Components is that most of the code is shared.
Posting Interface Changes
This ended up being about the easiest of all the different bits and pieces. The JavaScript function was just modified to know about a URL hash parameter.
async function post(evt)
{
[snip]
// Try to get the In-Reply-To from the location.
const hash = window.location.hash.slice(1).split('=');
let in_reply_to = null;
if (hash[0].toLowerCase() == 'in-reply-to') {
in_reply_to = hash[1];
}
[snip]
/* According to the table at:
* https://docs.microsoft.com/en-us/rest/api/storageservices/payload-format-for-table-service-operations
* The JSON type for Edm.Int64 is... STRING!
*/
const row = {
'PartitionKey': 'thought',
'RowKey': id.toString(),
'id@odata.type': 'Edm.Int64',
'id': id.toString(),
'message': nl2br(msg)
};
if (attachments) {
row['attachment'] = JSON.stringify(attachments);
}
if (link) {
row['link'] = JSON.stringify(link);
}
if (in_reply_to) {
row['in_reply_to@odata.type'] = 'Edm.Int64';
row['in_reply_to'] = in_reply_to;
}
await connections.table.insert(row);
In my view of Thoughts I get an extra control that provides a link to the posting interface with the hash parameter set, so I can simply browse to a Thought that already exists and reply to it.
Overall
To look at the amount of changes that this took it is useful to think of the system as two parts. Vociferate is the Azure Functions, Table Storage and Blob Storage bits (mostly the Function as far as where code lives) and Thoughts is the web application parts that live here on going-flying.com.
vociferate changes
.funcignore | 1 +
.gitignore | 2 +
Makefile | 5 +-
README.txt | 13 +-
TODO | 4 -
cli/add-in-reply-to.py | 2 +-
lib/__init__.py | 5 +-
lib/blobcache.py | 103 +++++++++++
lib/decorators.py | 31 ++++
lib/thought.py | 235 +++++++++++++++++++++++++
lib/thoughtservice.py | 339 ++++++++++++++----------------------
list/__init__.py | 9 +-
lib/oembed.py => parent/__init__.py | 55 ++++--
parent/function.json | 17 ++
pypeteer/__init__.py | 9 +-
requirements.txt | 7 +-
resolver/__init__.py | 18 +-
rss/__init__.py | 114 ++++++++++--
tests/__init__.py | 1 +
tests/test_thought.py | 96 ++++++++++
tests/test_thoughtservice.py | 69 ++++++++
thought.html | 2 +-
thoughts/__init__.py | 25 ++-
thumbnailer/__init__.py | 21 +--
24 files changed, 889 insertions(+), 294 deletions(-)
Thoughts changes
js/classes.js | 62 +++++++++++++++++++++--------------------------
js/lave.js | 6 +----
staticThought.py | 49 +++++++++++++++++++++++++++++++------
templates/components/large.html | 20 +++++++++++----
templates/components/small.html | 8 +++++-
templates/thoughts/index.html | 2 +-
templates/thoughts/onethought.html | 10 +-------
thoughts/css/thoughts.css | 35 +++++++++++++++++++++------
thoughts/js/components.js | 164 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------------------------------
thoughts/js/thoughts.js | 184 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------------------------
10 files changed, 373 insertions(+), 167 deletions(-)
Most of the heavy lifting was quite happily done in Python on the Function
end of the equation. There is no reason you couldn't do it the other way
around and just return JSON with the in-reply-to properties in them and
do your threading in the JavaScript if you are more comfortable with it.
Hell if you really wanted to Functions supports JavaScript as a language so
you could eschew Python entirely if you happened to be some sort of masochist.
Conclusion
I pushed the changes live on the 1st of November and I've been happy with it so far. I went back and tagged a few threads and have posted a few replies to Thoughts with the new interface and all in all it seems to work pretty well.