Adding Videos To Thoughts_

June 25, 2021 @13:00

When I originally built Thoughts, I only supported images as attachments. I used the accept parameter of the file input tag in the posting interface to prevent myself from trying to attach anything other than images, leaving the pipeline simply unimplemented. This made it easy to add.

kitsune@11:49:09 vociferate >git stat 6ce39e9..4b0c6c7
 Caviar Dreams Bold.ttf  | Bin 0 -> 55988 bytes
 playbutton.png          | Bin 0 -> 3774 bytes
 requirements.txt        |   1 +
 thumbnailer/__init__.py | 204 +++++++++++++++++++++++++++++++++++++++++-------
 4 files changed, 178 insertions(+), 27 deletions(-)
 create mode 100644 Caviar Dreams Bold.ttf
 create mode 100644 playbutton.png

Most of the work was updating the front-end to handle displaying the animated thumbnails / video.

kitsune@11:50:21 going-flying.com >git stat b47d53b...f34a272
 Makefile                               |   3 +-
 js/classes.js                          |  11 +-
 js/lave.js                             |  53 +-
 templates/components/large.html        |  47 +-
 templates/thoughts/index.html          |   5 +-
 templates/thoughts/single.html         |   3 +
 thoughts/css/embed.css                 |   1 +
 thoughts/css/glightbox/glightbox.css   | 938 +++++++++++++++++++++++++++++++++
 thoughts/js/components.js              | 134 ++++-
 thoughts/js/glightbox/glightbox.min.js |   1 +
 thoughts/js/thoughts.js                | 101 ++--
 11 files changed, 1177 insertions(+), 120 deletions(-)
 create mode 100644 thoughts/css/glightbox/glightbox.css
 create mode 100644 thoughts/js/glightbox/glightbox.min.js

Video Processing on Azure Functions

Similar to the image processing, I create a thumbnail of the video once it hits the Azure Blob container, I also create a short animated gif preview. Both of these operations take place in the thumbnailer Function and leverage Pillow and PyAV to do all the heavy lifting.

Static Poster

Creating the poster is a pretty standard Pillow pattern, I take the image, paste a play icon on top of it, then I draw some text with the video duration and write out a JPEG.

def generate_poster(xy: tuple, fn: str, img: Image, duration: int):
        ''' Create the poster image by compositing a frame with a play symbol
        and the duration.
        '''
        FONT_SIZE = 32
        PAD = 5
        TEXT_OFFSET = 10

        new_img = Image.new('RGBA', xy)
        new_img.paste(img, box=(0, 0))

        with Image.open('/home/site/wwwroot/playbutton.png') as play_img:
                x = (xy[0] - play_img.size[0]) // 2
                y = (xy[1] - play_img.size[1]) // 2
                play_img = play_img.convert('RGBA')
                new_img.paste(play_img, box=(x, y), mask=play_img)

        font = ImageFont.truetype(
                '/home/site/wwwroot/Caviar Dreams Bold.ttf',
                FONT_SIZE
        )
        draw = ImageDraw.Draw(new_img)
        box = draw.textbbox(
                (xy[0] - TEXT_OFFSET, xy[1] - TEXT_OFFSET),
                duration,
                font,
                'rd'
        )

        box = (box[0] - PAD, box[1] - PAD, box[2] + PAD, box[3] + PAD)
        draw.rectangle(box, '#000000')
        draw.text(
                (xy[0] - TEXT_OFFSET, xy[1] - TEXT_OFFSET),
                duration,
                '#00DD00',
                font,
                'rd'
        )

        outBytes = io.BytesIO()
        new_img = new_img.convert('RGB')
                new_img.save(
                outBytes,
                format='JPEG',
                optimize=True,
                progressive=True,
                quality=95
        )
        writeBlob(fn, 'image/jpeg', outBytes.getvalue())
        logging.info(f'Created poster {fn}')

Animated Preview

The next part could be made a bit more clever, but I settled on it after a bunch of trial-and-error testing to see what looked good to me. I wanted an animated preview, like what you get on several video sharing sites. I settled on taking 10 frames evenly spaced from the video to creae a 12 frame animation with a 300ms intra-frame delay (repeating the last frame to pad the animation out and provide a pause before it loops). I take the second frame and use that to make the poster image (so the poster is a frame at roughly 1/10th of the way through the video). PyAV makes this really easy to do, wrapping all the ugliness of ffmpeg up into a reasonably pythonic API.

def processVideo(inName: str, inBlob: func.InputStream, attachSvc):
        # Maximum height of the resultant animation and poster.
        MAX_HEIGHT = 360

        try:
                inBytes = io.BytesIO(inBlob.read())
                container = av.open(inBytes)
                duration = container.duration // 1000000
                step = (duration // 10) * 1000000
                if step < 1:
                        step = 1

                # Calculate the size based on the current size and aspect
                # ratio.
                h = container.streams.video[0].height
                w = container.streams.video[0].width
                thumb_size = (int(w * (MAX_HEIGHT / h)), MAX_HEIGHT)

                frames = []
                for i in range(1, 10):
                        container.seek(i * step)

                        for frame in container.decode(video=0):
                                img = frame.to_image()
                                img = img.resize(thumb_size, resample=Image.LANCZOS)
                                frames.append(img)

                                # Special case handling.
                                if i == 1:
                                        generate_poster(
                                                thumb_size,
                                                change_exten(inName, 'jpg'),
                                                img,
                                                pretty_timestamp(duration)
                                        )

                                elif i == 9:
                                        frames.append(img)
                                        frames.append(img)

                                break

                outBytes = io.BytesIO()
                outName = change_exten(inName, 'gif')
                frames[0].save(
                        outBytes,
                        format='GIF',
                        save_all=True,
                        append_images=frames[1:],
                        duration=300,
                        loop=0
                )

                writeBlob(outName, 'image/gif', outBytes.getvalue())
                attachSvc.mark_processed(inName, thumb_size[1], thumb_size[0])
                logging.info(f'Created slideshow anim {outName}')

        except Exception as e:
                logging.error(f'processVideo(): error processing {inName}')
                logging.exception(e)
                return

Conclusion

This really cements the decision to write the processing pipeline in Python for me. I get to use the extremely robust Python ecosystem in a modern, serverless cloud environment. For those who may be curious, the current average monthly Azure bill for the entire Thoughts ecosystem (storage and compute) is around 9ยข.

Now I need to build something for this guy to lurk in. Good thing I don't have 200 projects in some form of 'in progress' or anything... Oh wait.

As you can see, it works pretty well. I still have some work to do in the static thought renderer that archives the data from the cloud to my local server but honestly that can wait. You can find the entire source of the Azure Functions in my git repository.

Subscribe via RSS. Send me a comment.