Tweetus Deletus, tweets and json sequences

This you?

I recently downloaded my Twitter archive, and was it a trip down memory lane; I’ve had a Twitter account for almost 10 years! I laughed, reminisced, and also cringed. I’ve changed quite a bit, and like to think of myself as more poised and refined: I needed to do a self audit.

So here I am with an archive of some 11K+ tweets in a json file some 65K lines long. I have the following options for parsing it:

1. Read the entire file into memory as a json blob with Gson or something similar
2. Sequentially parse each line and manually keep track of where in the json tree I am, and lazily extract and process the information I need as I find it.

The solution as it would turn out like most things, is somewhere in the middle; use a library to sequentially parse the tweets and delete them as I go along. I opted to use Moshi and its excellent JsonReader class for parsing, however going through it’s docs, the example used is classically imperative:

There’s nothing wrong with this, and as it turns out, it’s often the best way to go for strictly performance reasons. Unfortunately for performance, I have the luxuries of time, CPU, RAM and the slightly overzealous need to use functional programming constructs and kotlin. With that said what does Json parsing and functional programming in kotlin have in common? Glad you asked.

The expression while(reader.hasNext()) loops through entries in a json mapping, which is imperative speak for a sequence of tuples. The above then can be represented more f̶a̶n̶c̶i̶l̶y̶ functionally with:

Creating a sequence of an arbitrary type on a JsonReader

Notice both the open and close functions are lazily evaluated by representing them as empty sequences with the actual sequence sandwiched in the middle. Let’s set about actually parsing that json then, shall we?

Using the generic JsonReader.jsonSequence function, I can easily represent all the tweets I want to process as a lazy sequence. Deleting them then becomes as simple as:

Deleting tweets in style

Source for Tweetus Deletus can be found here:

tunjid/TweetusDeletus

A young impressionable me

Used to tweet

With reckless glee

Age however

Has stayed my feet

With the above

I am born anew

Less in fear of the dreaded

“This you?”