Set up a Dreamwidth queue, drafts, and local backup on OS X

[ljwrites]

There isn't much I miss from my stay on Tumblr and it's a great relief to spend more time on Dreamwidth-- and, these days, my feed reader. However, there are still some functions that Tumblr has and Dreamwidth lacks, like a draft folder and the ability to queue posts.

Tumblr aside, I also found myself wanting to keep local copies of my posts and to compose them in an editor with more robust functions, such as live Markdown preview, than the DW editor offers. I like the new DW beta editor but it's still not an actual text editor, nor does it write to my own machine without an extra step like copy-paste.

So I figured, why not set it up on my computer? It seemed simple enough with Dreamwidth's post by email function and Terminal on Mac. It took me longer than I thought to get it working reliably because there were a few different components to it, like Postfix, Bash, and Launchd.

I wrote this documentation both as a note to myself and a reference for anyone who wants something similar. With a little modification I think this setup could be used for other operating systems like Linux distros and other blogging platforms that support post-by-email, such as Wordpress.

1. Configure Dreamwidth to post by email

Since this system depends on DW's post by email feature, that needs to be set up on the site by going to the link and following the instructions.

Note: In setting up your system for the steps below, it may be useful to look at your system.log if things don't work as expected. On Mac OS X press Ctrl+Space and type "Console" into Spotlight Search, then click system.log on the left column. There is a search box on the upper right-hand corner to help you look for specific file names and processes.

2. Configure Postfix to send email from Terminal

Postfix should be configured to send emails through terminal commands. By default Terminal will send from a local email address, like USERNAME@MacBook.local, emails from which tend to get shunted into junk and which I am not sure DW will recognize as an account address.

I personally found it easier to relay the emails through a Gmail account, and got it to work by following the instructions on the guide Send Emails on Mac OS X with Postfix and a Gmail Relay. The information there is probably good with some modification for other email services that allow external access, too.

In editing the relevant setting files I found Vim for Beginners very useful. Basically you start in Normal mode, which can also be understood as navigation mode. Type /search term to look for the necessary settings, for instance /inet_protocols to see if this setting exists in the configuration file, and tap n to search for the next occurrences of the term. When you find something to modify, type A to go to Insert mode where you can modify the file as usual. To return to Normal mode tap Esc. To save, type :w from Normal mode, and to quit type :q (or :wq to do both at once).

Sending an email didn't work the first time because Gmail was scandalized by my attempts to access its service on an insufficiently secure app and blocked the email. I had to change security settings to make Gmail allow low-security apps, and then it worked.

Also, right after I got the queue working at last, Postfix threw a weird tantrum that had it sending endless error messages without working. I solved it by stopping and restarting it:

    $ sudo postfix stop && sudo postfix start

3. Write some posts!

I wrote plaintext .md (Markdown) posts, one file for one post, formatted as usual for DW in HTML and/or Markdown. I use GitHub's Atom for its live Markdown preview and sidebar file tree, but anything is fine, like the built-in GUI editor TextEdit or even Vim from Terminal. These are plain text files, after all.

In this setup the first line in the body of the file should be the subject line of the DW post, followed by a blank line and then the body of the post. The post-icon and post-mood and other post-headers are commented out at the bottom of the post, each command on its own line. A sample local post file would look like this:

Subject line

First paragraph of text

Second paragraph, etc.

Last paragraph
<!--
post-icon: icon keyword
post-tags: tag 1, tag 2, tag 3
post-mood: happy
-->

If you know a little about formatting posting emails to Dreamwidth you'll realize that the post-headers should be at the start of the email. The script takes care of that automatically, however, and once activated will take that block of information to put it at the top where it should be. The post-headers can be at the top, middle, or wherever in the file; I just put them at the bottom to keep things tidy.

I also did not bother starting the file with !markdown. I use Markdown for almost every post anyway so it's tedious to type out, and I don't want that distraction in my local file. I put that part in the shell commands instead, see below.

The rules of file formatting under this setup are:

1. The first line is the subject line
2. The second line is blank. No spaces or anything, just blank.
3. The third line starts the actual text, which is formatted like any DW post. Markdown is allowed but you shouldn't put !markdown in the post, since that's in the script.
4. Post-header information is commented out, and each header item is on its own line starting with post-.
5. Due to the above, lines that are not post-headers may not start with post-. "Post-" is okay, as is "post " and " post-" and "postapocalypse" and "- post-modernism" so on, but "post-" is reserved for headers such as tags and icons that will be put in the start of the email body. This shouldn't be hugely restricting in most cases, but it is something to keep in mind.

4. Set up folders

I have a drafts folder, a posted folder, a script folder, and a queue folder in my setup. Drafts is where I keep the files for the posts that are still being written, while the posted folder is where the script will place the files after they have been posted, with a timestamp added to the filename to indicate when they were sent for posting. Script is where I keep script-related files like the shell file and the error and output logs.

The queue folder is where the files will sit until they are grabbed and processed by the script to be posted. I also have seven different subfolders under queue, one for each day of the week and on a different subject like Reading Wednesday and so on.

In this example, for the sake of brevity, all the files are subfolders of the ~/Documents folder, that is /Users/USERNAME/Documents/drafts and /Users/USERNAME/Documents/posted and so on where USERNAME is replaced by your own user name in the system. The tilde is shorthand for the current user's home folder, which I will use in the Terminal commands assuming that you will be logged in as the same user who owns the relevant files. The shell script file that holds the commands to process the files is in the ~/Documents/script folder.

I draft posts, formatted as above, in the drafts folder, then drop the completed files into the appropriate queue folder. The shell commands won't touch anything in drafts and will work only with what's in the queue folders, then fill up the posted folder on their own.

5. Prepare the posting script

Create a file under ~/Documents/script called dw_qscript.sh with the following content:

    #!/bin/sh

    #move to the queue folder
    cd /Users/USERNAME/Documents/queue

    #for a setup with a different folder for each day of the week, uncomment the line below 
    #cd $(date +"%u_%a")

    #the above line will look for a subfolder of queue named for the day of the week 
    #starting with the corresponding number for sorting, "1_Mon" "2_Tue" "3_Wed" etc.

    #get oldest file in the queue folder
    qfile="$(ls -tr1 | head -1)"

    #if the folder is empty, abort
    if [ "${qfile:-0}" == 0 ]; then
        exit
    fi

    #take all lines starting with 'post-' and put them in a temporary post file. 
    #This is the post-header information.
    echo "$(sed -n '/^post-/p' $qfile)" > postbody.temp

    #add a blank line to denote the start of the post body to DW post-by-email system
    echo "" >> postbody.temp

    #enable Markdown
    echo '!markdown' >> postbody.temp

    #add everything in the post file after the first blank line to the temporary postbody file
    echo "$(sed '1,/^$/d' $qfile)" >> postbody.temp

    #send an email to the DW posting address
    #the first line of the original post file is the subject line & the postbody file is the email body
    #Replace DREAMWIDTH_ID and DREAMWIDTH_PW with the actual ID and password. Keep the + sign.
    mail -s "$(head -1 $qfile)" DREAMWIDTH_ID+DREAMWIDTH_PW@post.dreamwidth.org < postbody.temp

    #delete the temporary file
    rm postbody.temp

    #insert time sent for posting to file
    echo '<!-- Time sent for posting: ' $(date +"%Y/%m/%d %H:%M") ' -->' >> $qfile

    #move the post file to the 'posted' folder with current timestamp on the filename
    mv $qfile /Users/LJ/Dropbox/writing/blog/dreamwidth/posted/`date +"%Y%m%d-%H%M_$qfile"`

The script takes the content of the post file and rearranges it so that the posting email to Dreamwidth looks like this:

Subject: Subject line
To: DREAMWIDTH_ID+DREAMWIDTH_PW@post.dreamwidth.org
Email body:
post-icon: icon keyword
post-tags: tag 1, tag 2, tag 3
post-mood: happy

!markdown
First paragraph of text

Second paragraph, etc.

Last paragraph
<!--
post-icon: icon keyword
post-tags: tag 1, tag 2, tag 3
post-mood: happy
-->

Make sure the script works by running it with the following command. You may want to back up the file and set post-security: private in case something goes wrong.

    $ cd ~/Documents/script && ./dw_qscript.sh

6. Set up Launchd

So now you have a script that posts your local files to Dreamwidth, but it's not automated. For that part, on Mac OS X, you can set up a scheduled job through Launchd. (1) You can check out more information about Launchd at the Launchd.info website and also the Apple technical documentation.

The basic way Launchd works is that you set up commands in a properties list (.plist) file, together with a condition to trigger those commands, then load the .plist through the launchctl command to register it as an automated job. These jobs can be daemons, which are not specific to a user, or agents, which are specific to users. The distinction is important because it decides what folder you save the .plist file to, and also the ownership and permissions of the .plist and any shell scripts you call.

For this setup I used an agent because all the files detected and changed by the script are specific to one user, me. (2) To have a similar setup you can make a file called com.USERNAME.dw.qscript.plist and put it in /Users/USERNAME/Library/LaunchAgents. Here's what the file would look like, based on the one I made:

    <?xml version="1.0" encoding="UTF-8"?> 
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> 
    <plist version="1.0"> 
      <dict>
        <key>Label</key>
          <string>com.USERNAME.dw.qscript</string>
        <key>Program</key> 
          <string>/Users/USERNAME/Documents/script/dw_qscript.sh</string> 
        <key>StandardErrorPath</key>
          <string>/Users/USERNAME/Documents/script/q_err.log</string>
        <key>StandardOutPath</key>
          <string>/Users/USERNAME/Documents/script/q_out.log</string>
        <key>StartCalendarInterval</key>
          <dict>
            <key>Minute</key>
              <integer>00</integer>
            <key>Hour</key>
              <integer>21</integer>
          </dict>
        <key>AbandonProcessGroup</key>
          <true/>
      </dict>
    </plist>

This file has the unique identifying label designated in the Label key, which generally seems to be the file name without the .plist extension. The Program key asks Launchd to run the designated shell command file. Any errors and outputs will be written to the paths designated with the StandardErrorPath and StandardOutPath keys. If the computer is asleep or turned off at that time, or the user is not logged in, it'll run the job the next time the user is logged on.(3)

About 90% of what I struggled with in setting up this job was ownership and permission of files. You have to make sure that every file involved in this job is owned by the right user, in this case you, and that it has the right permissions--permissive enough to run without being too permissive, which would also fail to run because it's a security risk.

First, check that your .plist file itself is owned by you and has the right permissions. That is, it can be read and written by you, the file owner, but is not writable by anyone else. After saving the file as above, run this command (assuming for all terminal commands that you are logged in as the user the agent is for):

    $ ls -l ~/Library/LaunchAgents

It'll show a list of your files in the LaunchAgents folder together with owners and permissions, and yours should look something like this:

-rw-r--r-- 1 USERNAME staff 830 Apr 9 21:49 com.USERNAME.dw.qscript.plist

The first and third columns are the important ones here. If the third column by any chance doesn't have your USERNAME in it, then you need to change that with the following command.

    $ chown USERNAME ~/Library/LaunchAgents/com.USERNAME.dw.qscript.plist 

If the current owner is "root," you'll need to use sudo and enter the admin password to transfer ownership to yourself.

    $ sudo chown USERNAME ~/Library/LaunchAgents/com.USERNAME.dw.qscript.plist 

If the first column which lists the permissions of the file looks different, run this command:

    $ chmod 644 ~/Library/LaunchAgents/com.USERNAME.dw.qscript.plist

Next, look at the owner and permissions on the script file called by the .plist file and make sure it is 1) owned by you and 2) has the right permissions. This file unlike the .plist file has to be executable. If it is not, the system log on Console will show one of the few useful Launchd errors saying that the file could not be executed.

    $ ls -l ~/Documents/script

The permissions and ownership of your script file needs to look something like this:

-rwxr-xr-x@ 1 USERNAME staff 1305 Apr 9 21:35 dw_qscript.sh

Again, same deal. If the third column does not list your USERNAME change that with chown, as admin if necessary.

    $ chown USERNAME ~/Documents/script/dw_qscript.sh

If the first column, the permissions, doesn't show the file to be readable, writable, and executable by the owner (you) and readable and executable by group and all, change it like this:

    $ chmod 755 ~/Documents/script/dw_qscript.sh

If the designated output and errors files don't exist, they will be created when the job runs and should have the correct ownership and permissions.

If all the ownership and permissions are done correctly you should be spared many of the frustrations I experienced getting this job up and running. System.log is spectacularly unhelpful with permission and ownership errors, only telling you that "Service could not initialize" without saying what exactly is wrong, so it's a good idea to address this issue systematically going in.

Basically, the .plist file together with every file and directory referenced in it has to be owned, readable, and writable by the user who owns the launch agent, and in the case of the .sh script file executable as well.(4) If any line of the .plist file wants to execute or write a file that does not belong to you or doesn't have the proper permissions, or references a file that doesn't exist or can't be found (which is why it's a good idea to have absolute paths for everything in your .plist file), Launchd will fail to run it with the aforementioned unhelpful error message.

The StartCalendarInterval key says that this job will be triggered at a specific time of day. There are a lot of other possible triggers for jobs which you can check in the relevant documentation. The Minute and Hour keys together with their value strings tell you what time to run it. Hours are based on a 24-hour format.

The AbandonProcessGroup key and the value of true are necessary because the script sends an email, which is a subprocess that we don't want to be shut down the moment the job is done. Don't ask me how it works, like everything else I grabbed it off Stack Overflow. :P

Once you have set up your Launchd job and have checked your file ownership and permissions, check to make sure the .plist file is well-formed:

    $ plutil -lint ~/Library/LaunchAgents/com.USERNAME.dw.qscript.plist

If it's well-formed, the following message will be returned.

    /Users/USERNAME/Library/LaunchAgents/com.USERNAME.dw.qscript.plist: OK

If not, go back to the .plist and make sure all your tags are closed, you have a unique string for your Label key, and the format is otherwise 100% correct. Remember, though, that the file being formally OK doesn't mean it's going to run. This form check doesn't verify things like whether the referenced script file exists and runs, or whether all file ownership and permissions are correct.

Once you have the .plist and its associated script, file, and folder ownership and permissions as correct as you can make it, it's time to load the .plist with launchctl.

    $ launchctl load ~/Library/LaunchAgents/com.USERNAME.dw.qsript.plist

This only loads the job into Launchd to run at the designated time, and does not run it right away. One tutorial I saw suggested it can be started right away with launchctl start but that didn't work for me. I just set the Hour and Minute keys to five minutes later to test things. (5)

If you make any changes to the .plist file you have to unload and then load the job so Launchd is aware of those changes.

    $ launchctl unload ~/Library/LaunchAgents/com.USERNAME.dw.qsript.plist
    $ launchctl load ~/Library/LaunchAgents/com.USERNAME.dw.qsript.plist

If everything was done correctly the Launchd service should run in the background at the designated time, and it should post any file in your queue folder by firing off an email to Dreamwidth before moving the file to your posted folder.

7. Enjoy!

After this setup was done my workflow looks like this: I write up posts as files in my drafts folder, then drag them to the right queue folders when I'm done. I have a separate little Launchd job to add the time the post was queued as a comment to the post, just to help me keep track. b (Edit 2019/04/20: This has become less relevant, however, since I put a git repo on my project folder. Git also keeps track of postings through the movement of post files to the posted folder, but I'm still keeping the file timestamps for sorting purposes.) Moving the files is even easier since I compose the posts in Atom and then drag them around the project folder tree in the Atom sidebar. Since the relevant folders are in my Dropbox, I can edit posts on mobile using the Dropbox app.

Launchd will fire up once a day at the designated time if my computer is on and active, go to the queue folder named for the current day of the week, and post the oldest file there by sending an email. It will then move the file to the posted folder with a timestamp added to the filename. This both takes the file out of the queue folder to avoid duplication and helps me keep track of posts made from the queue folders. This all takes place in the background so that it doesn't steal the focus from whatever I'm doing at the time, nor am I even aware it's happening.

This system has some obvious limitations, though. For one thing it's not a continuous synchronization; the posts are identical only up to the point of posting, and any changes made to the online version of the post will not be automatically reflected in the local version and vice versa.

Perhaps the most fundamental limitation is that this setup just runs on my local machine at home and not from a dependable always-on server. If my Mac is off or asleep at the time the job is scheduled the queue will wait until I am logged in, which defeats some of the purpose of having a queue. For now, though, this setup serves my needs well enough.

Possible future improvements include scheduling posts for specific dates and a finer-grained way to arrange the queue than simply posting the oldest file first. Those seem simple enough to implement with file sorting options, but I'll stop fiddling with this for now and use it for a while to see what improvements I need.

Feel free to ask me for questions or help if you're interested in setting this up and are stuck, though I'll have to warn you that I'm a total n00b and got everything off of other people's blogs and Stack Overflow. ^_^

Notes
1. Since crontab was deprecated in OS X this is the way you should go. Launchd is more versatile anyway, once you can get past the need for a well-formed .plist file and the need to get the file permissions and ownership just so.
2. I actually tried to make this job a daemon at first, then quickly realized that trying to send email from Terminal as root didn't work for a reason I couldn't figure out. I could have made it work with the UserName key, I suppose, but it was still more of an agent than daemon job so I made the switch to agent.
3. This is a nice change from crontab, which would simply skip that day if the computer was asleep at the time.
4. That last part might not be quite as strict for agents as it is for daemons, but I'm too lazy to test it and restricting writing privileges is a good security practice at any rate.
5. And did it over... and over... and over again fixing this or that, getting the same uninformative error message each time. That was fun.

Comments

Hooray!

[jesse_the_k]

You posted this!

I look forward to the day I am coherent enough to try this out, and will report back.

Re: Hooray!

[ljwrites]

I can't wait to see how it goes for you!

[mdlbear]

This is nifty! Mine's a lot clumsier and doesn't do scheduling, though it does have some features like git integration and the choice of markdown or HTML. Needs some refactoring, though.

Feel free to grab any ideas that look useful.

github.com/ssavitzky/MakeStuff

[ljwrites]

That's amazing! Love the Git integration and DW-specific tag handling.