Writing a Technical Book in Emacs and Org Mode

I spent the past year writing a book called Make Art with Python. It teaches Python programming through drawing and art, instead of print statements. If that sounds interesting to you, sign up to hear when it’s released here.

Why Write a Book in Emacs?

I needed a distraction free way to write my code and prose in one place. I also needed to track my progress during the writing itself, and settled on using a paper journal and org-pomodoro.

If you’ve ever wanted to write a technical book, the past few months have taught me it’s possible for mortals with day jobs. I’ve just sent my fourth draft off to my editor, and tens of thousands of lines of code and prose have already been written. It’s the first time in my life writing a book has felt achievable.

But there have been painful false starts and mistakes along the way.

In this article I’ll share some basic rules I stole to keep myself focused, motivated, and productive while writing. I’m sharing in the hopes that they will help you too, and you’ll begin writing your book much sooner than it took me.

Most of these ideas and habits are generally applicable, and although they may seem insignificant, improved my writing tremendously. In fact, all the writing and code for my first draft was written while I was the sole backend developer at a startup which grew to millions of users.

I’m a big believer in concrete examples as the only way to learn anything, so I’ll also walk you through the mechanics of the tools I use. We’ll cover the process of writing, file and time management, visualizing the work in progress, exporting, and maintaining a positive attitude to keep the work flowing. Of all these parts, the biggest challenge is mental.

When things get difficult, it’s mostly because you don’t know what to do next. It’s during these times that you need have a routine that happens, no matter what, to get you focused back on writing and making progress.

If you’ve been thinking about writing a book, this guide will hopefully convince you it’s possible.

And best case, soon I’ll be reading your book.

Let’s get started!

12 Steps to Write a Technical Book
(Assuming you already know what you want to write about)

  1. Create a terrible first draft by writing a set amount of words every day. (1000 words per day is a good start)
  2. If your book is about coding, add a new feature to your code, or a refactor to every day.
  3. When you get stuck, it’s always better to have something written for the day than nothing at all.
  4. When you’ve written enough for a first draft, stop.
  5. Pat yourself on the back for having been so good about writing.
  6. Let your first draft sit for a while, and begin writing / planning marketing for said book.
  7. Reread draft later, and realize 90% of it needs to be thrown away.
  8. Panic.
  9. Use pomodoros and a suspension of rationality to begin a rewrite.
  10. Start thinking about the visual presentation of your ideas, and structure of book.
  11. Finish rewrite, and begin to see glimmer of hope for book.
  12. GOTO 8
  13. Congratulations! You are now a writer!

The Writer’s Toolbox

Keep a Writing Journal

With every major software project I begin, I keep a coding journal. In it, I keep all the meta-information about my code that doesn’t fit into my source control. Things like why decisions were made, notes for later, conversations had, and ideas discussed. It’s been an incredible leg up in my programming career to be able to go back to conversations from months ago, and see exactly why decisions were made, and by who.

In the same way, when I began the book, I kept a writing journal. This was just a high level journal where I kept thoughts that didn’t fit into the existing book, and just a place for me to stick random ideas.

It looks something like this:


** Wed Apr 4 10:22:22 2016

- This entry mentioned some wrist pain I might have been experiencing while typing, and note whether or not some changes have helped.

** Tue Apr 5 12:46:55 2016

- This would be a note about whether a new approach is working or not.

- This would be another note about an idea I had, something to possible be expanded upon later.

- And this note would be an idea to touch upon tomorrow.

In my top level book directory, I just create a journal.org file, and I put the date into it every day with a couple of stars. This way, it’s set as a header, and I can view and collapse entries for each day by pressing tab.

By keeping a journal, I have a way to refer back to why decisions were made, long after the fact. I also have a way to analyze how things are progressing.

Also, I begin most writing days with something banal in my journal. It clears my mind and gets me ready for the focused mode of writing.

Know the Big Picture First

When I started, I didn’t know what my book’s goal was, or who my audience was. This was my biggest mistake, and cost me a lot of time.

I wasn’t sure whether I was writing a book for complete beginners, or for experienced developers who just wanted to start doing more creative programming.

Later, I decided writing a book for beginners was indeed the path I wanted to go down. Changing what had been written meant a pretty major rewrite.

If I’d started with clear goals in mind, I wouldn’t have written at least half of my first draft.

In fact, one of the most productive things for clarifying my book’s structure was building a landing page and pretending it already existed. I created it thinking of something that would sell, and what I wished existed. When I finally got something I was excited about I went back and did some major rewriting.

If I’d started with the landing page before my writing, I would have gone down a separate road entirely.

Stay Structured

Have a goal in mind for every day’s work. If possible, plan this goal out the night before, or at the end of your writing session, make a note of it.

For for the first draft, my goal was having at least a thousand words written a day. In addition, I wanted least one new feature added to the code base of the technical project built for the book. This tended to take over an hour each day to accomplish, because my first draft usually needed longer chunks of focused time to get both the code going and the initial draft of text down.

For the first draft, I had an emphasis on quantity, not quality. No excuses, I needed to have something written to work with later. Doesn’t really matter how terrible or insignificant the code or writing was.

By having this goal, at the end of the day, knew when I’d done the work necessary to have my book be written, eventually. And having that feeling of progress was enough for me to see through to the end of the first draft.

(Through some very difficult points!)

Use Time Blocking For Your Rewriting Sessions

You might have heard of “Pomodoro” before. If not, it means breaking down your work in to 25 minute highly focused sessions. Before you begin, you’re supposed to choose one task, and focus completely on it.

Of all the tools I’ve come across, this the most powerful for getting rewriting done. (Again, I prefer word count targets for first drafts.) Rewriting is more difficult and vague than writing a first draft, so blocked minutes of focus for rewriting give structure to something otherwise mostly chaotic.

Because for me, rewriting means endlessly bouncing back and forth between things like layout, graphics, editing, reshaping text, etc. It can go on forever, and feel as if it’s going nowhere after a while.

By breaking rewrite time down into focused sessions, I give myself an artificial limit, and force myself to stay on task. Also, there’s something addicting about measuring how many pomodoros you can do in a day.

There are a few different emacs packages for doing pomodoro. I use org-pomodoro because it integrates with my writing journal. With it I track how many pomodoros I accomplish in a day right in my journal.

To begin a pomodoro, I go into my current day in my writing journal, and do an M-x org-pomodoro. It starts my session, and it gives me a little bell when I’ve finished my 25 minute chunk, along with logging it to my org entry for the day. This way I can keep track of how many pomodoros, or units of work I accomplish day by day.

If you don’t want to keep track of how many pomodoros you accomplish every day, or you want a simpler interface, there’s also ‘pomodoro’. It’s available on MELPA.

If you’ve never installed a package in emacs, it’s pretty straightforward. M-x list-packages . C-s pomodoro. It’s the one that’s just called ‘pomodoro’. You may have to skip over other pomodoro packages my pressing C-s again a few times. Mark it by pressing ‘i’. Hit ‘x’ to execute it’s installation. Then add it to your startup.

Finally, just do an M-x pomodoro-start when you start writing. You can do an M-x pomodoro-stop when you finish.

Write and Test Code At the Same Time

Org mode is great, in that it lets you add and execute code inline while you’re writing.

Unfortunately for me, most of the code I’m writing doesn’t fit in this format, because it’s mostly graphics based. So I have to write separate files, run them, and then embed them in my org mode project.

The back and forth can be a little bit challenging, but all in all, it’s as close to a perfect situation as you can ask for.

To embed a source code into your org-mode file, you have a few different options.

You can use the built in org-mode source block, with the following style:


#+begin_src python
print('This is just an example')
#+end_src

Or, you can add the :tangle header and name to have org-mode export files with your code in them, for testing.

Personally, I’ve gone back and forth here, because the book I’m writing is so depended on side effects. I’ll write code in a buffer, and then switch over to another.

When you’re editing a code block, you can do C-C ‘, and you’ll get your native major mode to edit your source code in. When you’re finished, you can M-RETURN ‘ to exit.

Rewriting is the Real Work of Writing

It’s easy to write to a world count limit if you don’t care about the quality of your writing. Soon enough, if you keep writing a thousand words a day for months on end, you’ll have a really big pile of words to work with.

Ah, but that’s where the real work all begins. What to do with that mountain of words? How do you shape it into something ready for the world, and make it all into a cohesive whole?

And make it worth paying for?

Originally, I had no idea how much more work rewriting a book is than the writing itself. Because each time I look at what’s been written, either code or words, I see room for improvement. And that room to fix leads to room for restructuring, and then the whole process begin again. It soon feels as if forward progress is impossible.

Deal with Images and Figures Early

I do all my book writing in emacs on Ubuntu. This means I don’t have access to the normal tools, like Illustrator and Photoshop during my writing. This slows down how quickly I can make design changes, and clean up images.

Of course, there is the Gimp, an open source alternative to Photoshop, but I grew up on and learned everything I know from Photoshop. Making the effort to switch over is mostly a distraction from my main focus, writing the book.

So I’ve tried keeping my process as simple as possible. Instead of working with Illustrator or some other vector graphics program, I’ve just gotten myself a bit of paper and some markers. Whatever doesn’t fit from this, ends up getting screen shotted. Between the two, I have my images, and I have my ideas.

Later I’ll have an illustrator redo my general designs. This seems like a much better approach than just leaving them blank.

By the way, embedding images in org-mode is easy. Just place double brackets on both sides of the file path:

[[chapter1/images/filename.jpg]]

Marketing Is Almost As Important as the Book Itself

If the past two years of developing wildly successful apps has taught me anything, it’s the importance of marketing.

Don’t be the person who neglects marketing and never has anyone see their work.. If you’ve ever wondered why most popular things are trash, it’s because the real product they’re creating is their marketing. So balance it out, write great stuff, and do us all a favor and market it.

I started a marketing directory immediately after finishing my first draft of the book. In this marketing directory I put ideas for posts that I could write to gain traffic, and get people exposed to what I’m writing. (This article came out of that marketing directory.)

In the past, using blog posts was a great marketing channel for me. I had a launch post for the first app I ever made, and had around 10,000 visitors flow through to my landing page in the first few days. They came through videos, blog posts, and social media.

The problem is, coming up with these ideas, for posts, and being consistent in releasing them. (In hindsight, I didn’t have enough structure with this, and should have begun releasing things sooner.)

Preview What You’ve Written Often

Psychologically, I need to have physical evidence of all this work. One of the biggest drawbacks from being a software developer is that you never see or feel all the code you’ve written. Instead, you throw everything away to an imaginary world, never to be seen again.

So for me, it was terribly important that I was able to see more than just a filled text buffer.

By writing in org mode, I can build and export to a PDF as I’m writing, using C-c C-e l p. This tells org mode to export, to latex, to a pdf.

The default latex settings for emacs should be good enough, but if you want syntax highlighting in your preview, you can also use pandoc to get nice syntax highlighting in your preview. Use it like this:


pandoc -s -o chapter1.pdf chapter1.org

It will spit out your PDF, with its default layout. It’s good enough for proofreading, but not anything you’d want to actually publish.

Play the Long Mental Game

The mental game of writing a book is probably 99% of getting the damn thing done and out in the world. For me, I’m still struggling with working out a way to play the game that works. As mentioned before, bouncing between pomodoro and writing goals, depending on the type of work seems to have finally made everything click.

I wrote the entire first draft working at a startup. That meant not sitting down to write sometimes until 12 at night, and working until 1 AM, every day. So an hour’s worth of good effort every day, after having been through the wringer emotionally and mentally. Having that quota of words filled up at the end of the day was the ultimate triumph, and never negotiable.

For the second draft, I didn’t have a day job. So the hours should have hypothetically been much better. But instead, for the beginning at least, I mostly found the opposite. I think that lack of focus can be attributed to making the transition back to self motivation and direction. (As an aside here, I think I needed to relearn how to provide my own artificial structure to my home environment. It’s something I’ll talk about more some other time. Basically, it’s a skill, you just need to be deliberate about cultivating it, and be honest with yourself why you’re not doing something you know you should be.)

No Matter What, Keep Moving

One of the things that has always perplexed me is how (relatively) easy it is to do something big like hike a long trail, compared to writing a book. With hiking, all you have to do is put one foot in front of another.

But with writing a book, there is so much that can derail you. You can get lost in designing the book cover, get lost in picking the right flow of chapters, get lost in the right font, get lost in a rewrite, or in a change of emotion, decide to throw out everything you’ve written. With writing, and large projects like writing, it can seem as if there are too many decisions to be made to get anything done, ever.

The thing I learned this time though, is to simplify.

If you’re going to write, write. If you have to write with code, make sure that’s your daily goal, and you know when you’ve accomplished it. Again, for me, during the initial writing, it was 1,000 words in a day, and some new feature added to the code.

When it came time to rewrite, I didn’t have a daily goal, because I was entirely lost. Where do I go, where do I begin?

Honestly, I’d never done a large amount of rewriting. The biggest rewrite I’d ever done up until that point was the blog post before this one. And that had something like a hundred edits on it, and added up to almost a week’s worth of time. It meant writing Javascript, Python, and stringing together images from other hosting providers. I had no clue how I’d be able to deal with doing that level of a rewrite on every chapter I’d written. It was just too much to handle.

So again, I finally got things moving again by just breaking it down. Twenty five minute chunks of rewriting, what you get done, you get done. Take the breaks in between, and then back to focus.

Take it Easy On Yourself

It can be terrible to deal with all the unknowns while writing. Is this going to make it into the book, is the book a failure, is it going to even sell enough to make back my initial money? So many unknowns, you’ve got to learn to take it easy on yourself.

Work through things, and keep moving. Eventually all the effort will start to congeal, and you’ll see a direction. Take that direction, and don’t let up. Eventually it will all get finished, as long as you keep moving and don’t give up.

Support Your Fellow Authors

Let’s face it. Technical writers are a weird bunch in a world dominated by the crowd chasing the quick buck. We need all the peer support we can get when we choose to care about the craft of writing about technical things.

What better way to keep yourself motivated than by supporting your fellow authors, and seeing their success?

I’ve got a book shelf full of books written by other authors who have made the effort to write exceptional books. Every time I look at that book shelf I see the examples I want to follow.

The bookshelf is my peer group, and they are the people I most want to emulate.

I hope some day my book ends up on your shelf, and the same for you on my shelf.

To your book!

(In the meantime, don’t forget to check out mine!)

Training Neural Networks to Generate Terrible Amazon Products

Teaching Machines Nonsense

Last Wednesday, while joking at the end of the workday, the idea came up to make an neural network that generates Amazon reviews.

Above you can see the results for images, reviews, prices, and product names generated by neural networks trained on Amazon’s data.

All of this was possible thanks to the dataset provided by Julian McAuley, used in the SIGIR and KDD papers, along with the torch-gan and char-rnn sources. We’ll walk through adapting this dataset and adapting it to train these neural networks in this writeup.

We’ll step through the thought process behind adapting and extending a dataset, and we’ll document the road blocks as we run into them along the way. I hope leaving these in will help beginners see that very often, programming requires running into wall after wall until you finally reach the other side.

We’ll cover how to load the dataset, how to generate fake product images, reviews, prices, and product names, and then export them for presentation. I hope you’ll enjoy the ride, even if you’re not necessarily into programming. I just want to give you an idea for how writing and extending an AI bot works when working from a given data set.

The finished neural networks and code with instructions are at Github.

So, let’s begin.

Meeting Your Data

The very first thing I did once I received the data set was to take a look at it, to get an idea for its formatting. It was originally compressed with gzip, and so I needed to uncompress it on my external drive.

Uncompressed, it was 68 gigabytes. When you’re working with large files, it can get tricky to figure out what you’ve got, and to make the most basic of assumptions. So the first thing to do is take a look at what you’re working with, and how messy your data might be.

In my case, I just used the ‘less’ command, to take a look at the first few lines in my terminal. This is easy enough to do in the command line:

$ less user_dedup.json
{"reviewerID": "A00000262KYZUE4J55XGL", "asin": "B003UYU16G", "reviewerName": "Steven N Elich", "helpful": [0, 0], "reviewText": "It is and does exactly what the description said it would be and would do. Couldn't be happier with it.", "overall": 5.0, "summary": "Does what it's supposed to do", "unixReviewTime": 1353456000, "reviewTime": "11 21, 2012"}
{"reviewerID": "A000008615DZQRRI946FO", "asin": "B005FYPK9C", "reviewerName": "mj waldon", "helpful": [0, 0], "reviewText": "I was sketchy at first about these but once you wear them for a couple hours they break in they fit good on my board an have little wear from skating in them. They are a little heavy but won't get eaten up as bad by your grip tape like poser dc shoes.", "overall": 5.0, "summary": "great buy", "unixReviewTime": 1357603200, "reviewTime": "01 8, 2013"}
{"reviewerID": "A00000922W28P2OCH6JSE", "asin": "B000VEBG9Y", "reviewerName": "Gabriel Merrill", "helpful": [0, 0], "reviewText": "Very mobile product. Efficient. Easy to use; however product needs a varmint guard. Critters are able to gorge themselves without a guard.", "overall": 3.0, "summary": "Great product but needs a varmint guard.", "unixReviewTime": 1395619200, "reviewTime": "03 24, 2014"}
{"reviewerID": "A00000922W28P2OCH6JSE", "asin": "B001EJMS6K", "reviewerName": "Gabriel Merrill", "helpful": [0, 0], "reviewText": "Easy to use a mobile. If you're taller than 4ft, be ready to tuck your legs behind you as you hang and pull.", "overall": 4.0, "summary": "Great inexpensive product. Mounts easily and transfers to the ground for multiple push up positions.", "unixReviewTime": 1395619200, "reviewTime": "03 24, 2014"}

We can immediately see that our JSON file isn’t really in JSON, but it’s in a kind of Python dictionary object. And indeed, when I look at the web url that I got this info from, it says specifically that each line can just be ‘eval’d in Python in order to generate one object at a time.

This makes it easier to deal with this large of a file, because it means we don’t need to read the entire list into memory just to create our object. (Which could take minutes to hours to do.) Instead, we can go line by line in our file, and each review individually into memory.

Continue reading