Six months ago, I had a crisis.

At my day job, the amount of data I had to keep track of and manage was exploding. I felt like I was constantly putting out technical fires.  As lead web developer at an eCommerce company, I’d built myself into a corner, and the majority of the infrastructure depended on my physical presence to maintain it.

So I started building a system to get the local software and infrastructure out of my head.

Initially, it was simply a Markdown based file that simply had a heading of the day’s date, and then entries for what I’d done. I figured, this way I was at least getting the data of how software was set up, why routers were configured a certain way, and that the daily decisions had gotten out of my head. Someone could figure out later how to search through everything, and eventually I’d get a bit of peace of mind.

This initial system actually ended up being an incredible help. I experimented with other text file modes, and eventually settled on an Emacs org mode based journal. I put this file on my Dropbox, and would make entries from where ever I was.

When it came time to figure something out, and I had a question, or another programmer had a question, we’d both go looking into this journal for the answer before interrupting anyone.

In time, it because a crucial asset in being able to instantly report back answers from questions about what had been done months ago. I got better at my job, and I began to feel like I’d be able to reach the point where I could leave and the infrastructure I’d built wouldn’t implode on itself.

I’d grown in love with the concept of a coder journal. So I decided to start building the infrastructure to improve the way communication happens amongst programmers.

I had the following priorities:

• Input Must Be 100% Passive for Coders – Interruptions are a Coder’s kryptonite
• Management Must Not Interrupt for Updates – Management must be able to see progress and know what’s going on, without having to interrupt the creative process.
• Searching Must Be Intelligent – I want to be able to search with fuzzy logic, and find things out based upon what I’ve entered before.
• Input Must Be As Effortless As Possible – This means mobile, web, and also Dropbox based files.

Building it Out, Iterating in Steps

I decided to build a basic iteration in Flask. I normally approach web applications with a Django / Pinax stack, but in this case, I figured I’d need as much flexibility as possible.

And this has been a very good decision, as I’ve made quite a few ugly hacks to test out features which would have taken much longer in Django.

The initial iteration just used the Dropbox API to pull my journal, read the org mode data, and then create entries based upon date and time entered. For simplicity, it used SQLite to manage all the entries, and didn’t do much, except for spit things out in a nicer format.

I then added better styling with Bootstrap to get a quick idea of what a possible user interface would look like.

I came back with something like bellow:

Adding the Icing of Functionality with Flask Extensions

With this very basic functionality implemented (search didn’t work), I began to fill out all the parts I knew I’d need as a web based application. Fortunately, even though Flask is dead simple, there are some great extensions to get a few things built in rapidly.

To this extent, I added the following:

• Users – Implemented by the great Flask-Login extension. Super quick to implement, and get up and running with.
• Pages – Implemented using Flask-FlatPages. Just use Markdown and a template to get things up and running rapidly.
• Admin – Implemented using Flask-Admin. You need to be careful with this in production, as it doesn’t have any authentication or protection out of the box. But for local dev, it’s great!
• Models – Implemented with Flask-SQLAlchemy. And what a dream it’s been working with SQLAlchemy. Really, if you know what you’re doing with a database, there is no better way for interacting with models. Highly recommend Essential SQLAlchemy for getting started.
• Forms – Implemented with Flask-WTForms. Really a slick way for building forms, fairly similar to Django forms.

With these few little pieces, I had a very first iteration of my web application, and it became time to start building the first iteration of search.

Adding Apache Solr to My Web Application

So, if you’ve never had to deal with search before, let me spare you some time:

Just use Apache Solr.

It really isn’t that difficult to set up (you don’t need to know Java), and it’s incredibly powerful. I got up and running with it in half an hour after failing with Haystack for a couple hours. It just works.

You design your models in XML files, and then you simply send requests with the information you want to add or search for. There’s a module written for Python called “solr” that just works. That’s it. Easy peasy.

Getting Mobile with It

So, I recently got an iPhone 4S, and I’ve been blown away by it. The new lens is incredible, but my favorite feature, by far, is Siri. She’s not very good at much, but she’s pretty good at voice recognition and sending text messages.

So I decided to add text message receiving functionality to my web application. And that means adding a whole barrel of technical monkeys.

• Twilio – Use the Twilio API for sending and receiving text messages. It’s just so good. It’s so easy, it just works. It’s an incredible product, and it’s revolutionizing the world.
• Celery – You’re going to have to start managing all the tasks that come in, and keep track of what’s been done, what needs doing. Bonus, Flask-Celery already exists, and is great!
• RabbitMQ – This is just the best task server, from what I’ve heard. But I know nothing. It’s what Celery recommends, so I’d rather take their word for it.

With this infrastructure, I can now use Siri to create text messages, and interface with my cell phone!

 

Thinking Bigger, How Far Can I Take This?

After having added mobile to my web application, I started to get really excited for the potential. Imagine how useful an intelligent journal could be in the developing world, where cell phone usage is exploding, and where access to computers will still be limited for the next few years.

Millions of people could potentially have an intelligent journal for managing their lives with what I’ve started building.

I decided to try and get this initial prototype into the hands of 10,000 people in the developing world. To that end, I’ve created a Kickstarter page.

I’d love your support!