Building a Mobile Intelligent Journaling System in Python

Six months ago, I had a crisis.

At my day job, the amount of data I had to keep track of and manage was exploding. I felt like I was constantly putting out technical fires.  As lead web developer at an eCommerce company, I’d built myself into a corner, and the majority of the infrastructure depended on my physical presence to maintain it.

So I started building a system to get the local software and infrastructure out of my head.

Initially, it was simply a Markdown based file that simply had a heading of the day’s date, and then entries for what I’d done. I figured, this way I was at least getting the data of how software was set up, why routers were configured a certain way, and that the daily decisions had gotten out of my head. Someone could figure out later how to search through everything, and eventually I’d get a bit of peace of mind.

This initial system actually ended up being an incredible help. I experimented with other text file modes, and eventually settled on an Emacs org mode based journal. I put this file on my Dropbox, and would make entries from where ever I was.

When it came time to figure something out, and I had a question, or another programmer had a question, we’d both go looking into this journal for the answer before interrupting anyone.

In time, it because a crucial asset in being able to instantly report back answers from questions about what had been done months ago. I got better at my job, and I began to feel like I’d be able to reach the point where I could leave and the infrastructure I’d built wouldn’t implode on itself.

I’d grown in love with the concept of a coder journal. So I decided to start building the infrastructure to improve the way communication happens amongst programmers.

I had the following priorities:

  • Input Must Be 100% Passive for Coders – Interruptions are a Coder’s kryptonite
  • Management Must Not Interrupt for Updates – Management must be able to see progress and know what’s going on, without having to interrupt the creative process.
  • Searching Must Be Intelligent – I want to be able to search with fuzzy logic, and find things out based upon what I’ve entered before.
  • Input Must Be As Effortless As Possible – This means mobile, web, and also Dropbox based files.

Building it Out, Iterating in Steps

I decided to build a basic iteration in Flask. I normally approach web applications with a Django / Pinax stack, but in this case, I figured I’d need as much flexibility as possible.

And this has been a very good decision, as I’ve made quite a few ugly hacks to test out features which would have taken much longer in Django.

The initial iteration just used the Dropbox API to pull my journal, read the org mode data, and then create entries based upon date and time entered. For simplicity, it used SQLite to manage all the entries, and didn’t do much, except for spit things out in a nicer format.

I then added better styling with Bootstrap to get a quick idea of what a possible user interface would look like.

I came back with something like bellow:

Initial Iteration of Flask based Web Application

Adding the Icing of Functionality with Flask Extensions

With this very basic functionality implemented (search didn’t work), I began to fill out all the parts I knew I’d need as a web based application. Fortunately, even though Flask is dead simple, there are some great extensions to get a few things built in rapidly.

To this extent, I added the following:

  • Users – Implemented by the great Flask-Login extension. Super quick to implement, and get up and running with.
  • Pages – Implemented using Flask-FlatPages. Just use Markdown and a template to get things up and running rapidly.
  • Admin – Implemented using Flask-Admin. You need to be careful with this in production, as it doesn’t have any authentication or protection out of the box. But for local dev, it’s great!
  • Models – Implemented with Flask-SQLAlchemy. And what a dream it’s been working with SQLAlchemy. Really, if you know what you’re doing with a database, there is no better way for interacting with models. Highly recommend Essential SQLAlchemy for getting started.
  • Forms – Implemented with Flask-WTForms. Really a slick way for building forms, fairly similar to Django forms.

With these few little pieces, I had a very first iteration of my web application, and it became time to start building the first iteration of search.

Adding Apache Solr to My Web Application

So, if you’ve never had to deal with search before, let me spare you some time:

Just use Apache Solr.

It really isn’t that difficult to set up (you don’t need to know Java), and it’s incredibly powerful. I got up and running with it in half an hour after failing with Haystack for a couple hours. It just works.

You design your models in XML files, and then you simply send requests with the information you want to add or search for. There’s a module written for Python called “solr” that just works. That’s it. Easy peasy.

Getting Mobile with It

So, I recently got an iPhone 4S, and I’ve been blown away by it. The new lens is incredible, but my favorite feature, by far, is Siri. She’s not very good at much, but she’s pretty good at voice recognition and sending text messages.

So I decided to add text message receiving functionality to my web application. And that means adding a whole barrel of technical monkeys.

  • Twilio – Use the Twilio API for sending and receiving text messages. It’s just so good. It’s so easy, it just works. It’s an incredible product, and it’s revolutionizing the world.
  • Celery – You’re going to have to start managing all the tasks that come in, and keep track of what’s been done, what needs doing. Bonus, Flask-Celery already exists, and is great!
  • RabbitMQ – This is just the best task server, from what I’ve heard. But I know nothing. It’s what Celery recommends, so I’d rather take their word for it.

With this infrastructure, I can now use Siri to create text messages, and interface with my cell phone!

 

Thinking Bigger, How Far Can I Take This?

After having added mobile to my web application, I started to get really excited for the potential. Imagine how useful an intelligent journal could be in the developing world, where cell phone usage is exploding, and where access to computers will still be limited for the next few years.

Millions of people could potentially have an intelligent journal for managing their lives with what I’ve started building.

I decided to try and get this initial prototype into the hands of 10,000 people in the developing world. To that end, I’ve created a Kickstarter page.

I’d love your support!

Marketing Front Ends for Your MVP: WordPress vs. Flask vs. Django

This week I’ll be launching two separate Minimum Viable Products. I built the front-end marketing for one in WordPress, and the other in Flask. In the past, I’ve used Django for all of my product front ends, but this time I decided to experiment on my MVP development.

Keep in mind, for me, building a MVP means getting something going as quickly as possible. An extra day for building out a front end is wasted effort if we just want to test a market as quickly as possible.

I keep my code of my SAAS completely separate from my marketing front end. So this means my product resides at admin.myproduct.com, but customers land on my marketing at www.myproduct.com. This helps keep things separate, and keeps my marketing out of my codebase.

Remember, as programmers our main risk is finding a market fit, not solving a technical problem. So building a front end to test your potential market is a very good first step.

WordPress

WordPress is a great CMS, and has been around for nearly forever. It’s written in PHP, and has a million great, tested themes ready to go. You can find plugins for nearly everything, and you’ve got a great blog built right in.

That being said, customizing WordPress can be a bit daunting. If you want to extend functionality beyond what’s built in, you’re in for a real treat. It’s been a while since I last developed a plugin for WordPress, but learning how WordPress has decided to do things can take a while.

Also, if your designer hasn’t worked with WordPress before, it can be a bit confusing to build out a nice looking design or customization. As happens in most large PHP projects, the template language has logic built into it, and calls five or six different pages. That can be a pain in the ass for your designer to try and track down what page needs changing.

For my MVP which used WordPress, I simply bought a decent looking theme from WooThemes, and then wrote a form page with Mustache.js and Stripe to process payments and create accounts on my application.

So, a user enters form information at my WordPress frontend, and all information gets posted to my admin.companyname.com domain. It’s a simple way to keep marketing separate from my web application. If (god forbid) my WordPress gets hacked, the admin site is separate and stays up.

Pros: Easy setup, CMS for non-techies, readily available themes

Cons: Difficult to extend, Susceptibility to hacks, Designing can be difficult

Flask

I can’t think of a more fun environment to develop in than Flask. If you and your designer are both fairly comfortable writing raw HTML and CSS, you’ll have a great time building a front end out in Flask. The code ends up simple, and pages get exactly what they need, no more.

However, if you’ve got a non-technical person on your team who needs to edit the site, Flask is going to be a bit disappointing. There are projects out there that extend Flask into a more robust CMS, but that kind of defeats the purpose of using Flask in the first place.

Writing a front end in 100 lines of code that just displays exactly what you need, and does no more. With Flask-Pages you can build out a markdown blogging system in just 30 or so lines of code.

Your designer will love Flask, because building a template for Flask to use is dead simple. Unlike WordPress, which mixes PHP and logic with its templates, Flask just uses variables and a few tags to display what’s needed. It’s much less intimidating to someone who can’t code.

For my MVP project using Flask, I simply built out based upon the urls I knew I needed. Development went by extremely quickly, and it took about as much time to build out the front end as it did to set up a basic WordPress installation with the proper extensions and plugins.

Pros: Quick start, clean codebase, better for designers, flexible, low complexity

Cons: Not good for nontechnical people, Everything is built out manually, Need Python experience to get started

Django

Ah, Django. Up until a few months ago, I had only built out my MVPs using something like Django and Pinax. And it’s a great combination, because the two projects are really starting to mature.

That being said, the dream of truly pluggable Django apps seems nearly dead. It can take even longer to read and fully grok some of these pluggable apps than to write them yourself.

If you’re going to keep a separate Django project for your frontend, I suggest using one of the Django CMSs that are out there. The one that I’ve found most impressive is Mezzanine.

If you’ve never worked with Django before, it’s a great way to get started. Mezzanine is a very clean project that incorporates just about everything you could want to learn from how to structure a Django project.

But that knowledge again comes at a price. There is a bit of complexity in building out from Mezzanine, in that if you aren’t already experienced in building Django projects, it may be too much of a distraction for you, unless the built in template is good enough.

Pros: Built in CMS, Python flexibility, Mobile Compatible with Mezzanine

Cons: Complexity, intellectual overhead, themeing can be a bit difficult

Declaring a Winner

If you want to get something up and running as quickly as possible, I have to recommend WordPress. It gets up and running very rapidly, and there’s a huge user base. When it comes time to have non-technical people add marketing to your front end, you’re already prepared.

Drop in split testing, a landing page, and Google analytics, and you’re well on your way to seeing whether there’s a market for your product or not. When it gets time to build something more complicated, just suck it up. Your front end marketing shouldn’t be too technical anyways.

What About Your Experience?

Do you have a different process for building a frontend to test a MVP? Is it better? Did I miss something? Let me know bellow.