Jan 09

Old-style Mac OS X Leopard Exposé in Snow Leopard

Progress is progress, except when it gets in the way of your workflow. Let's compare these two screenshots:

Old-style Leopard Exposé

New-style Snow Leopard Exposé

Notice how much more pleasant the old-style Exposé is? Introduced in Mac OS X 10.3 Panther, and virtually unchanged until OS X 10.6 Snow Leopard, it featured proportional windows. By just looking at the size of the window relative to the other windows, you can get a fair idea of what the application is.

The proportional windows went out the window with the new Exposé. Now it features an inexplicable grid, with windows resized to all different dimensions relative to their original size.

Old-style Exposé in Snow Leopard

The great news is that you can get the old-school Exposé back. The beta builds of Snow Leopard included a new Dock.app that used the old-style exposé. By installing the old Dock.app, you get the new Dock features of Snow Leopard, while preserving the legendary Exposé.

Installation

  1. Download the Snow Leopard beta-build of Dock.app
  2. Save to your Desktop and unzip.

Run the following commands in Terminal.app:

sudo chown -R root ~/Desktop/Dock.app;
sudo chgrp -R wheel ~/Desktop/Dock.app;
sudo killall Dock && \
sudo mv /System/Library/CoreServices/Dock.app ~/Desktop/OldDock.app && \
sudo mv ~/Desktop/Dock.app /System/Library/CoreServices/

Easy to do and indispensible now that you have it back. Hat-tip to miknos at MacRumors for the original find.

Note that you will have to repeat this process every time you upgrade your Mac OS to a new patch release (10.6.6 -> 10.6.7).

@samuelclay is on Twitter.

Use Google Reader? I built NewsBlur, a new feed reader with intelligence.

Nov 03

What Happened to NewsBlur: A Hacker News Effect Post-Mortem

Last week I submitted my project, NewsBlur, a feed reader with intelligence, to Hacker News. This was a big deal for me. For the entire 16 months that I have been working on the project, I was waiting for it to be Hacker News ready. It's open-source on GitHub, so I also had the extra incentive to do it right.

And last week, after I had launched premium accounts and had just started polishing the classifiers, I felt it was time to show it off. I want to show you what the Hacker News effect has been on both my server and my project.

Hacker News As the Audience

When I wasn't writing code on the subway every morning and evening, I would think about what the reaction on Hacker News would be. Would folks find NewsBlur too buggy? Would they be interested at all? Let me tell you, it's a great motivator to have an audience in mind and to constantly channel them and ask their opinion. Is a big-ticket feature like Google Reader import necessary before it's Hacker News ready? It would take time, and time was the only currency which I could pay with. In my mind, all I had to do was ask. ("Looks cool, but if there's no easy way to migrate from Google Reader, this thing is dead in the water.")

Kurt Vonnegut wrote: "Write to please just one person. If you open a window and make love to the world, so to speak, your story will get pneumonia." (From Vonnegut's Introduction to Bagombo Snuff Box.)

Let's consider Hacker News as that "one person," since for all intents, it is a single place. I wasn't working to please every Google Reader user: the die-hards, the once-in-a-seasons, or the twitter-over-rss'ers. For the initial version, I just wanted to please Hacker News. I know this crowd from seeing how they react to any new startup. What's the unique spin and what's the good use of technology, they would ask. What could make it better and is it good enough for now?

If you're outsourcing tech and just applying shiny visuals to your veneer, the Hacker News crowd sniffs it out faster than a beagle in a meat market. So I thought the best way to appeal to this crowd is to actually make decisions about the UI that would confuse a few people, but enormously please many people. From comments on the Hacker News thread, it looks like I didn't wait too long.

How the Server Handled the Traffic

Have I got some graphs to show you. I use munin, and god-love-it, it's fantastic for monitoring both server load and arbitrary data points. I watch the load on CPU, load average, memory consumption, disk usage, db queries, IO throughput, and network throughput (both to external users and to internal private IPs).

I also have a whole suite of custom graphs to watch how many intelligence classifiers users are making, how many feeds and subscriptions users are adding, the rate of new users, premium users, old users returning, new users sticking around, and load times of feeds (rolling max, min, and average).

Used to be that when a thundering herd of visitors came to NewsBlur, I'd have to watch the server nervously, as CPU would smack 400% (on a 4-core machine), the DB would thrash on disk, and inevitably some service or another would become overrun.

Let's see the CPU over the past week:

CPU - Past week

Spot the onslaught? NewsBlur's app server is only responsible for serving web requests, queueing feeds to be updated, and calculating unread counts. Needless to say, even with nearly a thousand new users, I offloaded so much of the CPU-intensive work to the task servers that I didn't have a single problem in serving requests.

This is a big deal. The task server was overwhelmed (partially due to a bug, but partially because I was fetching tens of thousands of new feeds), but everybody who wanted to see NewsBlur still could. Their web requests, and loading each feed, were near instantaneous. It was wonderful to watch it happen, knowing that everybody was being served.

CPU - Past year

Clearly, bugs have been fixed, and CPU-intensive work has been offloaded to task servers.

Load average - Past week

The load of the server went up and stayed up. Why did it not fall back down? Because the app server is calculating unread counts, it has more work to do even after the users are gone. This will become a pain point when one app server is not enough for the hundreds of concurrent users NewsBlur will soon have. But luckily, app servers are the easiest to scale out, since each user will only use one app server at a time, so the data only has to be consistent on that one server, as it propagates out to the other app servers (which may become db shards, too).

# of feeds and subscriptions - Past week

Economies of scale. The more feeds I have, the more likely a subscription to a feed will be on a feed that already exists. I want that yellow line to run off into space, leaving the green line to grow linearly. It's fewer feeds to fetch.

Memory - Past week

Memory doesn't move, because I'm being CPU bound. I'm not actually moving all that much more data around. I use gunicorn to rotate my web workers, so NewsBlur's few memory leaks can be smoothed over.

MongoDB Operations - Past week

I use MongoDB to serve stories. All indexes, no misses (there's a graph for this I won't bother showing). You can extrapolate traffic through this graph. Sure, you don't know average feeds per user, but you can take a guess.

My Way of Building NewsBlur

In order to build all of the separate pieces, I broke everything down into chunks that could be written down and crossed off. Literally written down. I have all of my priorities from the past 7 months. It's both a motivator and estimator. I've learned how to estimate work load far better than back in May, when these priorities start. I finish more of what I tried to start.

The way it works is simple: write down a priority for the month it's going to be built in, number it, then cross it off if it gets built before the end of the month. You get to go back and see how much you can actually do, and what it is you wanted to build. This means I'm setting myself up for a pivot every month, when I re-evaluate what it is I'm trying to build.

Google Reader as a Competitor

Lastly, what more could you ask for? A prominent competitor, known to every Gmail user as the empty inbox link. Feed reading is a complicated idea made simple by having most users already exposed to a product that fulfills the feed reading need. By improving over that experience, users can directly compare, instead of having to learn NewsBlur on top of learning how to use RSS and track every site you read.

If your space has a major competitor and the barrier to entry is an OAuth import away, then consider yourself lucky. Anybody can try your product and become paid customers in moments. It's practically a Lotus123 to Excel import/export, except you don't need to buy the software before you try it out.

Going Forward

I'm half-way to being profitable. I only need 35 more premium subscribers. But so far, people are thrilled about the work I'm doing. Here are some tweets from a sample of users:

I'm e-mailing blogs, chatting with folks who have a blog influence, and most importantly, continuing to launch new features and fix old ones. Thanks to Hacker News, I get to appeal to a graceful and sharp audience. And good looking.

I'm on Twitter as @samuelclay, and I'd love to hear from you.

Aug 22

There are Two Paper Towel Rolls

It's almost time to restock, but the shelf can only hold 5 rolls, so you might as well restock at an appropriate time. But you have to choose which of the two remaining rolls is going in the business end of the side-gripping dispenser.

I can choose the larger of the two rolls. The Mega-Roll. Or I can choose the standard size, which is visibly puny compared to the bigger choice. If you know the answer, it seems obvious, and that's because it's an obvious answer.

But it's not so obvious if you start thinking about why choose one in the first place. The larger roll is larger, but does that mean it should go first simply because it is preferable? The assumption is that you don't like changing rolls often and you don't think larger rolls look or work any differently than their smaller counter-part.

And maybe the smaller roll has preference, just to get it out of the way for more Megas when it's time to buy more. You need to remember to buy more. What causes you to remember to buy more? Absence or a dwindling stock. Once you get down to having one left, and it gets placed into service, you commit to memory that you need to stock up next time you remember. It's a modified version of The Game that you play with yourself, except that by remembering, you win.

The smaller roll goes in first, so that at exhaustion the larger roll has a longer opportunity for you to remember to buy more. Nothing shocks you more than an absence.

Jul 18

Migrating Django from MySQL to PostgreSQL the Easy Way

I recently moved NewsBlur from MySQL to PostgreSQL for a variety of reasons, but most of all I want to use connection pooling and database replication using Slony, and Postgres has a great track record and community. But all of my data was stored in MySQL and there is no super easy way to move from one database backend to another.

Luckily, since I was using the Django ORM, and with Django 1.2's multi-db support, I can use Django's serializers to move the data from MySQL's format into JSON and then back into Postgres.

Unfortunately, If I were to use the command line, every single row of my models has to be loaded into memory. Issuing commands like this:

python manage.py dumpdata --natural --indent=4 feeds > feeds.json

would take a long, long time, and it wouldn't even work since I don't have even close to enough memory to make that work.

Luckily, the dumpdata and loaddata management commands are actually just wrappers on the internal serializers in Django. I decided to iterate through my models and grab 500 rows at a time, serialize them and then immediately de-serialize them (so Django could move from database to database without complaining).

import sys
from django.core import serializers

def migrate(model, size=500, start=0):
    count = model.objects.using('mysql').count()
    print "%s objects in model %s" % (count, model)
    for i in range(start, count, size):
        print i,
        sys.stdout.flush()
        original_data =  model.objects.using('mysql').all()[i:i+size]
        original_data_json = serializers.serialize("json", original_data)
        new_data = serializers.deserialize("json", original_data_json, 
                                           using='default')
        for n in new_data:
            n.save(using='default')

migrate(Feed)

This assume that you have both databases setup in your settings.py like so:

DATABASES = {
    'mysql': {
        'NAME': 'newsblur',
        'ENGINE': 'django.db.backends.mysql',
        'USER': 'newsblur',
        'PASSWORD': '',
    },
    'default': {
        'NAME': 'newsblur',
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'USER': 'newsblur',
        'PASSWORD': '',
    }
}

Note that I changed my default database to the Postgres database, because otherwise some management commands would still try to run on the default MySQL database. This is probably resolved and I didn't do something right, but when I migrated, I changed Postgres to be the default database.

I just run the short script in the Django console and wait however long it takes. This script prints out which set it's working on, so you can at least track the progress, which might take a long, long time, but is much less prone to crashing like dumpdata and loaddata.

A word of warning to those with large datasets. Instead of iterating straight through the table, see if you have a handier index already built on the table. I have a table with a million rows, but there are a few indices which can quickly find stories throughout the table, rather than having to order and offset the entire table by primary key. Adapt the following code to suit your needs, but notice that I use an index on the Feed column in the Story table.

import sys
from django.core import serializers

def migrate_with_model(primary_model, secondary_model, offset=0):
    secondary_model_data = secondary_model.objects.using('mysql').all()
    for i, feed in enumerate(secondary_model_data[offset:].iterator()):
        stories = primary_model.objects.using('mysql').filter(story_feed=feed)
        print "[%s] %s: %s stories" % (i, feed, stories.count())
        sys.stdout.flush()
        original_data = serializers.serialize("json", stories)
        new_data = serializers.deserialize("json", original_data, 
                                           using='default')
        for n in new_data:
            n.save(using='default')

migrate_with_model(primary_model=Story, secondary_model=Feed)

This makes it much faster, since I only have to sort a few hundreds records rather than the entire Story table and its million rows.

Also of note is that while all of the data made it into the Postgres tables, the sequences (counts) were all off. Many were at 0. To remedy this easily, just use the count of the table itself and store it in the sequence table, like so:

select setval('rss_feeds_tag_id_seq', max(id)) from rss_feeds_tag;
select setval('analyzer_classifierauthor_id_seq', max(id)) from analyzer_classifierauthor;            
select setval('analyzer_classifierfeed_id_seq', max(id)) from analyzer_classifierfeed;              
select setval('analyzer_classifiertag_id_seq', max(id)) from analyzer_classifiertag;               
select setval('analyzer_classifiertitle_id_seq', max(id)) from analyzer_classifiertitle;             
select setval('analyzer_featurecategory_id_seq', max(id)) from analyzer_featurecategory;

I just made a quick text macro on the table names. This quickly set all of the sequences to their correct amounts.

Jun 28

NewsBlur: Most Watched This Week

It's always nice to see that after working on a project for 13 months, people are finally starting to use it. The source behind NewsBlur is available on GitHub: http://github.com/samuelclay/NewsBlur/. And recently, in response to a Hacker News thread about why RSS readers sucks, I linked to NewsBlur and explained my rationale:

I think I created a very nice feed reading experience with NewsBlur: http://www.newsblur.com.

It shows the original site, allows you to read as you normally would, but keeps track of the stories you're scrolling past.

It also allows you to filter stories based on what you like and dislike about them: words/phrases in the title, tags and categories, authors, and the publisher themselves. There is a slider that allows you to show/hide stories based on this filter. It's very fast, too.

I am writing an iPhone app so you can use NewsBlur everywhere. It's just a hobby project, and people have so far been impressed. But I would love for NewsBlur to become a useful tool that people choose to use.

I wrote it because I was also dissatisfied with readers, especially Google Reader. I also knew Python (Django!), JavaScript, and wanted to put them together to test my abilities.

Currently, I am writing the iPhone app that will allow NewsBlur to be useful to a significant portion of Internet users who read RSS. Everybody that I have talked to says they are waiting for a good mobile version before they sink in time and curation into NewsBlur. Let's hope I am not underestimating when I say 1-2 months.

Mar 24

Code Snippet: jQuery Edit In Place

There are many solutions to the edit-in-place problem, but I wanted to make an easy solution that wasn't as complicated as some of the other edit-in-place JavaScript scripts, like jEditable.

Features:

  1. Detects surroundings and keeps the input container as either a block or inline display.
  2. Highlights text if it is the original text. If the text has changed, the entire text is not highlighted on edit.
  3. Easy customizable and styleable.

Demo

Test Input: Click here to change this text.

JavaScript Code

(function($) {
    $.fn.extend({
        edit_in_place: function(opts, callback) {
            var self = this;
            var defaults = {
                'input_type': 'text'
            }
            var options = $.extend({}, defaults, opts);

            return this.each(function() {
                var $this = $(this);
                var $input;
                var original_value = $this.html().replace(/<br.*?>/g, '\n');
                var original_display = $this.css('display');

                $this.bind('click', function() {
                    var starting_value = $this.html().replace(/<br.*?>/g, '\n');

                    if (options['input_type'] == 'text') {
                        $input = $.make('input', { type: 'text', name: 'eip_input', value: starting_value });
                    } else if (options['input_type'] == 'textarea') {
                        $input = $.make('textarea', { name: 'eip_input' }, starting_value);
                    }

                    var $form = $.make('div', { className: 'eip-container' }, [
                        $input,
                        $.make('button', { className: 'eip-submit' }, 'OK'),
                        $.make('button', { className: 'eip-cancel' }, 'Cancel')
                    ]);

                    $this.css({'display': 'none'});
                    $this.after($form);
                    $input.focus();
                    if (original_value == starting_value) {
                        $input.select();
                    }

                    var restore_input = function(input) {
                        return function($this, $form) {
                            $this.css({'display': original_display});
                            $form.empty().remove();
                            if (input) {
                                $this.html(input.replace(/[\n\r]+/g, "<br /><br />"));
                                $.isFunction(callback) &amp;&amp; callback.call(self, input);
                            }
                        }($this, $form);
                    };

                    setTimeout(function() {
                        $(document).one('click.edit_in_place', function() {
                            restore_input($input.val());
                        });
                        $form.click(function(e) {
                            if (e.target.className == 'eip-cancel') {
                                restore_input();
                                $(document).unbind('click.edit_in_place');
                            } else if (e.target.className == 'eip-submit') {
                                restore_input($input.val());
                                $(document).unbind('click.edit_in_place');
                            }
                            e.stopPropagation;
                            return false;
                        });
                    }, 10);
                });

            });
        }
    });

    $.extend({

        make: function(){
            var $elem,text,children,type,name,props;
            var args = arguments;
            var tagname = args[0];
            if(args[1]){
                if (typeof args[1]=='string'){
                    text = args[1];
                }else if(typeof args[1]=='object' &amp;&amp; args[1].push){
                  children = args[1];
                }else{
                    props = args[1];
                }
            }
            if(args[2]){
                if(typeof args[2]=='string'){
                    text = args[2];
                }else if(typeof args[1]=='object' &amp;&amp; args[2].push){
                  children = args[2];
                }
            }
            if(tagname == 'text' &amp;&amp; text){
                return document.createTextNode(text);
            }else{
                $elem = $(document.createElement(tagname));
                if(props){
                    for(var propname in props){
                      if (props.hasOwnProperty(propname)) {
                            if($elem.is(':input') &amp;&amp; propname == 'value'){
                                $elem.val(props[propname]);
                            } else {
                                $elem.attr(propname, props[propname]);
                            }
                        }
                    }
                }
                if(children){
                    for(var i=0;i&lt;children.length;i++){
                        if(children[i]){
                            $elem.append(children[i]);
                        }
                    }
                }
                if(text){
                    $elem.html(text);
                }
                return $elem;
            }
        }

    });
})(jQuery);

To use this code, simply use this HTML, CSS, and small JavaScript snippet:

<div class="eip">
    Test Input: <span class="eip-text">Click here to change this text.</span>
</div>

And this CSS:

.eip {
    font-family: Helvetica;
    font-size: 16px;
}

.eip .eip-text {
    font-weight: bold;
    padding: 2px 3px;
    border: 1px solid white;
}

.eip .eip-container {
    display: inline;
}

.eip input {
    font-family: Helvetica;
    font-size: 16px;
    font-weight: bold;
    padding: 2px;
    border: 1px solid #A0A0A0;
    display: inline;
    width: 250px;
}

And this simple piece of JavaScript, which includes a callback function that has the same scope as the original selectors:

$(document).ready(function() {
    $('.eip .eip-text').edit_in_place({}, function() {
        var $this = $(this);
        $this.animate({'backgroundColor': 'orange'}, {'duration': 300, 'queue': false, 'complete': function() {
            $this.animate({'backgroundColor': 'white'}, {'duration': 300, 'queue': false});
        }});
    });
});

Note that I am animating background colors in this small JavaScript snippet. To animate colors, you need John Resig's excellent jQuery.color.js.

Mar 02

Code snippet: Stopping a jQuery AJAX Request

I want JavaScript to feel as smooth as a native application. I think scrolling is one of the largest issues, but this code snippet is more about aborting the jQuery AJAX event before it has a chance to complete.

There's no good documentation in the jQuery docs about how to do this. other than to just use this command on an existing AJAX request:

var request = $.ajax('/url', data, callback); request.abort();

That doesn't work. Well, it does work, but if you try to run it again or synchronously with other requests, you'll run into issues.

The issues are non-trivial, but avoidable. I'll cut to the chase; I came up with a solution, then found that somebody did it better and more correct.

Rather than spreading incorrect (rather, incomplete) code, I'll just show the proper way to do it and then link to the source.

_isAbort: function(xhr, o){ var ret = !!( o.abortIsNoSuccess && ( !xhr || xhr.readyState === 0 || this.lastAbort === o.xhrID ) ); xhr = null; return ret; },

That's a lot of work. Don't bother, just use jquery.ajaxManager v.3.0: http://www.protofunc.com/scripts/jquery/ajaxManager3/

Note, however, that if you just google "jquery ajax manager" or some variant, you will end up at the old version, which is at: http://www.protofunc.com/scripts/jquery/ajaxManager/. They could do some work on their google juice pointing to the latest version.

Hope this helps somebody else, even if part of a google search for "jquery ajax stop request" someday.

Feb 16

A jQuery Plugin: Default Values for Input Fields

One of the best ways to write code that you tend to have to re-use is to put it in the public domain. That way when you need it again, it's a Google search away from your own blog.

This is a rather simple working example of default text on an input field. Click on the field, the text disappears, only to reappear if the user clicks somewhere else on the page without typing. The input also has a special class signifying that it is empty, so you can style the empty input.

Demo

JavaScript Code

(function($) { $.fn.extend({ input_default: function(default_text, opts) { if (typeof default_text !== 'string') { opts = default_text; } else if (!opts) { opts = { 'default_text': default_text }; } else { $.extend(opts, {'default_text': default_text}); } var defaults = { 'default_text': 'Type here...', 'class_name': 'empty-input' }; var options = $.extend({}, defaults, opts); return this.each(function () { var $this = $(this); if ($this.val() == '' || $this.val() == options['default_text']) { $this.addClass(options['class_name']) .val(options['default_text']); } $this.bind('focus', function() { if ($this.val() == options['default_text']) { $this.val('') .removeClass(options['class_name']); } else { $this.select(); } }).bind('blur', function() { if ($.trim($this.val()) == '') { $this.val(options['default_text']) .addClass(options['class_name']); } else { $this.removeClass(options['class_name']); } }); }); } }); })(jQuery);

Usage

First, the HTML you can use: You can call `input_default` with no arguments and get the defaults: $('.text').input_default(); Specify an optional string or class: $('.text').input_default('Enter text here...', {'class_name': 'empty'}); Here is some sample CSS to use: .default-text { border: 1px solid #C0C0C0; padding: 2px; font-weight: bold; font-size: 14px; } .empty-input { color: #A0A0A0; } .default-text-label { font-size: 16px; font-weight: bold; color: #303030; }
Jan 18

A Faulty Heist: A Storybird

This Storybird is written by thesundaybest, found on twitter: @thesundaybest.

A Faulty Heist by thesundaybest on Storybird

Storybirds like this remind me why I love working with a community of artists and children's literature.

Jan 15

Syntax highlighting for Django using Pygments

The wonderful django-mingus includes a few separate syntax highlighters, including one from django-sugar. However, the pygmentize template filter only works on <code> blocks and tries to guess the language.

A better syntax would be to include the language in the class of the code block, like so:

<code class="python"> 
    import this 
    print [r for r in range(0,10,2)] 
</code> 

You can use this template filter, which is adapted from the Pygments Rendering Template Filter at Django Snippets.

import re 
import pygments 
from django import template 
from pygments import lexers 
from pygments import formatters 
from BeautifulSoup import BeautifulSoup 
 
register = template.Library() 
regex = re.compile(r'<code(.*?)>(.*?)</code>', re.DOTALL) 
 
@register.filter(name='pygmentize') 
def pygmentize(value): 
    last_end = 0 
    to_return = '' 
    found = 0 
    for match_obj in regex.finditer(value): 
        code_class = match_obj.group(1) 
        code_string = match_obj.group(2) 
        if code_class.find('class'): 
            language = re.split(r'"|\'', code_class)[1] 
            lexer = lexers.get_lexer_by_name(language) 
        else: 
            try: 
                lexer = lexers.guess_lexer(str(code)) 
            except ValueError: 
                lexer = lexers.PythonLexer() 
        pygmented_string = pygments.highlight(code_string, lexer, formatters.HtmlFormatter()) 
        to_return = to_return + value[last_end:match_obj.start(0)] + pygmented_string 
        last_end = match_obj.end(2) 
        found = found + 1 
    to_return = to_return + value[last_end:] 
    return to_return 

This is a template filter, which can be applied like so:

{{ code|pygmentize }} You can read more about custom tempalte filters at the Django Project: Writing Custom Template Filters.
Jan 13

NewsBlur: Feed Reader with Artificial Intelligence

Every two to three years I start a new project. Having recently completed the SunRayLab blog, I turned to another problem that I have been having since i started visiting blogs and other sites through the lens of my RSS feed reader, NetNewsWire.

I am starting a new project called NewsBlur. NewsBlur is an RSS feed reader with intelligence. See NewsBlur in action.

My problem is simple: I subscribe to a few dozen blogs. I do not want to read all of their many stories, yet my unread count keeps climbing. Some blogs post frequently while other blogs post once a month. I want a feed reader that easily trims the undesirable fat from my feeds.

This has been done in varying degrees before, but each implementation has been lacking for any number of reasons:
  1. The reading interface was convulted, slow, difficult, too busy, or rough.
  2. The intelligence used to pare down my feeds to just what I want was either difficult to setup, hard to train, overly-burdensome, or even just plain stupid.
  3. Marking feeds and stories as typical of what I want in the future has never been done.
I am solving this problem in a unique way.

The challenge in creating an interesting experience that is different from any other experience is what drives me to work on NewsBlur. I want a service that works for me. I expect that before I finish, I will already have abandoned my tried-and-true NetNewsWire (on both Mac OS and iPhone) in favor of NewsBlur.

I will use this blog space as a means to communicate advances in NewsBlur, as well as methodologies, practices, and thoughts behind the creation of NewsBlur.

My hope is that NewsBlur becomes part of a community that further enhances the news and blog reading experience for everybody.
Jan 12

Powered by Kevin Fricovsky's django-mingus

Since 1999 I have had a presence on the web. I bought conesus.com back then and used it to link to projects, code samples, writings, and photographs. I now own conesus.com, samuelclay.com, and ofbrooklyn.com, all of which now serve different purposes.

My primary blog platform has been a hand-coded blog engine called SunRayLab that I used for ofbrooklyn.com for the past three years. It's time to retire it, as great as I have felt I made it. The whole caboodle is in aging PHP and the future is definitely not there.

Thanks to Kevin Fricofsky's hard work, I have django-mingus to use for a blog. It combines a few dozen different django applications with a minimal amount of glue. Surprisingly, everything hums along nicely and Kevin listens and acts quickly, so I have faith in the future of this platform, for once.

It is humbling to take down my old creations, but it also signifies a maturity inherent in code obsolescence. Code ages, much like a house. If you don't take care of it and upgrade it with working utilities, then it will begin to crumble. Whether the water main breaks (perhaps you upgraded a dependency elsewhere on your server and the new version breaks your current code), or the appliances are no longer shiny and clean (and you want code highlighting in your blog platform, but that requires a lot of tangentially work that cascades into more work quicker than you can say 'enough').

My goal is to post code, help others in a way similar to how I have been helped by developers posting tidbits, tutorials, and findings. I plan to write an occasionally essay, since I tend to disagree with opinions fairly often and want something to show for my trouble. I may even post photos.

The future is bright and my fingers are not yet tired.

Oct 01

Raphaël Demo - Cross Browser Vector Graphics - Topic Connections Graph

SVG, an open, standardized vector graphics markup language is now supported on 30% of all web browsers. The other 70% of browsers (specifically, Internet Explorer), support VML, a close cousin of SVG. If you write both SVG and VML, you can provide rich, interactive vector graphics > 99% of all browsers.

Raphaël JavaScript Library does all the heavy lifting for the developer. Write your circles, paths, animations, and interactions (events, mouseovers, clicks) in Raphaël, and it will write compatible SVG and VML that works almost everywhere. I have contributed to Raphaël and am currently presenting a paper and a workshop on Raphaël at SVG Open 2009 at Google HQ in Mountain View, California.

I wrote this demo for Daylife, but this demo uses a number of features in Raphaël, showcasing a number of features that can be used to make interactive graphics that are unlike anything else natively supported by almost all web browsers. This means no plug-ins or installs. No Flash. Just what is already built into every web browser.

Demo - Topic Connections Graph

Code

1) Download jquery.topicconections.js

2) This code should go into your <style>:


.SO-module .SO-name {
    display: block;
    font-size: 10px;
    line-height: 11px;
    margin: 5px 0;
    position: absolute;
    text-align: center;
    visibility: hidden;
    width: 80px;
    color: #416799;
    font-family: "Lucida Grande";
}

.SO-module a {
    text-decoration: none;
    background: none;
}

.SO-active a {
    text-decoration: underline;
}

.SO-module ol li {
    list-style: none !important;
}

.SO-module {
    margin: 20px;
    position: relative;
}

3) Include these scripts in your <head>:


<script src="http://raphaeljs.com/raphael.js" 
        type="text/javascript" charset="utf-8"></script>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js" 
        type="text/javascript" charset="utf-8"></script>    
<script src="/scripts/jquery.topicconnections.js" 
        type="text/javascript" charset="utf-8"></script>

4) This code goes inline:


<div class="SO-module">
    <div id="SO-connections-graph-wrapper">
        <div id="connections_graph_186817" class="SO-connections-graph"></div>
        <ol class="SO-topics">
            <li class="SO-topic">
                <a class="SO-name" href="http://beta.daylife.com/topic/White_House">White House</a>
            </li>
            <li class="SO-topic">
                <a class="SO-name" href="http://beta.daylife.com/topic/U.S._Republican_Party">U.S. Republican Party</a>
            </li>
            <li class="SO-topic">
                <a class="SO-name" href="http://beta.daylife.com/topic/U.S._Democratic_Party">U.S. Democratic Party</a>
            </li>
            <li class="SO-topic">
                <a class="SO-name" href="http://beta.daylife.com/topic/U.S._Congress">U.S. Congress</a>
            </li>
            <li class="SO-topic">
                <a class="SO-name" href="http://beta.daylife.com/topic/George_W._Bush">George W. Bush</a>
            </li>
            <li class="SO-topic">
                <a class="SO-name" href="http://beta.daylife.com/topic/U.S._Senate">U.S. Senate</a>
            </li>
            <li class="SO-topic">
                <a class="SO-name" href="http://beta.daylife.com/topic/Economic_Recession">Economic Recession</a>
            </li>
        </ol>
    </div>
</div>

5) This JavaScript code also goes inline:


var images = [
    {
        'img': "/media/img/svg_demo/whitehouse.jpg",
        'text': "White House",
        'link': "http://beta.daylife.com/topic/White_House",
        's': "large"
    },
    {
        'img': "/media/img/svg_demo/repub.jpg",
        'text': "U.S. Republican Party",
        'link': "http://beta.daylife.com/topic/U.S._Republican_Party",
        's': "large"
    },
    {
        'img': "/media/img/svg_demo/demo.jpg",
        'text': "U.S. Democratic Party",
        'link': "http://beta.daylife.com/topic/U.S._Democratic_Party",
        's': "large"
    },
    {
        'img': "/media/img/svg_demo/congress.jpg",
        'text': "U.S. Congress",
        'link': "http://beta.daylife.com/topic/U.S._Congress",
        's': "medium"
    },
    {
        'img': "/media/img/svg_demo/bush.jpg",
        'text': "George W. Bush",
        'link': "http://beta.daylife.com/topic/George_W._Bush",
        's': "medium"
    },
    {
        'img': "/media/img/svg_demo/senate.jpg",
        'text': "U.S. Senate",
        'link': "http://beta.daylife.com/topic/U.S._Senate",
        's': "small"
    },
    {
        'img': "/media/img/svg_demo/econ.jpg",
        'text': "Economic Recession",
        'link': "http://beta.daylife.com/topic/Economic_Recession",
        's': "small"
    }
];

var topic_image = {
    'img': "/media/img/svg_demo/obama.jpg",
    's': 'xlarge'
};

var center_x = 125;
var center_y = 125;
var offset_x = 45;
var offset_y = 45;
var $graph = $("#connections_graph_186817")[0];
var graph_x = 340;
var graph_y = 340;

var sizes = {
    'small': 35,
    'medium': 55,
    'large': 75,
    'xlarge': 100
};

var topic_graph = new TopicConnectionGraph($graph, graph_x, graph_y, 
                                           center_x, center_y,
                                           offset_x, offset_y, sizes);
topic_graph.runner(images, topic_image);
Sep 01

Why Bother Going to College

In response to James Padolsey's blog post on his apprehension to studying Java before his first-year studying computer science.

University is unlike anything youve done, and unfortunately anything else youre going to do unless you really love the study of computer science. I recently graduated from university with a degree in Computer Engineering, taking roughly similar courses as a Comp Sci (except we also took harder classes related to engineering that were straight out of unrelated engineering disciplines, like thermodynamics, statics (bridges), and materials science).

Almost everybody enrolled in the computer science department started out years before school, programming in somethingusually web development (esp. Javascript, the bad parts). A healthy number of them drop out of the program and switch to math/physics/management or even art history. They drop comp sci not because they are dumb, or have a slower intake of the sheer volume of dense material fed to them. Its because they discover that the internals, things like how primitives are actually allocated in different ways in memory, sometimes even to your advantage to use the right one, are not as interesting as they had hoped. Low-level course material, such as transistor logic, VHDL, networking, operating systems (easily one of the most challenging and rewarded courses taught in computer science), and systems programming, are all above and beyond what most contractor/freelancers will face while they are busy earning 25% more than their comp sci cousins.

But the difference is worth a lot. Take a look at that one, systems programming. Here is a short description of whats taught:

EECS 337 Systems Programming - 4 credits Lexical analyzers; symbol tables and their searching; assemblers, one-pass and two-pass, conditional assembly, and macros; linkers and loaders; interpreters, pcodes, threaded codes; introduction to compilation, grammar, parsing, and code generation; preprocessors; text editors, line-oriented and screen-oriented; bootstrap loaders, ROM monitors, interrupts, and device drivers. Laboratory. Prereq: EECS 233 and EECS 281.

If you want to write code that does anything that involves automated decision making, the true back-end of great companies to work for or start, you need to actually study and learn about these low-levels to build better decision trees (data structures alone make it worthwhile) and neural networks. There are so many fascinating topics out there, and they are all more expressive than having to design web sites for clients.

I absolutely love the study of artificial intelligence, and am working on a project where the meat of the project is in the AI that feeds the site, but I dont write algorithms or architect systems for a living. I write Javascript code (and Python on the back-end serving up the front-end), designing a web app for publishers to create photo galleries. Its not quite the academic challenge I get on my project, but you also have to be very, very good at computer science to get recognized and to make big contributions, even to small companies (think startups).

Computer Science is a rich and engaging degree, and while it isnt for many who think it is, those who stick with it all 4 (to 5) years end up very happy. This is the golden age of software. Theres a lot of upheaval which will forge new empires, and I bet something as notable as the few winners of the California gold rush of the 19th century didnt even have the prestige that good engineers have. Think of all the famous and very wealthy nerds. Think about that when youre slogging through decimal floating-point arithmetics in binary by hand.

Jul 14

Presenting at SVG Open 2009: Workshop on Raphaël JS

We were accepted! Here is our finished abstract, also available on the SVG Open 2009 workshop page: http://www.svgopen.org/2009/registration.php?section=workshops.

Dmitry Baranovskiy - http://dmitry.baranovskiy.com - dmitry@baranovskiy.com
Samuel Clay - http://www.ofbrooklyn.com - samuel@ofbrooklyn.com

The power of web browsers has reached the point when it is easy to use native vector graphics without any 3rd party plugins and installs. In this session, we will discover how to build rich graphics in your web app using Raphal (http://raphaeljs.com/) JavaScript vector library, the features Raphal provides, and walk through how to apply these features into a fully built module.

This course will cover a number of features ranging in complexity from basic layout to event-driven interaction. The module will be a moderately complex application consisting of parts SVG, Javascript, jQuery, and HTML. While Javascript experience is expected, it is not necessary, considering the syntax of Javascript in this capacity is not far different than many other languages.

The module, which will be used to demonstrate a number of techniques in creating both interactive and layered elements in SVG, will be sufficiently advanced as to cover many topics, but so in-depth that it cannot be built entirely from scratch in the time allotted for the course--including time used for demonstrating Raphal.

The 150 minute course will consist of 3 parts, with questions taken throughout:

  • Introduction to Raphael (35 min)
    • 5 minutes: Introduction and Overview of the People Connection widget
    • 10 minutes: Looking at the Raphael JS Library's methods and documentation
    • 10 minutes: Setting up data for the module to be used in drawing
    • 10 minutes: Loading data into Raphael
  • Drawing the widget (45 min)
    • 10 minutes: Images
    • 15 minutes: Shapes
    • 20 minutes: Lines, straight and curved (quadratic and bicubic)
  • Interaction model (60 min)
    • 20 minutes: Handling user/triggered events
    • 15 minutes: Animation
    • 25 minutes: Interacting with the DOM and jQuery

There is a time buffer built into each part, and if there is extra time, we will be prepared to discuss:

  • Raphael versus other SVG/Javascript libraries
  • Other projects built on Raphael
  • Browser compatibility with Raphael

For each section of the course, we plan to discuss:

  • Implementation and coding conventions
  • Potential pitfalls with certain techniques (browser incompatibilities, misconceptions, and common mistakes)
  • Building the widget by coding in real-time (and not using blobs of code already written).

The flow of this walk-through course will be aligned with the building of the module, where we will start with a empty file and progress over each step used in creating the overall effect and feel of the finished widget. Each piece of the widget puzzle will be complemented with a demonstration of the incremental improvement. At the end of the course, a complete module, with all of its pieces, will be fully functional and fully explained. Participants will also have all of these Raphal resources to use in their own SVG projects.

Jun 04

Raphael Plugin: Animation Easing

Raphael is a wonderful Javascript library, used for creating vector graphics, and interacting with SVG/VML elements through Javascript. Raphael allows the developer to interact with these elements through a number of complex animations. 

Until now, the only speed for an animation was a linear movement from one state to another. This plugin extends the animation of Raphael to allow for easing animations. Derived from jQuery Easing Plugin (version 1.3): http://gsgd.co.uk/sandbox/jquery/easing/.

NOTE: As of June 2009, this plugin requires a forked version of RaphaelJS. You can download it here: https://github.com/conesus/raphael/tree. Hopefully, these forked changes will be merged with the master Raphael JS code.

Demo:

View a demo of animation easing techniques here: http://demos.daylife.com/samuel/svg_photo_flipper_prototype.xhtml

To use:

  1. Download raphael.easing.1.3.js from: http://github.com/conesus/raphael-easing/tree/master.
  2. Include the raphael.easing.1.3.js file after your raphael.js file.
  3. Specify the easing technique in the .animate() function, like so:

    Element.animate({"scale": [1.25, 1.25]}, 750, "easeOutCubic", callback);

Available easing techniques:

  • linear
  • easeInQuad
  • easeOutQuad
  • easeInOutQuad
  • easeInCubic
  • easeOutCubic
  • easeInOutCubic
  • easeInQuart
  • easeOutQuart
  • easeInOutQuart
  • easeInQuint
  • easeOutQuint
  • easeInOutQuint
  • easeInExpo
  • easeOutExpo
  • easeInOutExpo
  • easeInElastic
  • easeOutElastic
  • easeInOutElastic
  • easeInBounce
  • easeOutBounce
  • easeInOutBounce
May 18

Proposal for SVG Open 2009

The SVG Open 2009 is in October at Google's campus in California. I recently submitted my proposal for a course on writing SVG through Javascript and the Raphael Javascript Library. Below is my proposal. Let's hope it is approved and I am able to continue work in SVG.

I am proposing to teach a course about building an interactive SVG module on the web through the use of the Raphael Javascript Library. From start to finish, the course will walk attendees through the process of creating an interactive widget, with a number of features ranging in complexity from basic layout to event-driven interaction. The widget will be a moderately complex application consisting of parts SVG, Javascript, jQuery, and HTML. While Javascript experience is expected, it is not necessary, considering the syntax of Javascript in this capacity is not far different than many other languages.

The widget, which will be used to demonstrate a number of techniques in creating both interactive and layered elements in SVG, is a "People Connection" module which shows a graph of how various people are connected to each other. See Figure 1 and Figure 2 for illustrations, and see http://www.daylife.com/topic/Barack_Obama/beta#title=Connections for a demonstration.

The 150 minute course will consist of 3 parts, with questions taken throughout:

  • Introduction to Raphael (35 min)
    • 5 minutes: Introduction and Overview of the People Connection widget
    • 10 minutes: Looking at the Raphael JS Library's methods and documentation
    • 10 minutes: Setting up data for the module to be used in drawing
    • 10 minutes: Loading data into Raphael
  • Drawing the widget (45 min)
    • 10 minutes: Images
    • 15 minutes: Shapes
    • 20 minutes: Lines, straight and curved (quadratic and bicubic)
  • Interaction model (60 min)
    • 20 minutes: Handling user/triggered events
    • 15 minutes: Animation
    • 25 minutes: Interacting with the DOM and jQuery

There is a time buffer built into each part, and if there is extra time, I will be prepared to discuss:

  • Raphael versus other SVG/Javascript libraries
  • Other projects built on Raphael
  • Browser compatibility with Raphael

For each section of the course, I plan to discuss:

  • Implementation and coding conventions
  • Potential pitfalls with certain techniques (browser incompatibilities, misconceptions, and common mistakes)
  • Building the widget by coding in real-time (and not using blobs of code already written).

The flow of this walk-through course will be aligned with the building of the People Connection widget, where I will start with a empty file and progress over each step used in creating the overall effect and feel of the finished widget. Each piece of the widget puzzle will be complemented with a demonstration of the incremental improvement. At the end of the course, a complete widget, with all of its pieces, will be fully functional and fully explained.

Samuel Clay is a software engineer living in Brooklyn working on DocumentCloud, an open-source repository of primary source documents contributed by journalists, and NewsBlur, a feed reader with artificial intelligence.

His latest project is New York Field Guide, a photo-blog documenting New York City's 90 historic districts. You can read about his past and present projects at samuelclay.com.

Follow @samuelclay on Twitter.

You can email Samuel Clay at samuel@ofbrooklyn.com. He loves receiving email from new people. Who doesn't?

Elsewhere