OfBrooklyn.com Samuel Clay's public record

I recently moved NewsBlur from MySQL to PostgreSQL for a variety of reasons, but most of all I want to use connection pooling and database replication using Slony, and Postgres has a great track record and community. But all of my data was stored in MySQL and there is no super easy way to move from one database backend to another.

Luckily, since I was using the Django ORM, and with Django 1.2's multi-db support, I can use Django's serializers to move the data from MySQL's format into JSON and then back into Postgres.

Unfortunately, If I were to use the command line, every single row of my models has to be loaded into memory. Issuing commands like this:

python manage.py dumpdata --natural --indent=4 feeds > feeds.json

would take a long, long time, and it wouldn't even work since I don't have even close to enough memory to make that work.

Luckily, the dumpdata and loaddata management commands are actually just wrappers on the internal serializers in Django. I decided to iterate through my models and grab 500 rows at a time, serialize them and then immediately de-serialize them (so Django could move from database to database without complaining).

import sys
from django.core import serializers

def migrate(model, size=500, start=0):
    count = model.objects.using('mysql').count()
    print "%s objects in model %s" % (count, model)
    for i in range(start, count, size):
        print i,
        sys.stdout.flush()
        original_data =  model.objects.using('mysql').all()[i:i+size]
        original_data_json = serializers.serialize("json", original_data)
        new_data = serializers.deserialize("json", original_data_json, 
                                           using='default')
        for n in new_data:
            n.save(using='default')

migrate(Feed)

This assume that you have both databases setup in your settings.py like so:

DATABASES = {
    'mysql': {
        'NAME': 'newsblur',
        'ENGINE': 'django.db.backends.mysql',
        'USER': 'newsblur',
        'PASSWORD': '',
    },
    'default': {
        'NAME': 'newsblur',
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'USER': 'newsblur',
        'PASSWORD': '',
    }
}

Note that I changed my default database to the Postgres database, because otherwise some management commands would still try to run on the default MySQL database. This is probably resolved and I didn't do something right, but when I migrated, I changed Postgres to be the default database.

I just run the short script in the Django console and wait however long it takes. This script prints out which set it's working on, so you can at least track the progress, which might take a long, long time, but is much less prone to crashing like dumpdata and loaddata.

A word of warning to those with large datasets. Instead of iterating straight through the table, see if you have a handier index already built on the table. I have a table with a million rows, but there are a few indices which can quickly find stories throughout the table, rather than having to order and offset the entire table by primary key. Adapt the following code to suit your needs, but notice that I use an index on the Feed column in the Story table.

import sys
from django.core import serializers

def migrate_with_model(primary_model, secondary_model, offset=0):
    secondary_model_data = secondary_model.objects.using('mysql').all()
    for i, feed in enumerate(secondary_model_data[offset:].iterator()):
        stories = primary_model.objects.using('mysql').filter(story_feed=feed)
        print "[%s] %s: %s stories" % (i, feed, stories.count())
        sys.stdout.flush()
        original_data = serializers.serialize("json", stories)
        new_data = serializers.deserialize("json", original_data, 
                                           using='default')
        for n in new_data:
            n.save(using='default')

migrate_with_model(primary_model=Story, secondary_model=Feed)

This makes it much faster, since I only have to sort a few hundreds records rather than the entire Story table and its million rows.

Also of note is that while all of the data made it into the Postgres tables, the sequences (counts) were all off. Many were at 0. To remedy this easily, just use the count of the table itself and store it in the sequence table, like so:

select setval('rss_feeds_tag_id_seq', max(id)) from rss_feeds_tag;
select setval('analyzer_classifierauthor_id_seq', max(id)) from analyzer_classifierauthor;            
select setval('analyzer_classifierfeed_id_seq', max(id)) from analyzer_classifierfeed;              
select setval('analyzer_classifiertag_id_seq', max(id)) from analyzer_classifiertag;               
select setval('analyzer_classifiertitle_id_seq', max(id)) from analyzer_classifiertitle;             
select setval('analyzer_featurecategory_id_seq', max(id)) from analyzer_featurecategory;

I just made a quick text macro on the table names. This quickly set all of the sequences to their correct amounts.

It's always nice to see that after working on a project for 13 months, people are finally starting to use it. The source behind NewsBlur is available on GitHub: http://github.com/samuelclay/NewsBlur/. And recently, in response to a Hacker News thread about why RSS readers sucks, I linked to NewsBlur and explained my rationale:

I think I created a very nice feed reading experience with NewsBlur: http://www.newsblur.com.

It shows the original site, allows you to read as you normally would, but keeps track of the stories you're scrolling past.

It also allows you to filter stories based on what you like and dislike about them: words/phrases in the title, tags and categories, authors, and the publisher themselves. There is a slider that allows you to show/hide stories based on this filter. It's very fast, too.

I am writing an iPhone app so you can use NewsBlur everywhere. It's just a hobby project, and people have so far been impressed. But I would love for NewsBlur to become a useful tool that people choose to use.

I wrote it because I was also dissatisfied with readers, especially Google Reader. I also knew Python (Django!), JavaScript, and wanted to put them together to test my abilities.

Currently, I am writing the iPhone app that will allow NewsBlur to be useful to a significant portion of Internet users who read RSS. Everybody that I have talked to says they are waiting for a good mobile version before they sink in time and curation into NewsBlur. Let's hope I am not underestimating when I say 1-2 months.

There are many solutions to the edit-in-place problem, but I wanted to make an easy solution that wasn't as complicated as some of the other edit-in-place JavaScript scripts, like jEditable.

Features:

1. Detects surroundings and keeps the input container as either a block or inline display.
2. Highlights text if it is the original text. If the text has changed, the entire text is not highlighted on edit.
3. Easy customizable and styleable.

Demo

JavaScript Code

(function($) {
    $.fn.extend({
        edit_in_place: function(opts, callback) {
            var self = this;
            var defaults = {
                'input_type': 'text'
            }
            var options = $.extend({}, defaults, opts);

            return this.each(function() {
                var $this = $(this);
                var $input;
                var original_value = $this.html().replace(/<br.*?>/g, '\n');
                var original_display = $this.css('display');

                $this.bind('click', function() {
                    var starting_value = $this.html().replace(/<br.*?>/g, '\n');

                    if (options['input_type'] == 'text') {
                        $input = $.make('input', { type: 'text', name: 'eip_input', value: starting_value });
                    } else if (options['input_type'] == 'textarea') {
                        $input = $.make('textarea', { name: 'eip_input' }, starting_value);
                    }

                    var $form = $.make('div', { className: 'eip-container' }, [
                        $input,
                        $.make('button', { className: 'eip-submit' }, 'OK'),
                        $.make('button', { className: 'eip-cancel' }, 'Cancel')
                    ]);

                    $this.css({'display': 'none'});
                    $this.after($form);
                    $input.focus();
                    if (original_value == starting_value) {
                        $input.select();
                    }

                    var restore_input = function(input) {
                        return function($this, $form) {
                            $this.css({'display': original_display});
                            $form.empty().remove();
                            if (input) {
                                $this.html(input.replace(/[\n\r]+/g, "<br /><br />"));
                                $.isFunction(callback) &amp;&amp; callback.call(self, input);
                            }
                        }($this, $form);
                    };

                    setTimeout(function() {
                        $(document).one('click.edit_in_place', function() {
                            restore_input($input.val());
                        });
                        $form.click(function(e) {
                            if (e.target.className == 'eip-cancel') {
                                restore_input();
                                $(document).unbind('click.edit_in_place');
                            } else if (e.target.className == 'eip-submit') {
                                restore_input($input.val());
                                $(document).unbind('click.edit_in_place');
                            }
                            e.stopPropagation;
                            return false;
                        });
                    }, 10);
                });

            });
        }
    });

    $.extend({

        make: function(){
            var $elem,text,children,type,name,props;
            var args = arguments;
            var tagname = args[0];
            if(args[1]){
                if (typeof args[1]=='string'){
                    text = args[1];
                }else if(typeof args[1]=='object' &amp;&amp; args[1].push){
                  children = args[1];
                }else{
                    props = args[1];
                }
            }
            if(args[2]){
                if(typeof args[2]=='string'){
                    text = args[2];
                }else if(typeof args[1]=='object' &amp;&amp; args[2].push){
                  children = args[2];
                }
            }
            if(tagname == 'text' &amp;&amp; text){
                return document.createTextNode(text);
            }else{
                $elem = $(document.createElement(tagname));
                if(props){
                    for(var propname in props){
                      if (props.hasOwnProperty(propname)) {
                            if($elem.is(':input') &amp;&amp; propname == 'value'){
                                $elem.val(props[propname]);
                            } else {
                                $elem.attr(propname, props[propname]);
                            }
                        }
                    }
                }
                if(children){
                    for(var i=0;i&lt;children.length;i++){
                        if(children[i]){
                            $elem.append(children[i]);
                        }
                    }
                }
                if(text){
                    $elem.html(text);
                }
                return $elem;
            }
        }

    });
})(jQuery);

To use this code, simply use this HTML, CSS, and small JavaScript snippet:

<div class="eip">
    Test Input: <span class="eip-text">Click here to change this text.</span>
</div>

And this CSS:

.eip {
    font-family: Helvetica;
    font-size: 16px;
}

.eip .eip-text {
    font-weight: bold;
    padding: 2px 3px;
    border: 1px solid white;
}

.eip .eip-container {
    display: inline;
}

.eip input {
    font-family: Helvetica;
    font-size: 16px;
    font-weight: bold;
    padding: 2px;
    border: 1px solid #A0A0A0;
    display: inline;
    width: 250px;
}

And this simple piece of JavaScript, which includes a callback function that has the same scope as the original selectors:

$(document).ready(function() {
    $('.eip .eip-text').edit_in_place({}, function() {
        var $this = $(this);
        $this.animate({'backgroundColor': 'orange'}, {'duration': 300, 'queue': false, 'complete': function() {
            $this.animate({'backgroundColor': 'white'}, {'duration': 300, 'queue': false});
        }});
    });
});

Note that I am animating background colors in this small JavaScript snippet. To animate colors, you need John Resig's excellent jQuery.color.js.

I want JavaScript to feel as smooth as a native application. I think scrolling is one of the largest issues, but this code snippet is more about aborting the jQuery AJAX event before it has a chance to complete.

There's no good documentation in the jQuery docs about how to do this. other than to just use this command on an existing AJAX request:

var request = $.ajax('/url', data, callback); request.abort();

That doesn't work. Well, it does work, but if you try to run it again or synchronously with other requests, you'll run into issues.

The issues are non-trivial, but avoidable. I'll cut to the chase; I came up with a solution, then found that somebody did it better and more correct.

Rather than spreading incorrect (rather, incomplete) code, I'll just show the proper way to do it and then link to the source.

_isAbort: function(xhr, o){ var ret = !!( o.abortIsNoSuccess && ( !xhr || xhr.readyState === 0 || this.lastAbort === o.xhrID ) ); xhr = null; return ret; },

That's a lot of work. Don't bother, just use jquery.ajaxManager v.3.0: http://www.protofunc.com/scripts/jquery/ajaxManager3/

Note, however, that if you just google "jquery ajax manager" or some variant, you will end up at the old version, which is at: http://www.protofunc.com/scripts/jquery/ajaxManager/. They could do some work on their google juice pointing to the latest version.

Hope this helps somebody else, even if part of a google search for "jquery ajax stop request" someday.

One of the best ways to write code that you tend to have to re-use is to put it in the public domain. That way when you need it again, it's a Google search away from your own blog.

This is a rather simple working example of default text on an input field. Click on the field, the text disappears, only to reappear if the user clicks somewhere else on the page without typing. The input also has a special class signifying that it is empty, so you can style the empty input.

Demo

JavaScript Code

Usage

This Storybird is written by thesundaybest, found on twitter: @thesundaybest.

A Faulty Heist by thesundaybest on Storybird

Storybirds like this remind me why I love working with a community of artists and children's literature.

The wonderful django-mingus includes a few separate syntax highlighters, including one from django-sugar. However, the pygmentize template filter only works on <code> blocks and tries to guess the language.

A better syntax would be to include the language in the class of the code block, like so:

<code class="python"> 
    import this 
    print [r for r in range(0,10,2)] 
</code> 

You can use this template filter, which is adapted from the Pygments Rendering Template Filter at Django Snippets.

import re 
import pygments 
from django import template 
from pygments import lexers 
from pygments import formatters 
from BeautifulSoup import BeautifulSoup 
 
register = template.Library() 
regex = re.compile(r'<code(.*?)>(.*?)</code>', re.DOTALL) 
 
@register.filter(name='pygmentize') 
def pygmentize(value): 
    last_end = 0 
    to_return = '' 
    found = 0 
    for match_obj in regex.finditer(value): 
        code_class = match_obj.group(1) 
        code_string = match_obj.group(2) 
        if code_class.find('class'): 
            language = re.split(r'"|\'', code_class)[1] 
            lexer = lexers.get_lexer_by_name(language) 
        else: 
            try: 
                lexer = lexers.guess_lexer(str(code)) 
            except ValueError: 
                lexer = lexers.PythonLexer() 
        pygmented_string = pygments.highlight(code_string, lexer, formatters.HtmlFormatter()) 
        to_return = to_return + value[last_end:match_obj.start(0)] + pygmented_string 
        last_end = match_obj.end(2) 
        found = found + 1 
    to_return = to_return + value[last_end:] 
    return to_return 

This is a template filter, which can be applied like so:

Since 1999 I have had a presence on the web. I bought conesus.com back then and used it to link to projects, code samples, writings, and photographs. I now own conesus.com, samuelclay.com, and ofbrooklyn.com, all of which now serve different purposes.

My primary blog platform has been a hand-coded blog engine called SunRayLab that I used for ofbrooklyn.com for the past three years. It's time to retire it, as great as I have felt I made it. The whole caboodle is in aging PHP and the future is definitely not there.

Thanks to Kevin Fricofsky's hard work, I have django-mingus to use for a blog. It combines a few dozen different django applications with a minimal amount of glue. Surprisingly, everything hums along nicely and Kevin listens and acts quickly, so I have faith in the future of this platform, for once.

It is humbling to take down my old creations, but it also signifies a maturity inherent in code obsolescence. Code ages, much like a house. If you don't take care of it and upgrade it with working utilities, then it will begin to crumble. Whether the water main breaks (perhaps you upgraded a dependency elsewhere on your server and the new version breaks your current code), or the appliances are no longer shiny and clean (and you want code highlighting in your blog platform, but that requires a lot of tangentially work that cascades into more work quicker than you can say 'enough').

My goal is to post code, help others in a way similar to how I have been helped by developers posting tidbits, tutorials, and findings. I plan to write an occasionally essay, since I tend to disagree with opinions fairly often and want something to show for my trouble. I may even post photos.

The future is bright and my fingers are not yet tired.

SVG, an open, standardized vector graphics markup language is now supported on 30% of all web browsers. The other 70% of browsers (specifically, Internet Explorer), support VML, a close cousin of SVG. If you write both SVG and VML, you can provide rich, interactive vector graphics > 99% of all browsers.

Raphaël JavaScript Library does all the heavy lifting for the developer. Write your circles, paths, animations, and interactions (events, mouseovers, clicks) in Raphaël, and it will write compatible SVG and VML that works almost everywhere. I have contributed to Raphaël and am currently presenting a paper and a workshop on Raphaël at SVG Open 2009 at Google HQ in Mountain View, California.

I wrote this demo for Daylife, but this demo uses a number of features in Raphaël, showcasing a number of features that can be used to make interactive graphics that are unlike anything else natively supported by almost all web browsers. This means no plug-ins or installs. No Flash. Just what is already built into every web browser.

Demo - Topic Connections Graph

Code

1) Download jquery.topicconections.js

2) This code should go into your <style>:

3) Include these scripts in your <head>:

4) This code goes inline:

5) This JavaScript code also goes inline:

In response to James Padolsey's blog post on his apprehension to studying Java before his first-year studying computer science.

University is unlike anything youve done, and unfortunately anything else youre going to do unless you really love the study of computer science. I recently graduated from university with a degree in Computer Engineering, taking roughly similar courses as a Comp Sci (except we also took harder classes related to engineering that were straight out of unrelated engineering disciplines, like thermodynamics, statics (bridges), and materials science).

Almost everybody enrolled in the computer science department started out years before school, programming in somethingusually web development (esp. Javascript, the bad parts). A healthy number of them drop out of the program and switch to math/physics/management or even art history. They drop comp sci not because they are dumb, or have a slower intake of the sheer volume of dense material fed to them. Its because they discover that the internals, things like how primitives are actually allocated in different ways in memory, sometimes even to your advantage to use the right one, are not as interesting as they had hoped. Low-level course material, such as transistor logic, VHDL, networking, operating systems (easily one of the most challenging and rewarded courses taught in computer science), and systems programming, are all above and beyond what most contractor/freelancers will face while they are busy earning 25% more than their comp sci cousins.

But the difference is worth a lot. Take a look at that one, systems programming. Here is a short description of whats taught:

EECS 337 Systems Programming - 4 credits Lexical analyzers; symbol tables and their searching; assemblers, one-pass and two-pass, conditional assembly, and macros; linkers and loaders; interpreters, pcodes, threaded codes; introduction to compilation, grammar, parsing, and code generation; preprocessors; text editors, line-oriented and screen-oriented; bootstrap loaders, ROM monitors, interrupts, and device drivers. Laboratory. Prereq: EECS 233 and EECS 281.

If you want to write code that does anything that involves automated decision making, the true back-end of great companies to work for or start, you need to actually study and learn about these low-levels to build better decision trees (data structures alone make it worthwhile) and neural networks. There are so many fascinating topics out there, and they are all more expressive than having to design web sites for clients.

I absolutely love the study of artificial intelligence, and am working on a project where the meat of the project is in the AI that feeds the site, but I dont write algorithms or architect systems for a living. I write Javascript code (and Python on the back-end serving up the front-end), designing a web app for publishers to create photo galleries. Its not quite the academic challenge I get on my project, but you also have to be very, very good at computer science to get recognized and to make big contributions, even to small companies (think startups).

Computer Science is a rich and engaging degree, and while it isnt for many who think it is, those who stick with it all 4 (to 5) years end up very happy. This is the golden age of software. Theres a lot of upheaval which will forge new empires, and I bet something as notable as the few winners of the California gold rush of the 19th century didnt even have the prestige that good engineers have. Think of all the famous and very wealthy nerds. Think about that when youre slogging through decimal floating-point arithmetics in binary by hand.

We were accepted! Here is our finished abstract, also available on the SVG Open 2009 workshop page: http://www.svgopen.org/2009/registration.php?section=workshops.

Dmitry Baranovskiy - http://dmitry.baranovskiy.com - dmitry@baranovskiy.com
Samuel Clay - http://www.ofbrooklyn.com - samuel@ofbrooklyn.com

The power of web browsers has reached the point when it is easy to use native vector graphics without any 3rd party plugins and installs. In this session, we will discover how to build rich graphics in your web app using Raphal (http://raphaeljs.com/) JavaScript vector library, the features Raphal provides, and walk through how to apply these features into a fully built module.

This course will cover a number of features ranging in complexity from basic layout to event-driven interaction. The module will be a moderately complex application consisting of parts SVG, Javascript, jQuery, and HTML. While Javascript experience is expected, it is not necessary, considering the syntax of Javascript in this capacity is not far different than many other languages.

The module, which will be used to demonstrate a number of techniques in creating both interactive and layered elements in SVG, will be sufficiently advanced as to cover many topics, but so in-depth that it cannot be built entirely from scratch in the time allotted for the course--including time used for demonstrating Raphal.

The 150 minute course will consist of 3 parts, with questions taken throughout:

• Introduction to Raphael (35 min)
  • 5 minutes: Introduction and Overview of the People Connection widget
  • 10 minutes: Looking at the Raphael JS Library's methods and documentation
  • 10 minutes: Setting up data for the module to be used in drawing
  • 10 minutes: Loading data into Raphael
• Drawing the widget (45 min)
  • 10 minutes: Images
  • 15 minutes: Shapes
  • 20 minutes: Lines, straight and curved (quadratic and bicubic)
• Interaction model (60 min)
  • 20 minutes: Handling user/triggered events
  • 15 minutes: Animation
  • 25 minutes: Interacting with the DOM and jQuery

There is a time buffer built into each part, and if there is extra time, we will be prepared to discuss:

• Raphael versus other SVG/Javascript libraries
• Other projects built on Raphael
• Browser compatibility with Raphael

For each section of the course, we plan to discuss:

• Implementation and coding conventions
• Potential pitfalls with certain techniques (browser incompatibilities, misconceptions, and common mistakes)
• Building the widget by coding in real-time (and not using blobs of code already written).

The flow of this walk-through course will be aligned with the building of the module, where we will start with a empty file and progress over each step used in creating the overall effect and feel of the finished widget. Each piece of the widget puzzle will be complemented with a demonstration of the incremental improvement. At the end of the course, a complete module, with all of its pieces, will be fully functional and fully explained. Participants will also have all of these Raphal resources to use in their own SVG projects.

Raphael is a wonderful Javascript library, used for creating vector graphics, and interacting with SVG/VML elements through Javascript. Raphael allows the developer to interact with these elements through a number of complex animations. 

Until now, the only speed for an animation was a linear movement from one state to another. This plugin extends the animation of Raphael to allow for easing animations. Derived from jQuery Easing Plugin (version 1.3): http://gsgd.co.uk/sandbox/jquery/easing/.

The SVG Open 2009 is in October at Google's campus in California. I recently submitted my proposal for a course on writing SVG through Javascript and the Raphael Javascript Library. Below is my proposal. Let's hope it is approved and I am able to continue work in SVG.

I am proposing to teach a course about building an interactive SVG module on the web through the use of the Raphael Javascript Library. From start to finish, the course will walk attendees through the process of creating an interactive widget, with a number of features ranging in complexity from basic layout to event-driven interaction. The widget will be a moderately complex application consisting of parts SVG, Javascript, jQuery, and HTML. While Javascript experience is expected, it is not necessary, considering the syntax of Javascript in this capacity is not far different than many other languages.

The widget, which will be used to demonstrate a number of techniques in creating both interactive and layered elements in SVG, is a "People Connection" module which shows a graph of how various people are connected to each other. See Figure 1 and Figure 2 for illustrations, and see http://www.daylife.com/topic/Barack_Obama/beta#title=Connections for a demonstration.

The 150 minute course will consist of 3 parts, with questions taken throughout:

• Introduction to Raphael (35 min)
  • 5 minutes: Introduction and Overview of the People Connection widget
  • 10 minutes: Looking at the Raphael JS Library's methods and documentation
  • 10 minutes: Setting up data for the module to be used in drawing
  • 10 minutes: Loading data into Raphael
• Drawing the widget (45 min)
  • 10 minutes: Images
  • 15 minutes: Shapes
  • 20 minutes: Lines, straight and curved (quadratic and bicubic)
• Interaction model (60 min)
  • 20 minutes: Handling user/triggered events
  • 15 minutes: Animation
  • 25 minutes: Interacting with the DOM and jQuery

There is a time buffer built into each part, and if there is extra time, I will be prepared to discuss:

• Raphael versus other SVG/Javascript libraries
• Other projects built on Raphael
• Browser compatibility with Raphael

For each section of the course, I plan to discuss:

• Implementation and coding conventions
• Potential pitfalls with certain techniques (browser incompatibilities, misconceptions, and common mistakes)
• Building the widget by coding in real-time (and not using blobs of code already written).

The flow of this walk-through course will be aligned with the building of the People Connection widget, where I will start with a empty file and progress over each step used in creating the overall effect and feel of the finished widget. Each piece of the widget puzzle will be complemented with a demonstration of the incremental improvement. At the end of the course, a complete widget, with all of its pieces, will be fully functional and fully explained.