Python owns us - The weblog of Jarno Virtanen: Category python

Tue, 04 Jul 2006 10:37:33 GMT

Most programmers seem to have a pretty good idea what's wrong with their day job. Interruptions, wrong tools, wrong people, you name it. We dream of a "perfect software development shop" where everything would be just ... perfect.

So here's an idea. Go to the infogami site that I created, In a perfect software development shop ..., and add your ideas of a perfect software development firm.

(I've added a couple, which don't necessarily reflect my dayjob. Just some ideas.)

Spread the meme if you wish.

Sun, 02 Jul 2006 11:15:11 GMT

I was kinda delighted to see that Minix 3, the new version of Minix that is not a teaching tool, already has a Python port, but appalled when I realized it was Python 1.5.2 that someone had ported. Now, I don't have any plans to toy around with Minix 3, but, still, Python 1.5.2? That's like so last century.

Sun, 11 Jun 2006 11:36:11 GMT

Gary King asked why a person would choose Ruby on Rails over Lisp in web development. Well, I don't know about that, but now that I've used TurboGears, there are things that I've become to appreciate in frameworks.

So, from the top of my head, not a complete or decisive list, and in no particular order, some of the things that I've grown to like in a web development framework:

A decent website. (Which is kept up-to-date. Might even get a facelift now and then. Gasp.)
Continuous development (That is, new releases.)
Project quickstart. (tg-admin quickstart myproject and you're ready to hack and view results at localhost:8080 or whatever.)
Painless but powerful HTML-templating. (Kid in TurboGears.)
HTML-templates are can be viewed as HTML. (Ie. no custom "HTML" tags.)
Easy ajax integration. (Especially important: don't have to duplicate anything on the server because of AJAX. Meaning that controller methods return the exact same data whether it's rendered with a Kid template or with Mochikit.)
Ajax library that fixes some of Javascript's flaws (Mochikit makes Javascript look more like Python, in a good way.)
Works on windows, too. (I primarily work on Linux, but I do need Windows box also.)
Development environment. (I don't have to manually restart the webserver whenever I make changes in the templates or the code.)
Quickstart tutorial. (Get a good understanding of the framework in just few minutes.)
Documentation. (Well, the documentation of TurboGears is pretty scattered, but there is documentation.)
Plugged-in persistence. (If I just want to persist something, it should be easy. And it is, with SQLObject.)
"Shell access" to persistent data. (tg-admin shell ..)
Bugfixes, bugfixes, bugfixes.
Features, features, features.
Straight-forward server configuration. (No XML, please.)
Clear development/production distinction.
MVC. (No religion here, it's just nice to have.)
Fast and easy install.
Visibility and buzz.

So, there.

It's not to say that every framework should have just those and a bunch of other features, but they make TurboGears feel good for me.

Mon, 22 May 2006 12:24:47 GMT

It's funny how you sometimes bumb into mentions of Python in odd places. I've been obsessing about Jeff Hawkins and his theories of how the neo-cortex of the human brain works. He has founded a firm to build intelligent machines, Numenta, and I just happened to glance at their employment prospects —though I'm not applying there or anything— and there's a mention of Python. It's not a requirement or anything, but they're into Python too.

The core of their products will be so intensive computationally that Python as such won't be fast enough. It is based on algorithms that are the basis of human intelligence. Or so Hawkins argues. I'm not going to go into details of Hawkins' argument. You have to read his book for it. But I have a feeling that that's important stuff they're working on.

Wed, 12 Apr 2006 10:22:44 GMT

Now that people have noticed that reddit's recommendation engine really doesn't work, I'd suggest reddit developers and anyone else interested to take a look at Leonard's thoughts: The Ultra Gleeper: A Recommendation Engine for Web Pages. Leonard's solutions might not be applicable to every recommendation problem out there, but he has given a lot of thought and effort to the issue. His approach avoids (or at least tries really hard to avoid) all the usual pitfalls of such engines.

Thu, 30 Mar 2006 19:11:07 GMT

A short writeup about Python's interactive interpreter not quitting when you type 'quit'. And about comparing that to Ruby's interactive interpreter.

I would have written it here, but my hosting provider seems to have permanently decided that mentioning Python or Ruby [1] or similar words in my weblog entries is a security threat. (Nevermind other, real security threats.) I really need to do something about the situation soon.

[1] I have to bypass it with stupid tricks, sigh.

Update: Ian Bicking notes that Python 2.5 answers to "quit" with "use quit() to exit", which is adequate solution given the nasty technical details involved. The not-so-magical-any-more-Python-3000 might fix this for real.

(Yeah. Commenting on this weblog is broken too, sigh.)

Random book from Amazon

Fri, 24 Feb 2006 14:50:43 GMT

I revived my age old application that gives you (quasi-)random books from Amazon.

Just needed to update the pyamazon library and rewrote the application in CherryPy.

I find the application a bit addicting. For a little while, at least.

(And one day I'll get around ordering a real domain for myself, too.)

Deception, bias and programming language communities

Fri, 03 Feb 2006 19:06:31 GMT

After listening Robert Trivers' talk on deception, I began wondering about self-deception, bias and programming languages. Bias in the sense of favoring arguments that support your view and vice-versa.

The bias is worst for things that you have already made up your mind. Say you are choosing a college. Before choosing one, you probably weigh the alternatives quite critically and might even experience some anxiety over the choice. But once you've made up your mind, ie. chose to go to a certain college, you just ignore everything negative about it and endorse all positive tidbits you hear regarding it.

This all happens mostly unconsciously. Even if you know about it, it's extremely hard to take it into account. (Let alone if you don't know about it.) You'd have to always be consciously aware of this bias and deliberately take it into the calculation.

Another example would be a so called we-bias. You belong to a group and then there's everyone else. If a member of your group does something positive, say, gives a donation to a charity, you tend to generalize: "Bob is such a generous guy." Whereas if it's not a member of your group, you tend to be very specific: "Alice gave a small donation." And vice versa for negatives. Member of your group: "Bob stumped into Ned." Not a member of your group: "Alice is so hostile."

Again, this happens mostly unconsciously. You probably have to be a researcher, or at least an outside observer, to tap into this phenomenon. You're very unlikely to notice your own bias. (Unless you specifically look for it; a rare occasion I would assume.) Self-deception is a very strong power. As Trivers noted, it's likely that a selective force was in favor of genuine self-deception, because one is undoubtedly a better liar if one isn't aware of one's lying. And, boy, is lying good for your breeding prospects or what!

All this deception and group bias must be going on in programming language communities, too. And I'd argue that there isn't a simple cure for it. You might think you're immune to such deception, but, then again, you might be just cheating yourself. You see, you are prone to deception and bias if you have taken sides, so to speak. In other words: you are biased, if you have a favorite programming language.

No matter how hard you try to be objective, you probably still aren't. No matter how diverse you think your views are —if you still have a favorite programming language (or two)— you probably are biased.

I think the deception and especially bias is apparent in the way programming language communities are hostile to other, inferior languages. Once you've made your personal decision (which you might of course change later) on a preferred programming language, you tend become dismissive about the relative weaknesses of your language and you tend to emphasize to strengths of your language. What's more, you probably become extremely critical about other languages.

People can look for good ideas and strengths in other languages, but they do think that "their language" is one some sense the "best language", at least for them. They feel as if there was a genuinely all-around better language out there, they'd switch to it, immediately. But I would assume that this is not the case.

There are exceptions to this rule, of course, but if you're now thinking "yeah, that's me", I wouldn't be so sure about it. I am pretty sure I am biased and self-deceptive, especially on this subject!

And frankly, I don't know if this is such a bad thing, after all. Sure, self-deception and bias distort your objectivity and decisions, but it is so hard to always fight against self-deception. And once you've made your decision, you're probably going to stick with it for a while and knowing your language very well is good for your productivity. Furthermore, you can always explore other languages; it's just that you won't probably make objective and balanced observations of other languages. Get used to it.

Wed, 07 Dec 2005 03:46:05 GMT

Aaron says:

My friends over at reddit.com rewrote their site from Lisp to Python in the past week.
[...]
The idea that there is something better than Lisp is apparently inconceivable to some, judging from comments on the reddit blog. The Lispers instead quickly set about trying to find the real reason behind the switch.

Sat, 03 Dec 2005 18:25:12 GMT

Holy shit. CherryPy is damn impressive and slick. And easy too. No wonder Kevin Dangoor chose it for the Turbogears web stack.

(I haven't had much motivation to experiment with Zope 3 recently. Which is not to say that it's Zope 3's fault.)

The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Floating Point Numbers (No Excuses!)

Wed, 09 Nov 2005 10:54:44 GMT

(The title shamelessly mimicked from Joel Spolsky.)

If you ever happen to wonder what the hell is going on with situations like this:

 >>> print 0.1
 0.1
 >>> 0.1
 0.10000000000000001

please, please, please read the Python tutorial appendix: Floating Point Arithmetic: Issues and Limitations.

It's all there. Explained by a seasoned expert. If you don't understand floating point numbers after reading it, don't use floating point numbers. (At least for anything serious.)

A very important concept regarding floating points is the fact that you can't "round" the approximation for the number 0.1, that is, 0.10000000000000001, to get rid off the ugly tail; the approximation is already as accurate as possible!

Reloading views in Zope 3

Mon, 10 Oct 2005 18:17:55 GMT

For starters, I must say that the Zope 3 product named z3reload is a real life-saver for a Zope 3 newbie like me. It's pretty easy to install and configure, and works, of course, transparently. Exploring Zope 3 sometimes feels like trying to find a light-switch in a completely dark house and this process is much less painful with automatic reloading of views.

Over and out.

Learning, perhaps, Zope 3

Sat, 08 Oct 2005 18:25:19 GMT

With the little free time I have for myself, I've decided to try to grok Zope 3.

Years ago, I made websites with Zope 2 for a living (for a while), but I quit that job (for reasons that had nothing to do with Zope) and I haven't done anything with Zope ever since. (What stuck with me, of course, was Python.) My memories of Zope 2 were pretty much those of many others: an impressive system, but not quite the ideal web application platform.

Since then, I've been hearing that this new system, Zope 3 or X3, has all the goodies of Zope, but is a programmer friendly web application platform. Sounds good to me, so maybe it's time to find out about that.

I am in the happy position of not having to drag around the historical baggage of Zope 2. I do not have to move from Zope 2 to Zope 3, nor do I have to un-learn much of Zope 2. I'm doing this for a hobby, out of general curiosity. (I can quit whenever I want to.) I'll try to approach Zope 3 as just another web application platforms amongst many others. An old platform as whole, but a new platform to me.

Given the constraints of this hobby, limited resources and especially time, I cannot promise much of this venture, but I hope I can document parts of it.

My motivation to try and understand Zope 3 in particular is the fact that some people have claimed that it makes it easier to reuse code. We all should know by now that code reuse is hardly ever easy, but, given the opportunity, it's something to strive for. It should not be the Holy Grail of programming, but it's damn boring to do the same stuff all over again, one time after another.

I am also hoping to find out that Zope 3 makes web application programming fluent and ultimately fun activity.

Finally, I have no broader context for this venture; I have no job prospects nor opportunities to use Zope 3 in my current day job. I am doing it just for the fun of it.

Let's see.

(Maybe I give up in a day, and you'll never hear another word on this subject from me. That's a possibility, too.)

Off to Benji York's Zope 3 Quick Start Guide, then.

Tue, 04 Oct 2005 10:12:10 GMT

Question: "What exactly is python used for besides shell-type scripting?"

Answer:

Too busy to answer properly, but I'll just mention that Fermilab uses a home-grown python package called Enstore to manage their data store of 3 Petabytes of physics data, growing at 1PB/year. The transfers of ~25TB/day to and from that system is what keeps me busy.

(A presentation about their system.)

Wed, 28 Sep 2005 18:23:42 GMT

A marketing video of Python. I like it.

(The original site is not, at the moment, very fast, so you might want to search for the file on search engines.)

Wed, 07 Sep 2005 04:14:45 GMT

News from py2exe page:

Can now create single file executables.

I don't know whether it was a big job, but I know it's a big deal for many. Sure, it's really not that important to have a single executable, but it's the kind of things that somehow end up mattering anyhow.

Beautiful

Wed, 03 Aug 2005 05:15:11 GMT

Adding to the ever-growing body of Python tutorials, the material of Ka-Ping Yee's aptly named course, Beautiful Code, gives a pleasant yet efficient introductory lesson to the world of Python.

Mon, 01 Aug 2005 18:58:23 GMT

Some kind of a deliverable from PyPy team: seems they've made PyPy run on its own. See also the collection of photos building up to the moment.

I am still low on original content, sorry for that.

PyPy 0.6 released

Mon, 23 May 2005 11:50:22 GMT

PyPy Development Team announced the first public release of PyPy after two years of spare-time and half a year of EU funded development.

More details at PyPy homepage.

The Getting Started document provides quick steps to get your own copy of PyPy running.

More accurate language statistics from Roeland Rengelink

Thu, 07 Apr 2005 14:37:52 GMT

Roeland Rengelink did, back in 2001, just what I asked for, eg. compared the growth rate (among others) of various programming languages at Sourceforge. Now he did it again and the numbers aren't so great for Python, actually.

Wed, 30 Mar 2005 17:48:26 GMT

This week's mildly interesting (and certainly somewhat dubious) datapoint comes from an ancient Freshmeat article that complains of too few Python projects. The author proclaims:

As I write this, there are almost 3,000 projects in freshmeat's C category and almost 1,500 in the Perl category, but there are only about 400 projects in the Python category. SourceForge has similar statistics.

Comparing this with the current programming language statistics from Freshmeat, we compile the following table:

Language Growth (2001 - 2005)
C 2.4

Perl 2.2

Python 4.5

Language	Growth (2001 - 2005)
C	2.4
Perl	2.2
Python	4.5

The number of projects per programming language is consistent with Sourceforge's numbers:

Number of C projects /
number of Perl projects C / Python Perl / Python
Freshmeat 2.2 4.0 1.8

Sourceforge 2.5 3.8 1.5

(Consistency is caused partly by the overlap of projects in these sites.)

The article didn't provide numbers from Sourceforge, but noted that "SourceForge has similar statistics" so we can conclude that the growths are similar both in Freshmeat and Sourceforge. Thus, Python is catching up with at least C and Perl. (Though Python might be losing to other languages.)

Someone enthusiastic enough could gather more accurate statistics from Sourceforge's listings and generate nice plots of them. Scraping automatically all the needed pages from sf.net would be somewhat impolite, though, I think.

Tue, 29 Mar 2005 18:26:40 GMT

Although Joel gets the details a bit wrong, SpamBayes gets well-deserved publicity on Joel's essay The Road to FogBugz 4.0: Part II:

In the meantime I had been using SpamBayes for my own personal email. This is probably the best implementation of what is probably the best spam filtering algorithm out there: Bayesian filtering, invented by Paul Graham and first published in August 2002 in his seminal article A Plan for Spam.

...

It wasn't Graham who invented Bayesian filtering, though his essay certainly popularized the idea and made a huge impact on the adoption of statistical spam filters. Also, SpamBayes's algorithm is not exactly Bayesian, as far as I understand. And, actually, Paul Graham's Bayesian algorithm isn't Bayesian either, but I don't think we want to dwell on that. See also Tim's comments on relative merits of Graham's scheme vs. the so called naive bayesian classification:

I doubt that Graham's approach would work well for a subtle classification problem -- but it's not trying to, and it works exceedingly well for what it is trying to do. That seems a stroke of genius to me.

Mon, 28 Mar 2005 18:50:50 GMT

BeautifulSoup on Python for Series 60.

Tue, 22 Mar 2005 17:45:29 GMT

Having used JARs with Java, I think PythonEggs is a great idea. And, as of PyCon 2005, it seems it's a lot less of an idea and more of an implementation.

Sat, 19 Mar 2005 05:39:20 GMT

Nick Coghlan's weblog, Boredom & Laziness, seems informed and entertaining. One of the topics covered is, of course, Python. Check out, for example, his take on static typing in Python. (I also found the note about Yast Online Update's braindeadness accurate.)

Lesser known historical tidbit of Python

Sun, 20 Feb 2005 09:06:21 GMT

Most of us know, and some have even read, the original announcement of a new programming language called Python back in 1991 by Guido van Rossum. But did you know that shortly before that Guido wrote a rather apologetic response to being flamed about non-source posting to alt.sources.d:

OK, hey, I surrender! I won't ever post non-source to alt.sources again! But don't send me any more flames. How many self-appointed policemen does alt.sources have? Are there any other rules I should be aware of? I am about to post half a megabyte of source; wouldn't want to be flamed for *that*...

Well, I didn't.

Python whitespace FAQ, or, Python is not Fortran 77

Sat, 19 Feb 2005 05:19:46 GMT

It's really rather shame that I should write a page like this in these modern, wise times, but I have heard people mentioning this stuff now and then.

So, does Python's significant whitespace suck because Fortran sucks?

No, it doesn't.

Should I refer to Fortran (77)'s whitespace syntax and semantics when discussing Python's significant whitespace?

No, you shouldn't.

Why not?

See Fortran 77 language reference. Python's significant whitespace just means that you need to indent your code correctly, which you should do anyway, and that's it. There are no fixed fields or anything like that in Python. See this old Linux Journal article by Eric Raymond for overcoming similar preconceptions.

But am I allowed to dislike Python's significant whitespace anyway?

Sure, as long as you don't refer to Fortran (77).

Should I dislike Python's significant whitespace?

No, you shouldn't.

Why not?

Because it's the most natural thing, and, while it's a lovable feature of Python, it's not that big a deal anyhow. It shouldn't be a show-stopper.

But it's ugly and wrong.

No, it's not. It's beautiful and right. But this isn't going anywhere with us arguing whether it's right or wrong. The point is that if the only reason holding you back from trying out Python is its significant whitespace, then you're really stupid. I mean a total idiot. Even if the significant whitespace is only the proverbial straw that broke the camel's back, you're still an idiot for not trying out Python because of this.

You need not like it after you've really tried it. You may as well say that you don't like the significant whitespace because you couldn't get comfortable with it. I would find that odd, but I wouldn't argue with you on that. However, if you dismiss Python as a viable alternative because of its significant whitespace; well, as I said, that's just stupid.

But it's so brittle!

No, it's not.

I know of zero cases where a non-newbie has unintentionally screwed up his Python program totally because of the significance of the whitespace. That just does not happen. Again, these are not issues that make programming hard or easy (and, believe me, there are things that really make programming hard), but are of rather minor importance.

I am not saying that messing up one's Python program is something totally unheard of, but, on the other hand, I personally know of many cases where screwed-up indentation in a language that doesn't have significant whitespace has lead a programmer to astray. Yeah, sure, from time to time you introduce a bug to your program by mis-indenting in Python, but this happens in other languages too (only in reverse, because of the insignificance of the whitespace). On my personal account, then, significant whitespace has saved me from more trouble than it has put me into.

Well, the indentation might get screwed in web pages or emails, right?

Yeah, sure. And that's an annoyance. But if the code is anything more than ten lines, it should be laid out by a professional anyhow and if the code is something significant, you shouldn't distribute it that way. If the code is an example, it's a nuisance if the indentation gets screwed, but you can work the indentation out yourself, too.

(My own code examples in the archives of this blog are actually broken, because of some change in the blogging software I use. You can more easily copy & paste the Java code in my archives, right now, but I really should just fix the mark-up.)

If you're distributing the code in regular script files, the significant whitespace poses no extra threats.

It's difficult to program Python in Notepad/Word/Some-other-exotic-editor because of the significant whitespace.

Yes, it's a bit difficult, though not impossible. I hope you don't get too upset by this, but I'll have to be insulting you again: if you program in Notepad, then you are stupid. You are, believe me.

If you don't know what editor you should use, see the page Python Editors.

It's difficult to move around big code blocks because of the indentation issue.

This is true and a real issue. There are features in advanced editors that make it easier, but it's not real easy with them either. But it's not like you can move around big blocks of code easily in every other language either. The program might be syntactically correct after moving the code block, but you still have to fix the indentation.

But this is really also a question of one's personal programming habits.

(words: 758)

Python really becoming Lisp?

Wed, 09 Feb 2005 05:34:06 GMT

Y'all have of course read all the past Python vs. Lisp hoobla; Python being "Lisp-envy" of sorts, and you may have mostly ignored it, or tried to, at least. Nevertheless, Oliver Steel offers a clearer viewpoint to this kind of thinking with his post Becoming Lisp.

He points us to recent Python Cookbook recipes that are based on (to some extent) type/class unification, decorators and generator expressions:

I�ve listed below some of the functional and metaprogramming recipes and packages that have shown up in PyPI and the Python Cookbook over the past few weeks.
...
The point is that these are flowing in fast and furious, and that they don�t take much code. The recipes below are from just two weeks.

Ultra Gleeper, a recommendation engine that really works, unlike most

Mon, 07 Feb 2005 09:09:48 GMT

The brief exposure that I had with Leonard's Ultra Gleeper —he gave me a free test-drive after I whined about the sorry state of recommendation engines in my weblog— was sweet and joyful. It really did fix the problems I reported about recommendation engines I had had the displeasure to deal with. Check out his latest Ultra Gleeping status report and particularly the introduction paper: The Ultra Gleeper: A Recommendation Engine for Web Pages.

IDEs are not silver bullets either

Sun, 06 Feb 2005 11:28:02 GMT

I am really growing tired of hearing the same old "Java has Eclipse or IDEA or Some-other-IDE, therefore you can be as productive in Java as in any other language" phrase spouted whenever anyone mentions Python, Perl or Ruby (or Lisp or Smalltalk). Sure, Eclipse is neat. Sure, even I use Eclipse at work.

Eclipse makes it easy to find appropriate Java methods, because it has a powerful auto-completion. It even opens up the corresponding documentation in the context. And yes, Python IDEs do something similar too, but they can never be quite as complete as Java's, because of the dynamic typing. Ditto for refactoring. Ditto for many other features.

But it does not mean that programming in Java suddenly, magically becomes easy. Programming isn't hard because you can't always find the right libraries, classes and methods instantly. (Programming might be a bit inconvenient because of this.) Programming is hard, because it's complex. Programming is hard, because it's abstract. Programming is hard, because you can't visualize it. Etc.

Programming isn't hard at the level of individual expressions and statements. Programming is hard at higher levels, at higher levels of abstraction. But these two are not separable. The higher levels, where things are more hairy, are composed of the individual expressions and statements. What's more, you can't escape from the lower levels. You drag them with you whatever you do. This is why programming in assembler is harder: you have drag the most low-level statements with your abstractions. The devil really is in the details.

The issue is of course not as bleak as I have painted it. You can make your own abstractions, making it easier for you to work with the high level concepts. Making abstractions is indeed a very powerful tool. Unfortunately, abstractions are not silver bullets either. (Not in the form of Objects, or in the form of Subroutines.) You still can't completely escape the individual expressions. You just can't. Abstractions leak. Bugs creep in. Integration becomes harder the more components you have. In some sense, abstractions makes the thing just harder.

What dynamic languages target is the level of the individual statements. They make —it is argued— the lowest level visible to the programmer a bit higher. They make the base level a bit higher. It's the fundamental bottom-up programming. Make the language of the programmer a bit closer to the domain of problems. (Domain specific languages are even more powerful with this, but they have their own set of problems, too.)

The point is not to make programming more convenient (although that's arguably one side-effect of dynamic languages); the point is to make the difficult task of making large programs less daunting. It's of course debatable whether dynamic languages succeed in this mission, but IDEs do not belong to that discussion. It all boils down to this: No IDE is going to make Java a higher level programming language. Unless, of course, the "IDE" is actually a interpreter to some higher level language itself.

NewsBruiser plugin in development: Automatic paragraphs

Sun, 06 Feb 2005 06:13:54 GMT

I don't have reST support on the site where my NewsBruiser installation lives (and I don't particularly like reST anyway), so I decided to write a plugin that does the only thing I really need when writing entries: converting two consecutive linefeeds to a <p> tag. (Two consecutive linefeeds is, of course, the Universal Paragraph Separator; so much that even in applications where you don't need to type two consecutive linefeeds to start a paragraph, you do it anyhow.)

The only challenging thing writing the plugin was that my ISP doesn't provide a decent programming environment — I don't have command line Python or Python mode for Emacs there. Otherwise, writing the plugin, with all its warts and bugs, took about 15 minutes, including the "I Want Options" based option to NewsBruiser configuration.

So I propose a NewsBruiser Plugin Development slogan:

NewsBruiser Plugin Development — so easy that a child can do it (see the picture below)

(To be honest, though, Telma was IRCing at the moment..)

Here's the tgz package of the plugin: AutomaticParagraphs-1e-13.tgz. Extract it to your NewsBruiser base directory (it creates a directory nb/plugin/entry/filter/paragraph/) and configure it on the Entries tab in the NewsBruiser configuration.

Upgraded to NewsBruiser 2.6

Sat, 05 Feb 2005 19:42:21 GMT

Finally decided to upgrade to a more recent version of NewsBruiser, namely 2.6. I don't actually know whether there's a real NewsBruiser upgrade guide out there somewhere, but I did it by making a new installation of the more recent version and manually copying the data from the old installation. Then I checked that most of the stuff still worked and switched to the new location with Apache rewrite configuration change. Seemed to work. Let me know if anything's hosed.

(Well I did incorporate the couple of dirty tweaks I did for my Finnish weblogs, but you don't want to know about those..)

SummaryList does not work for POU right now, though it works for my other blogs; I need to figure that out.

I leaped from 2.3 to 2.6 and the change definitely seems substantial. Thanks again to the Evil^H^H^Hnergetic Master Mind behind all this, Leonard.

update: OK, some minor differences are visible to users. URLs are bit different. Old links to POU's articles should work, but links to individual entries now have longer URLs.

Mon, 31 Jan 2005 06:10:54 GMT

It seems Python owns us is not syndicated either in the "original" Python Programmer Weblogs or in Planet Python. I don't know what's up with Planet Python (maybe it's same as with the other), but Python Programmer Weblogs has a link to the RSS feed of POU's previous location (http://www.hole.fi/jajvirta/weblog/rss.xml), which, alas, is not up at the moment. (And the site is not in my control.) But the thing is that the previous location has been giving permanent redirect for at least eight months. Google, for example, correctly displays the new syndication location if you ask for http://www.hole.fi/jajvirta/weblog/rss.xml.

A permanent redirect means, according to RFC 2616, that

The requested resource has been assigned a new permanent URI and any future references to this resource SHOULD use one of the returned URIs.

It's only a SHOULD, sure, but it'd be nice if aggregators respected it. I cannot do much else myself, because I've corrected the wiki page, but it seems it isn't scraped automatically anymore.

And I know that this is the wrong place to whine about the issue, because anyone reading POU directly don't need it in the syndicated feeds. (Actually, it might be for the better this way, since you don't have to see my entries twice.) Anyway, this entry falls into the category "in case you wondered". I did.

Clever is better than explicit?

Sun, 30 Jan 2005 17:48:17 GMT

I suppose you all know The Zen of Python and its second line, "explicit is better than implicit". Sure, it's basically just a catch-phrase, but rings true for most of us. (Trying to define its exact meaning would be quite difficult, though.) We don't want no magic. We want programmers to express what they mean.

Then earlier today, somehow, I ended up reading a Python Cookbook recipe, written by the brilliant Alex Martelli, titled Add an entry to a dictionary, unless the entry is already there. It offers an alternative to the common pattern when dealing with, for example, lists embedded in dictionaries. The usual, "obvious" way is to do this:

 if d.has_key(key):
     d[key].append(value)
 else:
     d[key] = [value]

But it's rather many lines to say a fundamentally simple thing (a simple thing in Python programs, at least). So Alex proposes a more succinct way to express the same thing in Python code:

 d.setdefault(key,[]).append(value)

It exploits the setdefault() dict method, which, as Python Library Reference puts it, works like this:

a.setdefault(k[, x]) a[k] if k in a, else x (also setting it)

(OK, maybe "exploit" is too strong a word, because the above usage is just what setdefault() is for.)

Anyhow.

It got me thinking whether this is an improvement of the obvious way to do the same thing. (I am using "obvious" here in a probably non-obvious way: meaning that it's obvious to me.) I did not come to any decisive conclusion, but my initial reaction was that the latter way is not exactly explicit. The code doesn't say, directly (to me, again), that "I want this value to added to the list of this key". But then again, neither does the former.

When working with immutable values, like counting the number of occurances of different words, for example, you can express the similar pattern with the aid of dicts get(), like this:

 d[word] = d.get(word, 0) + 1

But this reads more clearly, I think. It says that "if we haven't found any occurances of word yet, that equals to 0 occurances so far".

What I wanted to say with all this, I guess, is that if I were to read, for the first time, the source code of some program that I needed to understand, code like d.setdefault(key,[]).append(value) would make me a bit baffled. (Before I knew this particular idiom, that is.) While the more verbose way to say it, if d.has_key(key): ..., would be instantly obvious. But I am not sure about this whole thing. I know that I am not on solid ground with these arguments. I do not know, anymore, what really is more obvious and more explicit, in the good way. Even for myself.

Jython Servlets, JSP and JSTL

Fri, 28 Jan 2005 19:27:59 GMT

I was supposed to say a few words about Jython Servlets, JSP and JSTL, but I ended up writing a whole tutorial on it:

Using JSP for HTML layout for a Jython Servlet.

Wed, 26 Jan 2005 19:58:57 GMT

For what it's worth, here are some numbers comparing cElementTree and Java DOM (through Jython). (Not a reasonable comparison, of course, but is supposed to serve as a rough reference point.)

The usual start = time.time(); do_stuff(); print time.time()-start maneuver was used for timing.
According to pystone.py, This machine benchmarks at 23201.4 pystones/second. It claims to have a 1.8GHz processor.
Python 2.3.3 was used.
The source XML was a 400k file that repeats one similar structure [1] N times.
As for why this machine is so damn slow although the pystone reports relatively big numbers, I have no idea. Maybe there's something weird with disk reading, dunno. Anyway, the relative speeds of the benchmark seem to match other people's benchmarks.

 xml.dom.minidom    5.7s  (Python 2.2)
 xml.dom.minidom    1.3s  (Python 2.3)
 ElementTree 1.2    1.1s
 Java DOM           0.4s  (Jython 2.1)
 cElementTree       0.06s

So, it's pretty clear that cElementTree beats this particular style of XML parsing hands down..

And yes, this machine seems to be particularly slow at it's job, consistently, as Fredrik wondered in the comments of my entry about the Jython/Java style XML parsing.

[1]

 <entry>
   <host>64.172.22.154</host>
   <referer>-</referer>
   ... 
   [ca. 10 elements like those above]
 </entry>

update: Oh yeah, and the numbers are definitely not totally consistent with my previous ad hoc benchmarks, but previously xml.dom.minidom probably suffered more because of its extravagant use of memory. Now that I tuned the size of the XML file down to 400k, the difference between xml.dom.minidom and Java DOM is much milder.

Sat, 02 Oct 2004 07:14:29 GMT

Quite a flurry of activity at the comments of Andrew's entry in which he says that Python development should focus more on the standard library rather than core language features (and thus CPython implementation).

Tue, 07 Sep 2004 19:00:10 GMT

Fredrik asked about my suspicious timings, and I'm a bit puzzled too, now. Well, anyway, with XML (stolen from some David Mertz article that benchmarked XML parsers) like this:

 
   64.172.22.154
   -
   -
   19/Aug/2001:01:46:01
   -0500
   GET
   /
   HTTP/1.1
   200
   2131

times 3000 (which equals around 1 megabyte), the following code prints out a time of around 3 seconds:

 from xml.dom.minidom import parse
 import time
 start_time = time.time()
 dom1 = parse('myfile2.xml')
 print time.time() - start_time

(And similarly, this parsing with elementree (w. Python parser) prints out a time of around 2.5 seconds:

 import time
 from elementtree import ElementTree
 t1 = time.time()
 tree = ElementTree.parse("myfile2.xml")
 print time.time()-t1

)

The similar parsing with Jython and Java's libraries takes around half a second, by the way. The difference 5-6 times to pure Python parsers versus difference of over 10 times in the previous timing. The previous Python minidom timing suffered from memory consumption and swapping, I think.

Parsing XML in Jython with Java SDK's libraries

Wed, 01 Sep 2004 19:58:38 GMT

In somewhat similar fashion, it's easy to parse an XML document to a DOM tree in Jython with Java SDK's libraries:

 from java.io import FileInputStream
 from javax.xml.transform.stream import StreamSource
 from javax.xml.transform.stream import StreamResult
 
 from javax.xml.parsers import DocumentBuilderFactory
 

 factory = DocumentBuilderFactory.newInstance()
 builder = factory.newDocumentBuilder()
 

 input = FileInputStream("myfile.xml")
 document = builder.parse(input)
 document.getDocumentElement()
 # etc.

There's no specific need for this, because you can do DOM stuff in recent Python implementations too, but with my naive benchmarks the above Jython snippet clocked at roughly 2 seconds with a 3 megabyte XML file and a Python 2.3 minidom parsing [1] clocked at 26 seconds. (I am not saying that the stock Python 2.3 minidom parser is representative for Python XML parsers, but to give some reference for the speed.)

I am not, of course, suggesting that you should switch to Jython, but I wish I did demonstrate the usefulness of Jython as a scripting tool for various XML tasks.

[1]

 from xml.dom.minidom import parse
 dom1 = parse('myfile.xml')

How to do an XSL transformation in Jython

Wed, 01 Sep 2004 19:13:09 GMT

Since Python standard library doesn't have (to my knowledge, at least) an XSL transformer, and if you happen to have Jython installed, here's how you can do an XSL transformation with Java SDK's libraries:

 from java.io import FileInputStream
 from java.io import ByteArrayOutputStream
 
 from javax.xml.transform import TransformerFactory
 from javax.xml.transform.stream import StreamSource
 from javax.xml.transform.stream import StreamResult
 

 input = FileInputStream("test.xsl")
 xslSource = StreamSource(input)
 xslTemplate = TransformerFactory.newInstance().newTemplates(xslSource);
 transformer = xslTemplate.newTransformer()
 

 output = ByteArrayOutputStream()
 result = StreamResult(output)
 

 source = StreamSource(FileInputStream("source.xml"))
 transformer.transform(source, result)
 

 print output # .. or output.toString()

Speed

Thu, 12 Aug 2004 06:46:22 GMT

I might be hallucinating, but it seems that there's a good chance that recent dynamic languages like Python, Perl and Ruby are on the verge to become a great deal faster. Apart from, say, Psyco, the efforts mostly remain vapourware, but people with brains the size of a small planet are serious about this stuff, and, you've got to admit, a small dose of additional speed for Python programs wouldn't hurt. This is not to say that the current situation is intolerable; Python's been doing fine for well over ten years now. It's just that it's somewhat inconvenient to drop back to C/C++/Java for performance bottlenecks.

Languages like Lisp, Smalltalk and Self are mostly over this hurdle already. They've been optimized by the power of academic interest and really smart people [pdf]. One might ask that why don't I just switch to some of these languages, and to tell you the truth, I don't have a perfect answer for that. It's just that I've already learned Python, and I'm really comfortable with it, and I'm not, right now, willing to invest time to become acquinted with some other language. Learning the language is not the hard part; the hard part is to learn to make real programs with the language. With languages like Java or PHP, it's really rather easy to learn the language and to learn how to make real programs with them. Past some point it might be laborous and difficult to make programs with Java or PHP, but that's not what counts at first. (Sure, of course, it should count.)

Well, anyway. It's things like Psyco, Parrot (and its JIT), pycore, Starkiller, a general purpose Just-In-Time library, IronPython, and, yes, even PyPy (which at its current stage is not what you would call fast) that bring hope to speed addicts. I'm not really qualified to judge any of the projects, but I would guess that if it was (and is) possible to substantially speed up Lisp, Smalltalk etc. in the past, why wouldn't it be possible to speed up Python. It just needs a handful of geniuses and a lot of work. I wish them all motivation and resources to finish their respective projects.

(There's also the slim chance that the different people working on such projects could collaborate, but I really don't know if that would be useful.)

std.utest, the complementary unit testing framework

Sat, 07 Aug 2004 08:18:53 GMT

Holger Krekel and Armin Rigo have been working on a "complementary second standard library" for Python. They want to make the standard library more Pythonic, rather than Java-like. Their motto is that "the best API is the one that doesn't exist."

And, indeed, their unit testing framework, std.utest, is mostly non-existent to the application programmer. Consider this simple routine, in a module named, say, list.py:

 def matching_names(prefix, names):
     """Case insensitive match."""
     
     prefix = prefix.upper()
     return [name for name in names
             if name.upper().startswith(prefix)]

Now, to test it with std.utest we make a module named test_list.py, and its complete listing is this:

 from list import matching_names
 
 def test_matching_names():
     names = ["abc", "Abac", "baca", "bbb"]
     assert matching_names("a", names) == ["abc", "Abac"]
     assert matching_names("ab", names) == ["abc", "Abac"]
     assert matching_names("ba", names) == ["bac"]

No special imports, no classes (although you can use a class to organize your tests if you want), no inheritance, no test runners, no suites. Running this with std.utest's supplied test runner (you can write your own if necessary) is as simple as this:

 python ..\std\bin\utest

Which spits out this:

 inserting C:\Documents and Settings\Jarno Virtanen
 _______________________________________________________________________________
 ______________________________  TESTS STARTING  ______________________________
 _______________________________________________________________________________
 F
 
 _______________________________________________________________________________
 ________________________________ Test Failure ________________________________
 

 >>> test_matching_names
 >>> 'C:\\Documents and Settings\\Jarno Virtanen\\cd\\test_list.py', line 9
 

 def test_matching_names():
     names = ["abc", "Abac", "baca", "bbb"]
     assert matching_names("a", names) == ["abc", "Abac"]
     assert matching_names("ab", names) == ["abc", "Abac"]
 >   assert matching_names("ba", names) == ["bac"]
 

 -------------------------------------------------------------------------------
 assert ['baca'] == ['bac']
  +  where ['baca'] = matching_names('ba', ['abc', 'Abac', 'baca', 'bbb'])
 ============================== 1 TESTS FINISHED ==============================
 0.05 seconds (0 passed, 1 failed, 0 skipped)

Notice how std.utest magically deduces what exactly was compared, in the runtime, in the failing assertion. It also shows how the tested routine was called by.

Unfortunately, no one has yet written much documentation about the complementary standard library. You pretty much have to figure it out yourself, but, as you saw, it's not too difficult once the basic idea is revealed.

The only way (that I know of) to obtain it right now is to use Subversion and check out the project like this:

 svn co http://codespeak.net/svn/std/trunk/src/std

Fri, 06 Aug 2004 19:40:19 GMT

A friend of mine, Markku H�nninen, starts weblogging by implementing type checking on a local namespace of an interactive Jython interpreter.

Also, he is yet another new NewsBruiser user.

Score one for Python

Fri, 06 Aug 2004 19:12:41 GMT

I had an amusing incident when I was fixing some bug in a software written in Java. I rather quickly managed to deduce the bug to a portion of a method and I began fixing it. I started by reorganizing the code and after a while I ended up with code somewhat like the following:

     while (true) {
         // ...
 
         for (int i = 0; ...) {
 	    doSomething();
 	    if (someCondition) {
                 updateSomething();
 	        someFlag = true;
                 break;
 	    }
         if (someFlag) {
 	    break;
         }
         if (somethingElse) {
             doSomethingElse();
         }
     }

And the damn thing didn't work correctly.

Furthermore, it seemed like the software was caught in an endless loop. It's of course blatantly obvious now, and might have been obvious to you at first glance, that the for loop is missing an ending curly brace. Turns out that the rest of the method was some 10 lines of comments and a closing curly brace. The curly brace was scrolled out of the screen.

It didn't take that long to figure out the problem, but still, it was rather frustrating. The problem was that my brain assumed that the indentation was correct and deduced the control flow from it. The thing is, such problem isn't (in practice) not even possible in Python. Score one for significant indentation.

Sun, 01 Aug 2004 10:53:02 GMT

By the way, did anyone talk about the possible EU funding for the PyPy project? Is there some sort of a public progress report on the subject? Can anyone shed some light on the funding? Does anyone actually know the situation? All I've seen is "it's too soon to say anything about the funding", which might still be true.

I'm just curious.

Implementations of Python

Thu, 29 Jul 2004 19:14:48 GMT

So, let's see. There's

Any others? (There used to be an implementation of Python named Vyper, but it no longer exists.)

I wouldn't complain about Python having too few independent implementations anymore.

Imperative programming

Thu, 29 Jul 2004 18:57:36 GMT

Just in time to fit to Michael's observation, I'm feeling like talking about Java and Python, again; nothing particularly deep, though, I'm afraid. You see, Ian Bicking ponders on application development models, PHP, and Zope. (And little Python too.) The point was, or the point as I twisted it to fit my universe, that implementing a software in a pure object oriented fashion does not give as much leeway as implementing it partly imperative way, partly object oriented. In imperative and iterative way, you pile functionality to the software and find the objects through experience during the development. And the abstractions needn't always be objects, they can be plain routines too.

This is exactly what I've been going through developing in Java versus in Python. Finding regular functions and procedures and modules, as opposed to objects, is just so much easier and natural to me. As you write code, you see that this chunk of code really is something you are going to need in some other context later; fine, let's put that into a routine. The great thing about this is that you don't lose anything with it. In fact, the code is generally more readable after such refactoring. The name of your routine succintly says what the chunk of code in it is supposed to do.

Whereas in a pure object oriented development, you generally need to think what this routine belongs to. Sure, you can slap the routine in Java to a static method and do the thing in a imperative way, but this is not the way Java programs are supposed to be structured; and I'm talking just about that. This gets into my way all the time. It's nice when things finally click and a thing can be stuffed into an object, but often they don't.

In other words, object orientation is a nice way to structure your program, though not necessarily the best way, but it's a poor way to figure out how to solve a specific "algorithmic" problem. The first thing that I do when I'm faced with a specific problem is to connect the dots from end to end (if that's possible), ie. to make sure that I can do the whole thing. Often skeching solution to the problem requires so much code that I need to restructure my code. This most often pretty easy in the imperative way, as I've explained.

I am willing to accept that I'm a poor object oriented programmer, but I think there's more to the issue than that. But I don't know.

...

As for PHP, I agree with Ian that PHP's strong point is specifically the fact that you can so easily slap a snippet of code to a web page, to, say, print a date on the page, and organically grow it by slapping more snippets to the page. I've found PHP rather icky to program with, but maybe that's just an attitude. PHP might not be a great programming language, as far as programming languages are considered as such, but it's definitely a great tool for a broad set of problems.

Python on Smalltalk VM?

Thu, 29 Jul 2004 18:09:54 GMT

It seems to have escaped from Daily Python URL, and other Pythonous news sources, that L. Peter Deutsch of the Ghostscript fame is working on a project to make Python programs run on a Smalltalk virtual machine. From CampSmalltalk's Wiki:

pycore is a project to run Python programs, unchanged, on Smalltalk. It currently runs a few tiny test programs on VisualWorks, 5-10x faster than the standard Python implementation. I'm going to get it to the point where it will run some small but realistic pre-existing Python program. I'd also love to get some help in making the code nicely transportable between VW and Squeak.

We now know to be careful predicting the future and hoping too much, but that sounds impressing to me.

[Spotted via Danny O'Brien's weblog.]

IronPython in the wild

Thu, 29 Jul 2004 04:56:00 GMT

In other news, IronPython is released to the world. Jim Hugunin also works for Microsoft now, with the Common Language Runtime team, giving him a possibility to concentrate on the development of IronPython. Congratulations to Jim.

I currently do not work on the CLR platform, but it's nice to see a functional and efficient implementation of Python on it.

Update: Edd Dumbill has a good coverage on the release.

Paul Graham and Arc

Wed, 28 Jul 2004 19:36:43 GMT

I listened to Paul Graham's interview at ITConversations. You've got to admire his approach to the development of Arc, a new Lisp dialect. He is not planning to release it until he feels it's ready, which might take several years. A lot of people are, even as of now, eagerly waiting for the release. And I bet Paul sometimes feels a strong urge to just show the world the amazing stuff he has done and get on with the project in public.

Even if you are just programming some little utility for yourself and your co-workers, there's a point in the development, when a feeling of wanting to share your work just overwhelms you. "Look at this neat script!" As soon as you actually share it, you suddenly realize dozen things that are broken and another dozen things wrong with the UI (be it graphical or command line UI or a programming API). And if the utility happens to be useful, you realize that you just cannot change it that easily anymore. There are people already using it and you find yourself at the maintenance phase of the classic waterfall model. It's not that it isn't fun developing the thing anymore, but that it's considerably more constrained.

And I'm just talking about a silly little utility.

Think of the implications in programming languages. It takes a heck of a lot of work to try to improve a language (and it's libraries) and to provide some sort of a backwards compatibility. (Now, place yourself in the shoes of a programmer that, yet again, runs into a Java API document, that describes a perfect method for your needs, just to find out that it's DEPRECATED. Bastards! It's not particularly a Java problem, though, but somehow I have the most painful experiences programming Java.) Furthermore, there's going to be a herd of vocal people that don't want you to change the language (at least in the way you are going to change it), even though they couldn't design their way through a cubicle. If only you could have anticipated all these things that need to be changed at the first place.

Of course, you rarely can anticipate future needs and future enlightenments. A bunch of languages have been born by sort of an accident. Guido never ment Python to be a full-blown general purpose programming language. Yet that's just what it is nowadays. And I'm not saying that Python isn't fit for general purpose programming, quite the contrary actually, but that it wasn't the idea Guido had in his mind when went on a prolonged Christmas vacation. (OK, the vacation didn't last longer, but its implications sure did.)

Anyways.

Graham obviously has a pretty strong agenda for Arc. He knows what he wants Arc to be good at, specifically web programming, and Arc, of course, has a rather solid foundation already, being a dialect of Lisp. Sure, Arc might get popular at some entirely different application area, or might not get popular at all, but there, nevertheless, will be a lot of hurdles along the way, a lot of things that need to be done completely different than originally thought. By holding of the release, he buys himself a lot of room and freedom with the design.

One might suggest that Graham could just release the damn thing, and after finding flaws in the core design, he could just say "fuck off, users", and release a new, totally incompatible version of Arc later, but that's not the way hackers go about. In fact, this relates to another concept Graham talked in the interview. Namely, empathy. He argues that great programmers can feel truly emphatic of their users. Great programmers can get into the shoes of the user, so to speak. And emphatic programmers just don't go introducing backwards incompatible changes to their language; unless necessary, of course. And if the users are programmers themselves, and the software is a programming language, the emotional and intellectual investment that the users devote to the software is something that you just can't ignore.

So, all in all, I wish that Graham can continue to design Arc in peace and comes up with a great programming language. I won't probably be using it, because I've kind of decided to stick with Python, for better of for worse, for my favorite language. I'll be using languages like Java or even Perl at work, and I'm fine with it; believe it or not, Paul, but Java is a right tool for a broad set of problems and contexts. But when I can make the decision myself, the implementation language is going to be Python. But never say never; I'll look forward to see as for what kind of language Arc turns out to be.

Category feeds, or, Python category feed

Thu, 15 Jul 2004 17:00:17 GMT

I realized that NewsBruiser has category feeds (if only I categorized my posts) and remembering the results of my reader survey earlier, I decided that I will categorize at least the posts related to Python so that you can filter out the non-relevant posts. Seems that I can't, or don't want to, restrict my posts strictly on the subject of Python.

So, here you are: Category feed containing only posts about Python.

Pie-thon update

Thu, 15 Jul 2004 16:42:47 GMT

Given that OSCON 2004 is approaching, I thought a brief update on Pie-thon should be in order.

So,

Trying out Quixote

Fri, 02 Jul 2004 07:13:59 GMT

Seeing Andrew mention the release of Quixote 1.0 (and the weather being rather moist), I decided to give it a try. I am impressed. I read through the demo application and a couple of articles, and I think I've grasped the basic architecture. The main design principle of Quixote is evident: Quixote applications are almost like normal Python applications.

(Flashback: I remember Etienne Posthumus (who blogs too) praising Quixote at Europython 2003 and searching for other Quixote users. I didn't know what the fuzz was all about then.)

Quixote, by design, does not make a clear distinction between web programmers and web designers (ie. HTML coders). While I can appreciate such a distinction in certain contexts, I think Quixote's approach fits better with my pet projects.

Let's see if I can come up with something for the public eye one day.

	Number of C projects / number of Perl projects	C / Python	Perl / Python
Freshmeat	2.2	4.0	1.8
Sourceforge	2.5	3.8	1.5