Language Musing - F*cking with Clusters

I had a great 10 line file earlier today that did just what I needed. At least some of those 10 lines were whitespace, one was a shebang, and two were import statements. But I had some trouble getting python running on our embedded platform, whereby I mean "import time" was failing on me. So I rewrote it to a 42 line C file. And cried a little.

import time has been the bane of my existance several times.

Ick.

Hmm, large projects?

Admittedly, I've only done one large project in python, and I probably made some mistakes in managing it that I won't make again, but nevertheless, the lack of enforced structure (and compiler warnings, I love those) completely shoots down my style of coding. I tried various linters, but it's not as effective. I still love Python, but I like to keep my programs under 2500 lines.

I also remember writing some kinds of file processing, mostly on a per-character basis, that were unfeasibly slow in Python, and ran in minutes in C, but I suppose those are special cases.

the nice thing is, you can get a whole lot done in 2500 lines :)

Its true. Also, what kind of large projects do you think would fail? There are large packages (numpy is the first that comes to mind) that work fairly well with a large developer base.

Do you mean "enterprise" code? If you do, I'm really curious why you think so, since that's the direction I was hoping to get some light on.

Hmm, I always thought that numpy was mostly C, with a python wrapper, but it turns out that it has 10x more python code than C code. Either way, there are countless other large python projects as well. I don't think large python projects have to fail, but I think that large projects deemphasize its strengths and emphasize its weaknesses.

Maybe I am thinking about component size, rather than project size. I just calculated the size of my second largest python project. It's not quite 2500 lines, but its split into numerous neat little scripts that communicate with each other through stdin/stdout. Within the components, most function calls pass around builtin types. No problem there, it wouldn't be overwhelming if it were twice as big.

My big python project consisted of much bigger individual components, and the components would pass around objects that I defined elsewhere. In this setup, the biggest problems occurred when I changed the "public interface" of a class, and forgot to change some of the places where the class was being used. Python issues no warnings, at best I get exceptions once the broken code is being executed. I can't even grep for the class name to find all the places I use it at, because I don't have to declare the types of my variables. Of course, many large python projects actually include that information in the comments or variable names, but then I might as well use a language with static type checking and enforce the convention.

In my understanding, enterprise code (you're talking about invoices, payroll, inventories and so on?) is written like that.

Another reason for me to not use Python in enterprise settings is that there are other languages that have been developed with that in mind, whereas Python hasn't. I assume that the creators of, say, Java are as a whole smarter than me (probably as individuals too, but I've never consciously met one of them ;-), and had good reasons to design the language the way they did, even if the reasons are not obvious to me at the moment.

I have the feeling this got way too long for a comment, but WTH.

(quick reply... I need to get to school)

Classes:
One thing I've forced myself to do is to never ever from foo import *. Having to type out the "class" every time makes my life a lot easier when I break something :-)

Enterprise Code:
Honestly, I have no idea what this means. Its just one of those fancy sounding words that associates in my mind to "grown-up code". Invoices and the like certainly fall into that category.
One thing.. I'm trying to find a source for this, but my PLs prof claims that Java was written at Sun as a quick printer control language, and then picked up by management because they wanted a "next-generation" language. His claim was that this is part of the reason Java I/O blows.

About the file processing... what data types were you using? What's unreasonable? It takes me a bit under half an hour to extract data from 17G text files in Python on a 256mb RAM machine.

I looked at the C code (I threw away the python version). It goes through a large file and replaces all [s with (s and ]s with )s, except at the outermost nesting level, eg, "[foo[bar]][foo[baz]]" becomes "[foo(bar)][foo(baz)]". I did that on a 17G file. I don't remember the exact timing, but the python version never finished, the C one takes an estimated 15 minutes on this box.

This is also a case where the python code is no simpler than the equivalent C code. It's 34 lines of C.

This comment is just to inform you that this and the previous post were dated 2026. Can I have a ride in your time machine?

So, I mistyped my journal date sometime senior year, and never bothered to fix it, and now its ballooned out of control :-P

I guess they need to import time...

I thought it was because the system clock on navi or polyp was broken?

You are absolutely right. I'd completely forgotten about that.

(and yea, it was Navi)