software engineering;  was: Is this official blog article intended as a joke?

Actually, the 'blog makes a useful point:

Most computer science students get to learn high-level programming languages and application development but they don’t really understand computers.

JMO/YMMV:

Now, while interfacing to the real world is fun, RasPi doesn't really address the high-level application problem.  It's still a GNU/Linux box, running a mainframe operating system with dozens of processes.  It's darn cheap for a mainframe, but it's still a mainframe.

If you really want to learn what computers do IMO you really need to learn assembly language and run it on a real mini- or micro-computer with no operating system present to protect you from stomping all over your code and data so that your program goes off to la-la land.  That's when you really see that all that's in computer memory is bit-strings, and the computer will interpret them as instructions or data depending on where you told it to execute -- whether you meant to or not.  Once you've mastered the machine-language level, then you can understand why your C program is misbehaving in that extraordinary fashion.  IMO this mastery is necessary for someone to be called a computer scientist, just as you expect an EE to know how to use Kirchhoff's Laws and a mathematician to be able to do Epsilon-Delta proofs.

> IMO this mastery is necessary for someone to be called a computer scientist,

Of course, computer science is a big field, with many specialties, and many layers

of abstraction, and it's entirely possible for hardware specialists to know very little

about software, and vice versa.  And theoretical computer scientists often know very

little about either software or hardware, focusing their attention on topics like

algorithmic efficiencies that don't provide any benefit on realisitic sized data sets.

Dijkstra famously said "computer science is no more about computers than

astronomy is about telescopes."

Of course, it's still important to know how to read an abend dump when you have to.

Very dangerous, since a different compiler or version might allocate the variables differently.  Much better to use EQUIVALENCE

Yes, that was my point, that languages that allow you to violate their abstraction models

are dangerous.

Awful C implementation, with 100 messages per syntax error

I've been noticing a lot of such cascading error messages in the Arduino forums when

for example there is a spurious semicolon before the opening '{' of a function.

http://www.element14.com/community/thread/10083?start=1138&tstart=0

Modern C compilers do an excellent job of optimization.

But the optimization suffers due to the availability of untyped memory operations

such as memcpy and memmove:

http://gcc.gnu.org/onlinedocs/gccint/Memory-model.html#Memory-model

and the difficulties of type-based alias analysis in such cases:

http://gcc.gnu.org/onlinedocs/gccint/Alias-analysis.html#Alias-analysis

But it can be a lot more fun to program in "unsafe" languages, and a lot more satisfying

you can also waste a lot of time debugging that way, because errors that would

otherwise be caught at compile time or in a run-time array bounds check, 

can turn into mysterious run-time behavior where evidence of the error may

not show up until long after the program has gotten off track.

coder27 wrote:

 

But it can be a lot more fun to program in "unsafe" languages, and a lot more satisfying

 

you can also waste a lot of time debugging that way, because errors that would

otherwise be caught at compile time or in a run-time array bounds check, 

can turn into mysterious run-time behavior where evidence of the error may

not show up until long after the program has gotten off track.

In my own programming, I rarely have bugs that could have been caught by run-time array bounds checking.  It's usually logic errors such as forgetting to handle an unusual case and the occasional dangling pointer, which is part of the price of writing code that uses (and reuses) linked data structures.  My solution is to try to keep things as simple as possible, and it works for me.  One way to keep things simple is to use a language that's well-adapted to the problem, so you don't have to encrypt the problem to fit the programming language.

My general opinion is that if you have sharp programmers, you can give them sharp tools.  If you don't, then you either need to give them "safe" tools or replace them with sharper programmers.

John Beetem wrote:

 

My general opinion is that if you have sharp programmers, you can give them sharp tools.  If you don't, then you either need to give them "safe" tools or replace them with sharper programmers.

Some programmers are sharper than others, but there are no human programmers sharp enough to fully comprehend algorithmic and state complexity in any non-trivial software system.  The disparity between programs and human capability to understand complexity and track dynamic state is vast, many orders of magnitide.  We would need IQs in the thousands (maybe millions) to program without introducing errors, no matter how many decades we've spent mastering a language.

This is why the bridges of software engineering collapse around the world millions of times each day.  It's not that the vast majority of programmers are amateurs or that the education system has failed us or that programming is a cool craft instead of an engineering discipline.  Those things could be fixed.  The underlying problem of human capability mismatch cannot be fixed, as long as we continue along the present track.

And there is little sign of anyone looking for alternative tracks, at least in practical computing.

Morgaine Dinova wrote:

 

Some programmers are sharper than others, but there are no human programmers sharp enough to fully comprehend algorithmic and state complexity in any non-trivial software system.  The disparity between programs and human capability to understand complexity and track dynamic state is vast, many orders of magnitide.  We would need IQs in the thousands (maybe millions) to program without introducing errors, no matter how many decades we've spent mastering a language...

Excellent comments, Morgaine.

You've heard me talk about this before, so the following is for others following this thread.  I believe we have complex systems because it's easier to slap together a complex system out of parts that "ought to work together" than to  think carefully through a system -- and document it -- before getting so far into the implementation that it's impossible to make fundamental changes.  I have met way too many soi-disant software engineers who hate writing software (not to mention documentation), so they mash together inappropriate modules from Edsger Dijkstra knows where.  I have also met way too many engineering managers who want something quick-and-dirty now instead of doing it right -- and then re-doing it righter after learning from the first go-around.

C.A.R. Hoare says it best, IMO:

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.

Actually, IMO humans are quite good at working with complex things once they have come up with good notations -- textual and graphical -- to describe them.  However, the development of those notations usually takes decades or centuries.

The solution to complexity is abstraction.  A single person can run a large company

or run a large nation, but not by doing all the jobs that a sole proprietor does.  Instead,

they operate at much higher levels of abstraction.

I don't think the analogy of "sharp tools" to "low level programming languages" works too well.

Certainly low-level features get removed from higher level languages, for example, gotos have

been removed from recent languages.  And if you focus on the loss, it may seem like your language

has gotten duller.  But that's usually not true, because new higher-level abstractions are added

as lower-level ones are removed.

In some cases, the benefits are not obvious to the programmer, but they are to the compiler

writer, because keeping lower-level abstractions around allows them to be used to subvert

the higher-level abstractions.

For example, the typical compiler intermediate language these days is based on SSA,

and that requires a control-flow graph in the form of a DAG (no cycles), which in turn requires

that you don't have arbitrary gotos, such as between branches of an if/then/else construct.

So by taking away the low-level goto, and replacing it with higher-level if/then/else and looping

constructs, you end up with a language that is no less capable, but much more suitable for

SSA-based optimization, code generation, and other analysis.

coder27 wrote:

 

The solution to complexity is abstraction.  A single person can run a large company

or run a large nation, but not by doing all the jobs that a sole proprietor does.  Instead,

they operate at much higher levels of abstraction.

Abstraction is part of the solution, but it's not sufficient by itself.  With abstraction alone, computers will continue to do exactly what we tell them to do, and will inevitably end up doing the wrong thing because it is entirely beyond our mental ability to fully understand (let alone specify) the behavior of a non-trivial system.  Removing unnecessary detail and bringing the language of specification closer to the natural language of the application domain through DSL abstraction clearly helps, but it is not enough.  The desired complex behavior still has to be defined, somewhere, and that's the problem.

Your example of the large company or nation is a very good one, but not as an illustration of the power of abstraction.  What it illustrates instead is that complex systems can work well when there is semantic commonsense built into the components of the system, so that daft commands from above don't cause an immediate collapse.  And that's what is lacking almost completely in software today.

Remember Bertrand Meyer's very insightful work on Eiffel, in which preconditions, postconditions, and invariants played such an important part?   He embedded semantic constraints into software components, although his focus was on simple design contracts at a level far below anything related to what people might call "commonsense".  Unfortunately, static constraints are fundamentally at odds with flexibility and reuse, so we see them only very rarely in practical software beyond basic assertions for defensive programming.  We are as far away as ever from the model of companies or nations working in software.

John Beetem wrote:

 

Actually, IMO humans are quite good at working with complex things once they have come up with good notations -- textual and graphical -- to describe them.  However, the development of those notations usually takes decades or centuries.

It's interesting (or perhaps sad) to contrast that with the typically short  timescales of product and company planning in the commercial software world.  One might be tempted to suggest that open source could be the answer, since it doesn't end-of-life code with quite the same degree of profit-oriented ruthlessness, but alas even open source programs bit rot and stop working owing to dependencies and changing architectures.

The long-term view just isn't there, and may not even be possible.  Things change, it's part of progress.  Whatever solutions we come up with for the software crisis, they'll need to be able to cope with the current state of technology.  I very much doubt that refinement of languages or written code over ages will help much.

Morgaine Dinova wrote:

 

coder27 wrote:

 

The solution to complexity is abstraction.  A single person can run a large company

or run a large nation, but not by doing all the jobs that a sole proprietor does.  Instead,

they operate at much higher levels of abstraction.

 

Abstraction is part of the solution, but it's not sufficient by itself.  With abstraction alone, computers will continue to do exactly what we tell them to do, and will inevitably end up doing the wrong thing because it is entirely beyond our mental ability to fully understand (let alone specify) the behavior of a non-trivial system.  Removing unnecessary detail and bringing the language of specification closer to the natural language of the application domain through DSL abstraction clearly helps, but it is not enough.  The desired complex behavior still has to be defined, somewhere, and that's the problem.

Another problem is that of leaky abstraction layers.  My favorite example of this is C.  As I said earlier in this discussion, I think of C as a portable assembly language which uses high-level notations, and if you think of C that way it works very sharply indeed.  However, if you think of C as a high-level language, you will be disappointed at run time when machine-level behavior damages your nice high-level model.

Just adding abstraction layers can be problematic, if only because the abstraction layer may hide terrible performance problems that you'll only uncover when the system is fully deployed.

Also, IMO opinion many large (and even small) companies and countries are very poorly run whether or not their leaders are capable of abstract thinking or common sense.  I'm sure you can think of examples.  I believe the principal cause of this is the Law of Large Organizations:

In any Large Organization, Loyalty will always be rewarded over Competence.

Come to think of it, many large software systems suffer from this as well

Morgaine Dinova wrote:

 

John Beetem wrote:

 

Actually, IMO humans are quite good at working with complex things once they have come up with good notations -- textual and graphical -- to describe them.  However, the development of those notations usually takes decades or centuries.

 

It's interesting (or perhaps sad) to contrast that with the typically short  timescales of product and company planning in the commercial software world.  One might be tempted to suggest that open source could be the answer, since it doesn't end-of-life code with quite the same degree of profit-oriented ruthlessness, but alas even open source programs bit rot and stop working owing to dependencies and changing architectures.

 

The long-term view just isn't there, and may not even be possible.  Things change, it's part of progress.  Whatever solutions we come up with for the software crisis, they'll need to be able to cope with the current state of technology.  I very much doubt that refinement over ages will help much.

Open source certainly helps, since it's not subjected to the limitations imposed by having to make money

Ars longa, vita brevis.

John Beetem wrote:

 

Open source certainly helps, since it's not subjected to the limitations imposed by having to make money

Unfortunately open source + free software is not fully isolated from the ill effects of preoccupation with money, short-term "product" focus, business planning and corporate competition.  Although individual projects may be created and developed by one or a small number of non-commercial and wholly altruistic enthusiasts, as soon as a project blooms into a global community resource it gains contributors with different needs and aspirations, many of them in direct support of business requirements and a focus on product and competitive advantage.

Open source code inevitably gets modified to support the latest devices or architectures, which typically undoes any previous validation that may have been done on it.  New drivers or patches are often added by corporate developers for whom getting support for a new gizmo is part of their jobs, but who will generally not continue to support their changes once those gizmos have been EOL'd in the company product cycle since they will not be paid to do so.  And unless they are exceptionally community-minded, they'll not be going out of their way to create generic infrastructure that supports their competitors, but instead focus on providing best support for their own requirements as their main priority.

So, open source doesn't exist entirely within an isolated collaborative space as is sometimes portrayed.  Even with everyone doing their level best and not having hidden agendas nor being deliberately evil, the nasty realities of the world outside FOSS idealism do intrude and mess things up to some degree.  Old code that is perceived as crufty is often removed purely to keep the more modern code sane, which means that honing code to a high state of reliability and trustworthiness over long periods of time just doesn't happen --- it bit rots or gets modified long before then, so validation and long-term stability just isn't useful or even possible in practice.

This is why I say that any solution to the software crisis cannot come from the "decades or centuries" that you mentioned, not even in FOSS.  It will have to be applicable to the code that exists at any given point in time or it won't be addressing the problem.

Morgaine.

John Beetem wrote:

 

Open source certainly helps, since it's not subjected to the limitations imposed by having to make money

Unfortunately open source + free software is not fully isolated from the ill effects of preoccupation with money, short-term "product" focus, business planning and corporate competition.  Although individual projects may be created and developed by one or a small number of non-commercial and wholly altruistic enthusiasts, as soon as a project blooms into a global community resource it gains contributors with different needs and aspirations, many of them in direct support of business requirements and a focus on product and competitive advantage.

Open source code inevitably gets modified to support the latest devices or architectures, which typically undoes any previous validation that may have been done on it.  New drivers or patches are often added by corporate developers for whom getting support for a new gizmo is part of their jobs, but who will generally not continue to support their changes once those gizmos have been EOL'd in the company product cycle since they will not be paid to do so.  And unless they are exceptionally community-minded, they'll not be going out of their way to create generic infrastructure that supports their competitors, but instead focus on providing best support for their own requirements as their main priority.

So, open source doesn't exist entirely within an isolated collaborative space as is sometimes portrayed.  Even with everyone doing their level best and not having hidden agendas nor being deliberately evil, the nasty realities of the world outside FOSS idealism do intrude and mess things up to some degree.  Old code that is perceived as crufty is often removed purely to keep the more modern code sane, which means that honing code to a high state of reliability and trustworthiness over long periods of time just doesn't happen --- it bit rots or gets modified long before then, so validation and long-term stability just isn't useful or even possible in practice.

This is why I say that any solution to the software crisis cannot come from the "decades or centuries" that you mentioned, not even in FOSS.  It will have to be applicable to the code that exists at any given point in time or it won't be addressing the problem.

Morgaine.

Morgaine Dinova wrote:

 

John Beetem wrote:

 

Open source certainly helps, since it's not subjected to the limitations imposed by having to make money

 

Unfortunately open source + free software is not fully isolated from the ill effects of preoccupation with money, short-term "product" focus, business planning and corporate competition.  Although individual projects may be created and developed by one or a small number of non-commercial and wholly altruistic enthusiasts, as soon as a project blooms into a global community resource it gains contributors with different needs and aspirations, many of them in direct support of business requirements and a focus on product and competitive advantage...

Good points.  However, I would say that FLOSS (Free-as-in-Liberty Open-Source Software) generally benefits from participation by for-profit entities as long as the software uses a copy-left license.  Yes, money-driven entities will want to add code for their own purposes, but if they want to use immature FLOSS for serious purposes they'll find bugs and they'll be required by the license to publish the fixes to those bugs.  The rest of the FLOSS community can scoop up the bug fixes and improve the quality of the software.  It's very hard for an individual developer or even a small team to catch and fix everything, so having people paid to help is usually beneficial.

Example number 1 is obviously Linux.  It has grown amazingly over its lifetime, both in use and... in size.  I think of Linux as a terrific mainframe OS kernel, and when you think of it that way then its size isn't so shocking.  I think there's room for a light-weight personal computer OS, but I think we'll have to wait for Microsoft to complete its self-destruction first.  I don't think the wait is going to be long -- they seem pretty eager.

Linux benefits from having a foul-mouthed benevolent dictator to enforce quality, and from early adoption by IBM who defended Linux from early attempts to destroy it.

Example number 2 is Open Office.  That's a good example of a project that had to "fork it" to remain FLOSS.  I agree with RMS that you can't get the benefits of open-source software without the Four Freedoms, because otherwise corporations will take a snapshot of OSS and then keep all improvements to themselves.  It will be interesting to see how long Open Office stays around versus Libre Office.